disk usage

Message boards : Number crunching : disk usage

To post messages, you must log in.

AuthorMessage
David SProject Donor
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 17042
Credit: 20,939,656
RAC: 6,116
United States
Message 1344184 - Posted: 8 Mar 2013, 20:24:29 UTC

I was rather surprised this morning when both Einstein and Seti told me that one of my rigs had no tasks in progress. Checking the messages, they are both saying I don't have enough room on my hard drive. It's a rather small drive, but I don't store a lot on it and don't use the machine for anything other than BOINC, MagicJack, and my radioreference.com feed.

Do Boinc projects ever leave stuff on hard drives that they don't need any more? Is there a way I can clean it out? Or should I be looking elsewhere for a culprit?


David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


ID: 1344184 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45907
Credit: 815,088,577
RAC: 125,440
United States
Message 1344188 - Posted: 8 Mar 2013, 20:27:50 UTC - in response to Message 1344184.

I was rather surprised this morning when both Einstein and Seti told me that one of my rigs had no tasks in progress. Checking the messages, they are both saying I don't have enough room on my hard drive. It's a rather small drive, but I don't store a lot on it and don't use the machine for anything other than BOINC, MagicJack, and my radioreference.com feed.

Do Boinc projects ever leave stuff on hard drives that they don't need any more? Is there a way I can clean it out? Or should I be looking elsewhere for a culprit?

In your Boinc manager, click on the 'disk' tab.
I believe Einstein is notorious for leaving behind lots of files.
Cats.....what more does one need?

Have made friends in this life.
Most were cats.

ID: 1344188 · Report as offensive
Profile Mike
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 29576
Credit: 49,078,600
RAC: 16,964
Germany
Message 1344229 - Posted: 8 Mar 2013, 22:15:51 UTC

You are totally right Mark.
Einstein leaves a lot on the hard drive.


With each crime and every kindness we birth our future.

ID: 1344229 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11140
Credit: 83,701,296
RAC: 45,329
United Kingdom
Message 1344239 - Posted: 8 Mar 2013, 22:32:20 UTC - in response to Message 1344229.

You are totally right Mark.
Einstein leaves a lot on the hard drive.

Yes, that's a deliberate feature called "locality scheduling".

You can get a lot of different workunits out of the same set of data files. Once you've downloaded them once, you may not need to download any more for several days.

But if you just delete the files, BOINC will download them again at the next restart. Search the Einstein forums (or ask Gary Roberts) how to clean your client_state.xml file so you don't waste that download bandwidth.

ID: 1344239 · Report as offensive
juan BFP
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 5847
Credit: 330,541,668
RAC: 7,723
Panama
Message 1344242 - Posted: 8 Mar 2013, 22:35:25 UTC

E@H uses a big 8MB or more WU, so if you have a large cache (10 days for example) it will use a lot of HDD space.

If you run both projects at the same time the best thing to do is use a small days cache (anyway the limit of 100WU still on SETI) to avoid that.

I use 1 day cache on my hosts, that allow SETI DL all the 100WU and keep the number of E@H WU in a confortable margin.


ID: 1344242 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11140
Credit: 83,701,296
RAC: 45,329
United Kingdom
Message 1344247 - Posted: 8 Mar 2013, 22:44:55 UTC

Note that Einstein has different types of WU, same as we have MB and AP.

The WUs for GPUS, 'BRP4' (Binary Radio Pulsar) use 'one time only' data files - a big download (although they've compressed them recently, so less than 16 MB per WU to download), but clean up after themselves. You don't need a big cache at Einstein, they have reliable servers.

It's the CPU tasks (Gravitational Wave S6 LineVeto search) which hang on to the data files for re-use.

ID: 1344247 · Report as offensive
Profile Ageless
Avatar

Send message
Joined: 9 Jun 99
Posts: 13815
Credit: 3,269,733
RAC: 0
Netherlands
Message 1344250 - Posted: 8 Mar 2013, 22:53:11 UTC - in response to Message 1344229.

It doesn't just leave a lot on the hard drive, it's keeping the files for when you're returning to running a task with similar sub-sets of data, do you don't have to download all that data again. Don't forget that Einstein uses Locality Scheduling which is to minimize the amount of data transfer to hosts by preferentially sending jobs to hosts that already have some or all of the input files required by those jobs.


Jord

Ancient Astronaut Theorists suggest that in many ways, you can be considered an alien conspiracy!

ID: 1344250 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6092
Credit: 155,171,465
RAC: 48,675
United States
Message 1344330 - Posted: 9 Mar 2013, 3:44:10 UTC

Sometimes I do find stray files from tasks that BOINC has run & for whatever reason not deleted once finished. Normally I only look on machines when they fall back to one of their backup projects. At most it only turns out to be a few tasks.

I would probably just check the size of the BOINC folder on that drive & see if it seems larger than expected.


SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

ID: 1344330 · Report as offensive
Profile MarkJProject Donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 978
Credit: 33,083,967
RAC: 7,691
Australia
Message 1344332 - Posted: 9 Mar 2013, 3:56:53 UTC

The quickest and easiest way to clean up Einstein is to detach and reattach (or as its now known remove and then add).

Quite often they have moved through files but won't clean up until you do the above or the entire run finishes.


BOINC blog

ID: 1344332 · Report as offensive
Profile Ageless
Avatar

Send message
Joined: 9 Jun 99
Posts: 13815
Credit: 3,269,733
RAC: 0
Netherlands
Message 1344465 - Posted: 9 Mar 2013, 9:28:42 UTC - in response to Message 1344332.
Last modified: 9 Mar 2013, 9:34:13 UTC

Quite often they have moved through files but won't clean up until you do the above or the entire run finishes.

Quite often? Weird then that I have never have seen this happen in all the years I've run Einstein. Got links to all the many threads on their forums where you and others complain about it and moderators/admins answer that they know it happens quite often? Or is the Quite Often just a personal feeling (because you don't understand LS)?

Edit: it may have happened once, when they switched from the S5 search to the S6 search, but since all searches since then have all been a form or continuation of the S6 search, the data files have been the same. Thus no need to remove them when e.g. S6Bucket stops and S6LV starts.
Jord

Ancient Astronaut Theorists suggest that in many ways, you can be considered an alien conspiracy!

ID: 1344465 · Report as offensive
Profile MarkJProject Donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 978
Credit: 33,083,967
RAC: 7,691
Australia
Message 1344482 - Posted: 9 Mar 2013, 10:09:51 UTC - in response to Message 1344465.
Last modified: 9 Mar 2013, 10:12:03 UTC

Quite often they have moved through files but won't clean up until you do the above or the entire run finishes.

Quite often? Weird then that I have never have seen this happen in all the years I've run Einstein. Got links to all the many threads on their forums where you and others complain about it and moderators/admins answer that they know it happens quite often? Or is the Quite Often just a personal feeling (because you don't understand LS)?

Edit: it may have happened once, when they switched from the S5 search to the S6 search, but since all searches since then have all been a form or continuation of the S6 search, the data files have been the same. Thus no need to remove them when e.g. S6Bucket stops and S6LV starts.


The WU are for a particular frequency which as far as I understand they work their way up the band. Even though they may be at the upper end of the band the data files for the lower ones will still be left on your machine. As I said these would not normally be deleted unless the search finished or the user were to detach/reattach.

While there may be no need to remove them the question was asked how to.

I don't need to remove them I have plenty of disk space. I understand how locality scheduling works so I have no reason to complain about this, I was merely answering the question.
BOINC blog

ID: 1344482 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 13328
Credit: 154,602,821
RAC: 116,557
United Kingdom
Message 1344492 - Posted: 9 Mar 2013, 10:22:17 UTC

Sorry to say Ageless Einsein is still "bad" at not clearing up after itself. Particularly so when it suffers time-outs.
One cruncher on which I ran Einstein as a "priority 0" back up had over 2Gb of time-out files, and another has over 200Mb of them. Both these were after all visible work had completed, with "No New Tasks" set and so should have been down to the barest minimum, not hundreds of Mb of "rubbish" left lying around.


Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

ID: 1344492 · Report as offensive
Profile SciManStev
Volunteer tester
Avatar

Send message
Joined: 20 Jun 99
Posts: 5846
Credit: 105,983,452
RAC: 2,150
United States
Message 1344503 - Posted: 9 Mar 2013, 11:41:58 UTC

This thread prompted me to look at my Einstein work. I had completed all my tasks a while back, but had about 6.5 Gig of files still on my rig. I detached, and reattached, and they cleared. I don't know what they were, and I have plenty of hard drive space, so I really didn't care. I was surprised that there was 6.5 Gig.

Steve


Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

ID: 1344503 · Report as offensive
juan BFP
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 5847
Credit: 330,541,668
RAC: 7,723
Panama
Message 1344510 - Posted: 9 Mar 2013, 12:15:24 UTC

I take a look at my E@H data and nothing like this noticed. Maybe something else is in place there.


ID: 1344510 · Report as offensive
Wedge009
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 448
Credit: 241,734,702
RAC: 93,190
Australia
Message 1344844 - Posted: 10 Mar 2013, 8:07:39 UTC

Very rarely, I've seen a scheduler request to the Einstein@Home servers result in a 'request from the server' to delete some data files, but they will only be very old ones. In general, they seem to keep data around for future search runs. They also seem very reluctant to delete old application versions.

For the record, I currently have 1.53 GiB in my Einstein directory accumulated since May 2012, compared with 213.61 MiB for SETI@home. I don't process Einstein WUs very much.


Soli Deo Gloria

ID: 1344844 · Report as offensive
David SProject Donor
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 17042
Credit: 20,939,656
RAC: 6,116
United States
Message 1345186 - Posted: 11 Mar 2013, 3:36:25 UTC

I detached and reattached Einstein, with no noticeable difference in the space used on my drive. It also did not solve my problem.

It turns out that, despite my earlier statement, I am storing a lot of files on it. I identified some I could safely delete and did so, then clicked Update for both projects. Seti deferred me and opportunistic Einstein downloaded over 100 tasks. Seti eventually got 100 as well.


David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


ID: 1345186 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45907
Credit: 815,088,577
RAC: 125,440
United States
Message 1345187 - Posted: 11 Mar 2013, 3:41:24 UTC - in response to Message 1345186.

I detached and reattached Einstein, with no noticeable difference in the space used on my drive. It also did not solve my problem.

It turns out that, despite my earlier statement, I am storing a lot of files on it. I identified some I could safely delete and did so, then clicked Update for both projects. Seti deferred me and opportunistic Einstein downloaded over 100 tasks. Seti eventually got 100 as well.

Don't forget to empty the trash bin if you have not already done so.
I also usually find a number of good sized log and dump files that Windows writes to disk after a crash. You can turn this off.

I used to have a nifty little program that would graphically display the entire hard drive disk usage, but I cannot find it now.
Cats.....what more does one need?

Have made friends in this life.
Most were cats.

ID: 1345187 · Report as offensive
Highlander
Avatar

Send message
Joined: 5 Oct 99
Posts: 167
Credit: 33,135,855
RAC: 3
Germany
Message 1345208 - Posted: 11 Mar 2013, 5:52:19 UTC

You mean this one?

TreeSize Free Portable -> http://portableapps.com/apps/utilities/treesize-free-portable




- Performance is not a simple linear function of the number of CPUs you throw at the problem. -

ID: 1345208 · Report as offensive
David SProject Donor
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 17042
Credit: 20,939,656
RAC: 6,116
United States
Message 1345317 - Posted: 11 Mar 2013, 14:03:20 UTC - in response to Message 1345187.

Don't forget to empty the trash bin if you have not already done so.

That was the first and last thing I did.

I also usually find a number of good sized log and dump files that Windows writes to disk after a crash. You can turn this off.

I used to have a nifty little program that would graphically display the entire hard drive disk usage, but I cannot find it now.

I also ran disk cleanup, but it didn't make a lot of difference either.

David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


ID: 1345317 · Report as offensive

Message boards : Number crunching : disk usage


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.