Power Outage Recovery - Information

Message boards : Number crunching : Power Outage Recovery - Information
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Pappa
Volunteer tester
Avatar

Send message
Joined: 9 Jan 00
Posts: 2562
Credit: 12,301,681
RAC: 0
United States
Message 959402 - Posted: 29 Dec 2009, 22:58:27 UTC

As everyone is waiting for the Workunit storage server to get healthy again. I have made the Thread sticky so that everyone Might be able to see.

Quote from the Main Page

Scheduled Power Outage - update
We are now recovering from a planned power outage, during which all services were off line. The workunit storage machine is undergoing a RAID resync and no new workunits will be generated until this has finished. The outage was for power upgrades on campus, and a similar outage will happen again this upcoming Sunday (01/03/2010) at 12 noon (Pacific time). 29 Dec 2009 4:26:16 UTC

What this means you we be getting messages in the Boinc > Messages Tab that state:

12/29/2009 2:49:01 PM SETI@home Scheduler request completed: got 0 new tasks
12/29/2009 2:49:01 PM SETI@home Message from server: (Project has no jobs available)


We Wait

Regards

Please consider a Donation to the Seti Project.

ID: 959402 · Report as offensive
Profile Pappa
Volunteer tester
Avatar

Send message
Joined: 9 Jan 00
Posts: 2562
Credit: 12,301,681
RAC: 0
United States
Message 959445 - Posted: 30 Dec 2009, 2:54:11 UTC

As it has been noted, the "resync" is completed and we are now in the recovery part.

All things said and done, it is better it you just let Boinc do its thing. I noticed that my only cold GPU (which is doing only Seti Main and Lunatics offline testing) managed to get a Workunit all by itself.

If the recovery is "typical" somewhere before/around midnight PST your machines will have work. That is based on a normal outages takes about 3 hours. An extended outage takes up to 6 hours. The outlier would be the newer versions of Boinc 6.10.xx which with the Project Backoff might take a bit longer to request work.

So now we get into the "http errors" as the servers are too busy to process your connection request.

Sometimes I think I have been watching this too long.

Regards


Please consider a Donation to the Seti Project.

ID: 959445 · Report as offensive
jstonge123

Send message
Joined: 27 Sep 99
Posts: 8
Credit: 908,512
RAC: 3
United States
Message 959483 - Posted: 30 Dec 2009, 5:47:11 UTC - in response to Message 959445.  

Is it possiable once the sytem recovers too download extra workunits to ride through the next outage?

Thanks for the update
ID: 959483 · Report as offensive
Luke
Volunteer developer
Avatar

Send message
Joined: 31 Dec 06
Posts: 2546
Credit: 817,560
RAC: 0
New Zealand
Message 959484 - Posted: 30 Dec 2009, 6:02:21 UTC - in response to Message 959483.  

Is it possiable once the sytem recovers too download extra workunits to ride through the next outage?

Thanks for the update


Yes it is. Go into Your Account > Computing Preferences > Network Usage > Edit Preferences > and change "Maintain enough work for an additional x days" (where x equals how many days of workunits you have stockpiled) to a number around 2-4. No more than 5. That oughta' be enough.

Basically, this means, you should have enough work to keep you going for x days before you run out, if the servers are down.
- Luke.
ID: 959484 · Report as offensive
Profile Pappa
Volunteer tester
Avatar

Send message
Joined: 9 Jan 00
Posts: 2562
Credit: 12,301,681
RAC: 0
United States
Message 959485 - Posted: 30 Dec 2009, 6:05:04 UTC - in response to Message 959483.  
Last modified: 30 Dec 2009, 6:06:40 UTC

Is it possiable once the sytem recovers too download extra workunits to ride through the next outage?

Thanks for the update


Yes, it is possible. In your Account settings Computing preferences and look for these items.

Computer is connected to the Internet about every
(Leave blank or 0 if always connected.

My setting is 0.1 days (or try to connect every 2.5 hours)

The next one is

Maintain enough work for an additional
My setting is 1 day.

This is the one if you increase it to 2 days then you should be fine for the next outage.

The "caution" is that when you do that Boinc may have issues for a while. There are times with the type of Workunits that you get you could end up in a nasty situation. The most I would ever recommend is 3 days. Yes, there are reasons I am set to only 1 day. -9 Workunits and the Quota could make things very nasty. Then you could spend time trying to recover.

Regards
Please consider a Donation to the Seti Project.

ID: 959485 · Report as offensive
Odysseus
Volunteer tester
Avatar

Send message
Joined: 26 Jul 99
Posts: 1808
Credit: 6,701,347
RAC: 6
Canada
Message 959486 - Posted: 30 Dec 2009, 6:10:07 UTC - in response to Message 959483.  
Last modified: 30 Dec 2009, 6:12:12 UTC

Is it possiable once the sytem recovers too download extra workunits to ride through the next outage?

Yes: under Computing preferences on your Account page, edit the “Maintain enough work for an additional __ days” entry in the Network usage section.

P.S. Three answers for the price of one!
ID: 959486 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 959541 - Posted: 30 Dec 2009, 15:00:51 UTC - in response to Message 959486.  

Don't forget to check in your local preferences in your BOINC Manager under advanced settings too. I didn't and ran out of work. I had set my account page to 3 days but local prefs were only set to 2.


PROUD MEMBER OF Team Starfire World BOINC
ID: 959541 · Report as offensive
Profile champ
Volunteer tester
Avatar

Send message
Joined: 12 Mar 03
Posts: 3642
Credit: 1,489,147
RAC: 0
Germany
Message 959545 - Posted: 30 Dec 2009, 15:30:46 UTC - in response to Message 959541.  

Don't forget to check in your local preferences in your BOINC Manager under advanced settings too. I didn't and ran out of work. I had set my account page to 3 days but local prefs were only set to 2.



I hope, three days are enough.
ID: 959545 · Report as offensive
Profile Cliff Harding
Volunteer tester
Avatar

Send message
Joined: 18 Aug 99
Posts: 1432
Credit: 110,967,840
RAC: 67
United States
Message 959549 - Posted: 30 Dec 2009, 15:52:05 UTC - in response to Message 959545.  

I have had my BOINC computing preferences set to 10 days since day one and never had a problem. In doing this I never have to worry about local preferences. The scheduler has always done a fantastic job and I have only ran out of work only once in ten years. Every once in a while I will get a SETI WU that will go to high priority, although it happens more with one of my other projects.


I don't buy computers, I build them!!
ID: 959549 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 959551 - Posted: 30 Dec 2009, 16:07:59 UTC - in response to Message 959545.  

Champ, with it only 2 days I ran out just before they started to come back up. If I had had 3 days I would have carried over to the end and been just fine. My machine is a little slow and I like to get my results back in as fast as I can. I hate to make my wingmen wait on me. :-)


PROUD MEMBER OF Team Starfire World BOINC
ID: 959551 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 959662 - Posted: 31 Dec 2009, 2:15:14 UTC - in response to Message 959551.  


I have 3 day WU cache on both machines.
Both had ~ 12 hours before idle, then SETI@home came up and the machines could DL new WUs.

ID: 959662 · Report as offensive
Profile Pooh Bear 27
Volunteer tester
Avatar

Send message
Joined: 14 Jul 03
Posts: 3224
Credit: 4,603,826
RAC: 0
United States
Message 959691 - Posted: 31 Dec 2009, 4:15:49 UTC

I do multiple projects, so I never run out of work.

My movie https://vimeo.com/manage/videos/502242
ID: 959691 · Report as offensive
Profile champ
Volunteer tester
Avatar

Send message
Joined: 12 Mar 03
Posts: 3642
Credit: 1,489,147
RAC: 0
Germany
Message 959744 - Posted: 31 Dec 2009, 12:50:22 UTC - in response to Message 959691.  

I do multiple projects, so I never run out of work.



I am doing the same, but my second project (Rosetta) is running dry as well. :-(
ID: 959744 · Report as offensive
Profile Bill Walker
Avatar

Send message
Joined: 4 Sep 99
Posts: 3868
Credit: 2,697,267
RAC: 0
Canada
Message 959767 - Posted: 31 Dec 2009, 15:10:54 UTC - in response to Message 959744.  
Last modified: 31 Dec 2009, 15:11:33 UTC

I do multiple projects, so I never run out of work.



I am doing the same, but my second project (Rosetta) is running dry as well. :-(


Try Milky Way. It has its ups and downs, but when it is up it is UP. It will grab all your CPUs for a few days, and then everything else will go High Priority for a few days. Also, I find the science behind it very interesting, and very well explained on their home page.

Note to Mods: of course, I'm saying run this AS WELL AS Seti.

ID: 959767 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 959868 - Posted: 1 Jan 2010, 0:17:03 UTC - in response to Message 959767.  

I do run Milkyway on one of my computer, but I am one of those ATI guys crunching though 2 units every 10.5 minutes.

ID: 959868 · Report as offensive
Profile Pappa
Volunteer tester
Avatar

Send message
Joined: 9 Jan 00
Posts: 2562
Credit: 12,301,681
RAC: 0
United States
Message 960450 - Posted: 3 Jan 2010, 1:49:14 UTC

At this posting it is about 18 Hours (local time) until Power Outage Shutdown, for the next power outage. The graphs showed other than a delay getting everything running, Recovery took ~18 hours. So if you have not set your "computing preferences" to obtain 2 days work. Or do not have Backup Projects...

Now is the time to consider that.

"We" know that going from "Cold Iron" to fuctioning Seti Seti Servers is not easy and takes time.

Regards

Please consider a Donation to the Seti Project.

ID: 960450 · Report as offensive
Profile Jim H

Send message
Joined: 28 Nov 06
Posts: 12
Credit: 2,186,439
RAC: 0
United States
Message 960989 - Posted: 5 Jan 2010, 21:06:15 UTC
Last modified: 5 Jan 2010, 21:09:52 UTC

Well now,
Since Sunday last, I have been unable to upload any "Completes Tasks".
"Messages":
1/5/2010 2:53:30 PM SETI@home update requested by user
1/5/2010 2:53:32 PM SETI@home Sending scheduler request: Requested by user.
1/5/2010 2:53:32 PM SETI@home Not reporting or requesting tasks
1/5/2010 2:53:37 PM SETI@home Scheduler request completed

Close to 40 "Transfers" still reading:
1/5/2010 2:53:52 PM SETI@home Temporarily failed upload of 10fe07ab.13730.18477.6.10.116_0_0: HTTP error
1/5/2010 2:53:52 PM SETI@home Backing off 1 min 0 sec on upload of 10fe07ab.13730.18477.6.10.116_0_0

Final Msg.:
1/5/2010 2:59:20 PM Internet access OK - project servers may be temporarily down.

Anyone have any idea when things will get sorted out?

Jim
Clear Skies to all amateur Astronomers out there...
ID: 960989 · Report as offensive
Profile Pappa
Volunteer tester
Avatar

Send message
Joined: 9 Jan 00
Posts: 2562
Credit: 12,301,681
RAC: 0
United States
Message 960995 - Posted: 5 Jan 2010, 21:13:57 UTC

While nothing has been stated, there were two clogs and both ahve been released.

Uploads will be iffy for a while, as Downloads are happening.

All we can do is let Boinc and the Seti Servers do their Job.

Thank You for Your Patience

Regards



Please consider a Donation to the Seti Project.

ID: 960995 · Report as offensive
Profile JimHilty2
Avatar

Send message
Joined: 30 Apr 03
Posts: 75
Credit: 7,199,464
RAC: 0
Germany
Message 961006 - Posted: 5 Jan 2010, 21:49:03 UTC - in response to Message 960995.  

Looking good from here. Uploads and downloads all working and even secondary server back so we can see results- Woo Hoo lol

Jim

ID: 961006 · Report as offensive
Profile Pappa
Volunteer tester
Avatar

Send message
Joined: 9 Jan 00
Posts: 2562
Credit: 12,301,681
RAC: 0
United States
Message 961060 - Posted: 6 Jan 2010, 1:25:05 UTC
Last modified: 6 Jan 2010, 1:25:56 UTC

A quick look at Cricket, Uploads are over 20meg/sec Downloads as fast as they can be split. Recovery is well underway (Matt threw a few images on the "barby").

Reminder:
Depending on how old a Boinc Version You are using... YOU May need to safely shutdown Boinc and Restart it as it loses track of DNS (and other things during an extended outage). Then uploads will continue to fail and you will get no new work.

Depending on the number of uploads you have you need to get "under" a certain threshold, then Seti will automatically request new work. Until it reaches that point, it does not matter how many time you "manually" press Update.

WE have been down this road before.

Regards
Please consider a Donation to the Seti Project.

ID: 961060 · Report as offensive

Message boards : Number crunching : Power Outage Recovery - Information


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.