Panic Mode On (58) Server problems?

Message boards : Number crunching : Panic Mode On (58) Server problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · 10 · 11 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 1160523 - Posted: 9 Oct 2011, 9:10:41 UTC - in response to Message 1160512.  

Even though the server status page shows the splitters as running, it also shows the creation rate for MB and AP at zero. And ready to send is dropping on both.

MB has just run out, AP is half way there.
Until the transitioners start up again, everything else is blocked up.
Grant
Darwin NT
ID: 1160523 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1160526 - Posted: 9 Oct 2011, 9:20:50 UTC - in response to Message 1160523.  

Even though the server status page shows the splitters as running, it also shows the creation rate for MB and AP at zero. And ready to send is dropping on both.

MB has just run out, AP is half way there.
Until the transitioners start up again, everything else is blocked up.

Probably toast until Monday......
The kitty kibble bowls should be OK until then.
For now,
Good night, and good luck.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1160526 · Report as offensive
Dave Stegner
Volunteer tester
Avatar

Send message
Joined: 20 Oct 04
Posts: 540
Credit: 65,583,328
RAC: 27
United States
Message 1160528 - Posted: 9 Oct 2011, 9:29:02 UTC

Hope the little yellow guy remembered his parachute.


Dave

ID: 1160528 · Report as offensive
Profile S@NL Etienne Dokkum
Volunteer tester
Avatar

Send message
Joined: 11 Jun 99
Posts: 212
Credit: 43,822,095
RAC: 0
Netherlands
Message 1160546 - Posted: 9 Oct 2011, 12:44:50 UTC
Last modified: 9 Oct 2011, 12:47:16 UTC

Servers are dead again... no smooth running as hoped for this weekend. MB's also not being validated anymore.
ID: 1160546 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22528
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1160550 - Posted: 9 Oct 2011, 12:52:36 UTC

If I get hold of that little yellow guy he'll need more than a parachute.....



Anyway it looks as if we've found the culprit - Step forward Sten-Arne and take a bow, you now hold the honours.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1160550 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1160556 - Posted: 9 Oct 2011, 13:10:50 UTC
Last modified: 9 Oct 2011, 13:26:19 UTC

It's the Transitioners which create Result records in the BOINC database, which in turn show as "Results ready to send", etc. The splitters produce WU files and records and may be continuing to do so, we can't tell.

Possibly if one of the staff can get the Transitioners going again there will be huge values for the "Current result creation rate" for awhile.

Edit: Actually we can tell that there are new WUs, in fact there have been over 600000 WUs created which do not yet have any Result records attached. See http://setiathome.berkeley.edu/workunit.php?wuid=835683000 for one created just a few minutes ago.
                                                                  Joe
ID: 1160556 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1160587 - Posted: 9 Oct 2011, 14:58:25 UTC

It looks like there are another ~144000 WUs being created per hour. I suppose it's lucky that the "Results out in the field" has been kept reasonably low by the limits, that means the WU storage shouldn't overflow soon. With the "tapes" nearly all split for MB the WU creation rate should slow soon, but AP splitting will continue to fill storage for many hours.
                                                                   Joe
ID: 1160587 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14679
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1160596 - Posted: 9 Oct 2011, 15:28:49 UTC - in response to Message 1160587.  

It looks like there are another ~144000 WUs being created per hour. I suppose it's lucky that the "Results out in the field" has been kept reasonably low by the limits, that means the WU storage shouldn't overflow soon. With the "tapes" nearly all split for MB the WU creation rate should slow soon, but AP splitting will continue to fill storage for many hours.
                                                                   Joe

IIRC, the splitters have an automatic shut-off when "Results ready to send" reach preset limits. Since RRTS are going nowhere ... oops.

As Joe says, it's a good job this happened at a time when the disks aren't too full. But I do hope the tape autoloader system has failed as well.....
ID: 1160596 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1160618 - Posted: 9 Oct 2011, 16:23:54 UTC

Now if some kind soul could possibly remote drop-kick Anakin, perhaps we could be off and running again.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1160618 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1160631 - Posted: 9 Oct 2011, 17:10:06 UTC - in response to Message 1160596.  

IIRC, the splitters have an automatic shut-off when "Results ready to send" reach preset limits. Since RRTS are going nowhere ... oops.

As Joe says, it's a good job this happened at a time when the disks aren't too full. But I do hope the tape autoloader system has failed as well.....

The question is though.. what process decides when there are enough RRTS? Is it the splitters and do they know how many there are, or is it a database query that governs it? The latter would keep the splitters going full-speed since RRTS is shown as zero.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1160631 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1160636 - Posted: 9 Oct 2011, 17:23:58 UTC

Somebody's pushing the buttons.....
Looks like they started a transitioner process on Synergy and things are moving a bit again.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1160636 · Report as offensive
AndyJ
Avatar

Send message
Joined: 17 Aug 02
Posts: 248
Credit: 27,380,797
RAC: 0
United Kingdom
Message 1160646 - Posted: 9 Oct 2011, 17:50:25 UTC - in response to Message 1160618.  
Last modified: 9 Oct 2011, 18:07:21 UTC

Now if some kind soul could possibly remote drop-kick Anakin, perhaps we could be off and running again.


Somebody did just that.
Thanks for putting in the time, and on a Sunday!

Regards,

Andy
ID: 1160646 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14679
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1160659 - Posted: 9 Oct 2011, 18:08:49 UTC - in response to Message 1160631.  

IIRC, the splitters have an automatic shut-off when "Results ready to send" reach preset limits. Since RRTS are going nowhere ... oops.

As Joe says, it's a good job this happened at a time when the disks aren't too full. But I do hope the tape autoloader system has failed as well.....

The question is though.. what process decides when there are enough RRTS? Is it the splitters and do they know how many there are, or is it a database query that governs it? The latter would keep the splitters going full-speed since RRTS is shown as zero.

I think we have our answer:

Results ready to send 1,740,609

RRTS tells the splitters 'enough, already'.

Zero != enough
ID: 1160659 · Report as offensive
MikeN

Send message
Joined: 24 Jan 11
Posts: 319
Credit: 64,719,409
RAC: 85
United Kingdom
Message 1160661 - Posted: 9 Oct 2011, 18:11:43 UTC
Last modified: 9 Oct 2011, 18:13:01 UTC

Jeff Cobb just posted this in the down loading new work units thread

The server anakin crashed. So I built a 64 bit version of the transitioner and deployed in on synergy. Anakin is also a download server (the other being bane) via DNS. I do not want to restart the lab wide DNS server remotely on a Sunday, so I hope this will sort itself out until someone can restart anakin (I am not back to work until Tuesday). DL traffic is maxed out in any case.


thanks for the info Jeff and for your efforts.
ID: 1160661 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 1160682 - Posted: 9 Oct 2011, 18:54:41 UTC - in response to Message 1160661.  


All i'm getting with my work requests are "Project has no tasks available". Hopefully it'll settle down over the next couple of hours.
Grant
Darwin NT
ID: 1160682 · Report as offensive
MikeN

Send message
Joined: 24 Jan 11
Posts: 319
Credit: 64,719,409
RAC: 85
United Kingdom
Message 1160693 - Posted: 9 Oct 2011, 19:23:42 UTC - in response to Message 1160682.  
Last modified: 9 Oct 2011, 19:50:20 UTC


All i'm getting with my work requests are "Project has no tasks available". Hopefully it'll settle down over the next couple of hours.


Curious since cricket graph is active but not maxed out and there are plenty of results available (both MB and AP), but the numbers available are decreasing suggesting that they are being sent somewhere.
ID: 1160693 · Report as offensive
AndyJ
Avatar

Send message
Joined: 17 Aug 02
Posts: 248
Credit: 27,380,797
RAC: 0
United Kingdom
Message 1160702 - Posted: 9 Oct 2011, 19:56:50 UTC - in response to Message 1160682.  
Last modified: 9 Oct 2011, 20:12:41 UTC


All i'm getting with my work requests are "Project has no tasks available". Hopefully it'll settle down over the next couple of hours.



Same here.

Edit: Uploads going ok, no downloads for a while, then suddenly I have reached the task limit? With no downloads? What the fudge? Have they lowered the cap?
Edit 2: Ok, it just gave me 13 new ones, strange behaviour though, somethings up.

Regards,

Andy
ID: 1160702 · Report as offensive
Profile Frizz
Volunteer tester
Avatar

Send message
Joined: 17 May 99
Posts: 271
Credit: 5,852,934
RAC: 0
New Zealand
Message 1160730 - Posted: 9 Oct 2011, 21:28:01 UTC

Server status page says:
Data Distribution State SETI@home # Astropulse # As of*
Results ready to send 1,581,902 25,027 0m

My BOINC client says:
10/9/2011 11:22:08 PM SETI@home [wfd] request: 1728001.73 sec CPU (0.00 sec, 0.00) ATI GPU (1728001.73 sec, 2.00)
10/9/2011 11:22:08 PM SETI@home Sending scheduler request: To fetch work.
10/9/2011 11:22:08 PM SETI@home Requesting new tasks for GPU
10/9/2011 11:22:11 PM SETI@home Scheduler request completed: got 0 new tasks
10/9/2011 11:22:11 PM SETI@home Message from server: Project has no tasks available


?????
ID: 1160730 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34379
Credit: 79,922,639
RAC: 80
Germany
Message 1160731 - Posted: 9 Oct 2011, 21:30:21 UTC
Last modified: 9 Oct 2011, 21:31:00 UTC

I got a few so it seems to work.
But the numbers are a bit out of hand.


With each crime and every kindness we birth our future.
ID: 1160731 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1160732 - Posted: 9 Oct 2011, 21:37:01 UTC - in response to Message 1160730.  
Last modified: 9 Oct 2011, 21:37:59 UTC

Server status page says:
Data Distribution State SETI@home # Astropulse # As of*
Results ready to send 1,581,902 25,027 0m

My BOINC client says:
10/9/2011 11:22:08 PM SETI@home [wfd] request: 1728001.73 sec CPU (0.00 sec, 0.00) ATI GPU (1728001.73 sec, 2.00)
10/9/2011 11:22:08 PM SETI@home Sending scheduler request: To fetch work.
10/9/2011 11:22:08 PM SETI@home Requesting new tasks for GPU
10/9/2011 11:22:11 PM SETI@home Scheduler request completed: got 0 new tasks
10/9/2011 11:22:11 PM SETI@home Message from server: Project has no tasks available


?????

That message means the feeder has no tasks available to send at the instant you asked, a moment later the feeder will have refilled it's One hundred slots,
and there will be work available for the next host to get,

Claggy
ID: 1160732 · Report as offensive
Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · 10 · 11 · Next

Message boards : Number crunching : Panic Mode On (58) Server problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.