Panic Mode On (14) Server problems

Message boards : Number crunching : Panic Mode On (14) Server problems
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 10 · Next

AuthorMessage
Fred W
Volunteer tester

Send message
Joined: 13 Jun 99
Posts: 2524
Credit: 11,954,210
RAC: 0
United Kingdom
Message 882603 - Posted: 5 Apr 2009, 20:58:28 UTC - in response to Message 882602.  


Looks like the storm is over. :D

Now, how else can you provoke Murphy?? :P

F.
ID: 882603 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 882612 - Posted: 5 Apr 2009, 21:36:20 UTC - in response to Message 882603.  


Looks like the storm is over. :D

Now, how else can you provoke Murphy?? :P

F.

I don't know. I think we've provoked Mr. Murphy in every way possible, so by that logic, we can't provoke him any more. :p
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 882612 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 944
Credit: 52,956,491
RAC: 67
United Kingdom
Message 882727 - Posted: 6 Apr 2009, 13:05:40 UTC
Last modified: 6 Apr 2009, 13:56:22 UTC

Sudden crash indicated by Cricket graphs? Good job they'll be back noses to grindstone soon.
[edit]No, all perfectly OK again[/edit]

ID: 882727 · Report as offensive
Andy Williams
Volunteer tester
Avatar

Send message
Joined: 11 May 01
Posts: 187
Credit: 112,464,820
RAC: 0
United States
Message 884090 - Posted: 10 Apr 2009, 23:06:23 UTC

Is it known and anticipated that the AP splitters would be down as of Friday afternoon?
--
Classic 82353 WU / 400979 h
ID: 884090 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13755
Credit: 208,696,464
RAC: 304
Australia
Message 884095 - Posted: 10 Apr 2009, 23:25:55 UTC - in response to Message 884090.  
Last modified: 10 Apr 2009, 23:26:23 UTC

Is it known and anticipated that the AP splitters would be down as of Friday afternoon?

What to believe, what to believe.
Ststus page also shows MB result creation rate is around 21/s. Unlikely if the splitters weren't running. So something somewhere isn't working quite right.

I have been getting more than the usual number of "No work available" messages when trying for work, but after the 5th or 7th attempt it finally gets allocated & downloads. Also notived the Validation & Assimilation queues have been growing steadily, although thy've just peaked & are now starting to decline.
Grant
Darwin NT
ID: 884095 · Report as offensive
Andy Williams
Volunteer tester
Avatar

Send message
Joined: 11 May 01
Posts: 187
Credit: 112,464,820
RAC: 0
United States
Message 884096 - Posted: 10 Apr 2009, 23:32:58 UTC - in response to Message 884095.  

Check the Astropulse graphs.
--
Classic 82353 WU / 400979 h
ID: 884096 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13755
Credit: 208,696,464
RAC: 304
Australia
Message 884098 - Posted: 10 Apr 2009, 23:37:59 UTC - in response to Message 884096.  

Check the Astropulse graphs.

Sorry, reading AP & thinking MB.

The AP graphs show that the Ready to Send buffer is way overfull. Usually it's about 4,500-4,700. At present it's about 13,000. I expect once the buffer drains to a more normal level (and storage space is freed up again) the AP splitters will kick in again.
Grant
Darwin NT
ID: 884098 · Report as offensive
Andy Williams
Volunteer tester
Avatar

Send message
Joined: 11 May 01
Posts: 187
Credit: 112,464,820
RAC: 0
United States
Message 884105 - Posted: 10 Apr 2009, 23:57:24 UTC - in response to Message 884098.  

The AP splitters have not been running for several hours. I suspect something other than disk space issues.
--
Classic 82353 WU / 400979 h
ID: 884105 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13755
Credit: 208,696,464
RAC: 304
Australia
Message 884111 - Posted: 11 Apr 2009, 0:08:15 UTC - in response to Message 884105.  

The AP splitters have not been running for several hours. I suspect something other than disk space issues.

Yep.
The threshold setting for producing more work. Until the Ready to Send buffer gets below 4,500 i wouldn't expect the splitters to start up again. If it gets below 2,000 & they don't start up, then you can start panicing.
Grant
Darwin NT
ID: 884111 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 884185 - Posted: 11 Apr 2009, 4:33:33 UTC - in response to Message 884111.  

The AP splitters have not been running for several hours. I suspect something other than disk space issues.

Yep.
The threshold setting for producing more work. Until the Ready to Send buffer gets below 4,500 i wouldn't expect the splitters to start up again. If it gets below 2,000 & they don't start up, then you can start panicing.

I believe you're partly right, but the AP splitters had been set to stop producing work when the "Ready to send" queue got to about 2500, see the 30 day Scarecrow graphs. Earlier today "Ready to send" reached about 20000 indicating the automatic limiting not working. I think someone had to just shut them down remotely, or maybe there's a last resort safety script which killed them.

When automatic limiting is working the status of the ap_splitter processes doesn't go to "Not running". I think it takes human intervention to start them again, if the queue is below 1200 or so tomorrow perhaps someone will do so.
                                                                Joe
ID: 884185 · Report as offensive
HarryM
Volunteer tester

Send message
Joined: 24 Jul 08
Posts: 68
Credit: 3,812,695
RAC: 0
United States
Message 884228 - Posted: 11 Apr 2009, 11:45:51 UTC

Their back up running now.
ID: 884228 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 884447 - Posted: 12 Apr 2009, 2:06:03 UTC - in response to Message 881366.  




Ehere can I the above graph in real time?
ID: 884447 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 884464 - Posted: 12 Apr 2009, 3:18:14 UTC - in response to Message 884447.  

Ehere can I the above graph in real time?

Copy the link out of the message you quoted, or use the standard page which contains the graph: http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=UcastPackets;ranges=d.
                                                                Joe
ID: 884464 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 884466 - Posted: 12 Apr 2009, 3:26:20 UTC - in response to Message 884464.  

Ehere can I the above graph in real time?

Copy the link out of the message you quoted, or use the standard page which contains the graph: http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=UcastPackets;ranges=d.
                                                                Joe

Yeah, either unicast packets/sec (above) or octets/sec can be used. There's different preferences for different people. I find octets to be a better measure of bandwidth consumption, seeing how packets can range in their size, but octets (bytes) are always the same.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 884466 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 884513 - Posted: 12 Apr 2009, 9:34:47 UTC - in response to Message 884464.  
Last modified: 12 Apr 2009, 9:36:27 UTC

Ehere can I the above graph in real time?

Copy the link out of the message you quoted, or use the standard page which contains the graph: http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=UcastPackets;ranges=d.

Thanks for that Joe & Cosmic_Ocean
ID: 884513 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 884587 - Posted: 12 Apr 2009, 15:27:22 UTC

I have the octet page bookmarked on my machines.

ID: 884587 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65794
Credit: 55,293,173
RAC: 49
United States
Message 884723 - Posted: 12 Apr 2009, 21:28:41 UTC
Last modified: 12 Apr 2009, 21:43:06 UTC

Is It My imagination or is the server taking a lot of time to refresh and to update?

Someone needs to get out and push, As output and input seem to have almost dropped to nothing.
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 884723 · Report as offensive
Profile White Mountain Wes
Avatar

Send message
Joined: 24 Jul 08
Posts: 259
Credit: 6,607,678
RAC: 4
United States
Message 884726 - Posted: 12 Apr 2009, 21:41:55 UTC - in response to Message 884723.  

Is It My imagination or is the server taking time to refresh and to update?


Yes... it's... running... very... slowly.
ID: 884726 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 884727 - Posted: 12 Apr 2009, 21:42:45 UTC

It could be your imagination I'm not sure. Here is the traffic graph
[img] http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=Octets;ranges=d[/img] Work is flowing
ID: 884727 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13755
Credit: 208,696,464
RAC: 304
Australia
Message 884728 - Posted: 12 Apr 2009, 21:47:24 UTC - in response to Message 884723.  

Is It My imagination or is the server taking a lot of time to refresh and to update?
Something needs to get out and push, As output and input seem to have almost dropped to nothing.

Possibly. There was a similar dive in the traffic at the same time yesterday. And at the moment the forums are slower than a month of wet Sundays.
Grant
Darwin NT
ID: 884728 · Report as offensive
Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 10 · Next

Message boards : Number crunching : Panic Mode On (14) Server problems


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.