Panic Mode On (14) Server problems

Message boards : Number crunching : Panic Mode On (14) Server problems
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 9 · Next

AuthorMessage
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 878951 - Posted: 24 Mar 2009, 22:52:25 UTC


Ops.. unplanned outage after the weekly outage?



ID: 878951 · Report as offensive
Profile Blurf
Volunteer tester

Send message
Joined: 2 Sep 06
Posts: 8964
Credit: 12,678,685
RAC: 0
United States
Message 878961 - Posted: 24 Mar 2009, 23:06:58 UTC

Might want to read THIS.


ID: 878961 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 881366 - Posted: 1 Apr 2009, 18:51:34 UTC
Last modified: 1 Apr 2009, 18:54:40 UTC


Woohoo.. the train is on a straight track? ;-D




BTW.
Someone know when we could get new WUs?

ID: 881366 · Report as offensive
Profile DPRGI - Luivul

Send message
Joined: 24 Jan 03
Posts: 17
Credit: 20,639,801
RAC: 0
Italy
Message 881654 - Posted: 2 Apr 2009, 13:29:22 UTC

I get some WU but can't upload? I see the cricket graph and see abnormal traffic :( . What happend?
ID: 881654 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 881679 - Posted: 2 Apr 2009, 14:57:11 UTC - in response to Message 881654.  

I get some WU but can't upload? I see the cricket graph and see abnormal traffic :( . What happend?

The flood gates opened. Matt turned all the MB splitters on yesterday, so there is much more work available to download, and thousands of clients world-wide ended up draining their cache, and are trying to refill it.

Once the bandwidth is no longer maxed out (as shown by the cricket graph, or that picture above..which is static), uploads should flow just fine.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 881679 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 881737 - Posted: 2 Apr 2009, 18:08:21 UTC


No, the graph I posted isn't static.. it's 'real time, live' or what ever.. ;-)

ID: 881737 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 881752 - Posted: 2 Apr 2009, 18:35:36 UTC - in response to Message 881737.  
Last modified: 2 Apr 2009, 18:37:10 UTC


No, the graph I posted isn't static.. it's 'real time, live' or what ever.. ;-)

Alright. I'm used to static pictures being posted. Packets/sec is a strange way of looking at it though. I have octets (bytes)/sec bookmarked (link here).

Oh, and my official submission for panic at the moment is.. I have 10 WUs that won't upload! One of them has an hour and a half of upload time on it. I know there's the backoff mechanism that they run through, but every 20-30 minutes I'll pick one and tell it to retry now, and that just doesn't work.

Oh well, I tried.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 881752 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 881774 - Posted: 2 Apr 2009, 19:53:22 UTC

Alright, [/panic]. They all uploaded at the same time (select all, retry now). It's too early to tell, but it looks like the bytes/sec rate has come down from the ceiling of ~93mbit to ~90mbit. That 3mbit drop in the project outbound has correlated to a 3mbit rise for inbound.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 881774 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 882096 - Posted: 3 Apr 2009, 21:56:46 UTC

Well, it appears (at 2155utc) that the database issues must be done. All four MB validators have been turned on, and I think the "waiting for assimilation" queue is getting smaller. There are several hundred thousand results/wus in the "waiting for deletion" queue.

And it must be doing something, because the splitters are creating at ~25/sec and there is a RTS queue. Of course the server status page isn't a "real time" picture, nor very accurate, but it's a general picture at least.

Yay!
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 882096 · Report as offensive
Niteryder
Volunteer tester

Send message
Joined: 1 Mar 99
Posts: 64
Credit: 22,663,988
RAC: 18
United States
Message 882294 - Posted: 4 Apr 2009, 16:47:36 UTC

Something has gone wrong. The graph has bottemed out and web pages are slow to load. Also I cannot get any mb jobs.
ID: 882294 · Report as offensive
Andy Williams
Volunteer tester
Avatar

Send message
Joined: 11 May 01
Posts: 187
Credit: 112,464,820
RAC: 0
United States
Message 882296 - Posted: 4 Apr 2009, 16:51:12 UTC - in response to Message 882294.  

It's the weekend...
--
Classic 82353 WU / 400979 h
ID: 882296 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 944
Credit: 52,956,491
RAC: 67
United Kingdom
Message 882318 - Posted: 4 Apr 2009, 18:08:46 UTC - in response to Message 882296.  

Good cache, nice cache.......

ID: 882318 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 882602 - Posted: 5 Apr 2009, 20:54:02 UTC

Looks like the assimilators are crunching away. "waiting for assimilation" is down under 1M, and the "waiting for db purge" queue is into the 7-digit range. Now just waiting for those purge queues to be processed and that should free up a TON of storage space and allow the splitters to provide a stable amount of work.

Looks like the storm is over. :D
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 882602 · Report as offensive
Fred W
Volunteer tester

Send message
Joined: 13 Jun 99
Posts: 2524
Credit: 11,954,210
RAC: 0
United Kingdom
Message 882603 - Posted: 5 Apr 2009, 20:58:28 UTC - in response to Message 882602.  


Looks like the storm is over. :D

Now, how else can you provoke Murphy?? :P

F.
ID: 882603 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 882612 - Posted: 5 Apr 2009, 21:36:20 UTC - in response to Message 882603.  


Looks like the storm is over. :D

Now, how else can you provoke Murphy?? :P

F.

I don't know. I think we've provoked Mr. Murphy in every way possible, so by that logic, we can't provoke him any more. :p
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 882612 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 944
Credit: 52,956,491
RAC: 67
United Kingdom
Message 882727 - Posted: 6 Apr 2009, 13:05:40 UTC
Last modified: 6 Apr 2009, 13:56:22 UTC

Sudden crash indicated by Cricket graphs? Good job they'll be back noses to grindstone soon.
[edit]No, all perfectly OK again[/edit]

ID: 882727 · Report as offensive
Andy Williams
Volunteer tester
Avatar

Send message
Joined: 11 May 01
Posts: 187
Credit: 112,464,820
RAC: 0
United States
Message 884090 - Posted: 10 Apr 2009, 23:06:23 UTC

Is it known and anticipated that the AP splitters would be down as of Friday afternoon?
--
Classic 82353 WU / 400979 h
ID: 884090 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13886
Credit: 208,696,464
RAC: 304
Australia
Message 884095 - Posted: 10 Apr 2009, 23:25:55 UTC - in response to Message 884090.  
Last modified: 10 Apr 2009, 23:26:23 UTC

Is it known and anticipated that the AP splitters would be down as of Friday afternoon?

What to believe, what to believe.
Ststus page also shows MB result creation rate is around 21/s. Unlikely if the splitters weren't running. So something somewhere isn't working quite right.

I have been getting more than the usual number of "No work available" messages when trying for work, but after the 5th or 7th attempt it finally gets allocated & downloads. Also notived the Validation & Assimilation queues have been growing steadily, although thy've just peaked & are now starting to decline.
Grant
Darwin NT
ID: 884095 · Report as offensive
Andy Williams
Volunteer tester
Avatar

Send message
Joined: 11 May 01
Posts: 187
Credit: 112,464,820
RAC: 0
United States
Message 884096 - Posted: 10 Apr 2009, 23:32:58 UTC - in response to Message 884095.  

Check the Astropulse graphs.
--
Classic 82353 WU / 400979 h
ID: 884096 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13886
Credit: 208,696,464
RAC: 304
Australia
Message 884098 - Posted: 10 Apr 2009, 23:37:59 UTC - in response to Message 884096.  

Check the Astropulse graphs.

Sorry, reading AP & thinking MB.

The AP graphs show that the Ready to Send buffer is way overfull. Usually it's about 4,500-4,700. At present it's about 13,000. I expect once the buffer drains to a more normal level (and storage space is freed up again) the AP splitters will kick in again.
Grant
Darwin NT
ID: 884098 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 9 · Next

Message boards : Number crunching : Panic Mode On (14) Server problems


 
©2025 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.