Panic Mode On (112) Server Problems?

Message boards : Number crunching : Panic Mode On (112) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 14 · 15 · 16 · 17 · 18 · 19 · 20 . . . 33 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1937302 - Posted: 26 May 2018, 20:44:49 UTC

Errored file has gone I see, splitter output still borked though.
However I just got a "Projects is temporarily shut down for maintenance" on my last Scheduler request.
*fingers crossed*
Grant
Darwin NT
ID: 1937302 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1937304 - Posted: 26 May 2018, 20:49:37 UTC - in response to Message 1937302.  

Errored file has gone I see, splitter output still borked though.
However I just got a "Projects is temporarily shut down for maintenance" on my last Scheduler request.
*fingers crossed*

I did too.
Manually updated and it went through OK.
That's the second time I had that today.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1937304 · Report as offensive
Michael Donikowski
Volunteer tester

Send message
Joined: 29 May 99
Posts: 8
Credit: 22,914,826
RAC: 38
United States
Message 1937305 - Posted: 26 May 2018, 20:55:44 UTC - in response to Message 1937226.  

Tuesday. Monday is a US holiday.
ID: 1937305 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1937307 - Posted: 26 May 2018, 21:02:22 UTC
Last modified: 26 May 2018, 21:04:34 UTC

Yuppers, 3 day weekend, and the 'official' beginning of summer! Bring it on, it's been a long time a coming... Had about 18 inches of snow fall around a month or so ago, and it felt that winter would never end. Ugh.

*edit* of course, for about a week stretch now, we're hovering between 85 and 95 for highs, thru the middle of next week, and normal highs for this time of year are in the low 70's. You won't find me complaining though, no way. I almost never complain about summer heat, because I know what is just 4-5 months away... :-/

ID: 1937307 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1937310 - Posted: 26 May 2018, 21:12:48 UTC - in response to Message 1937307.  

You got the same weather I got, Al.
Not really complaining either, although I did shut 4 of the older crunchers down for a few days so my window AC unit can keep up enough to save my kitties from melting.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1937310 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1937311 - Posted: 26 May 2018, 21:13:43 UTC - in response to Message 1937305.  

Tuesday. Monday is a US holiday.

The I hope today's remote fiddling can get the splitters working again.

Just did a manual update, it went through but no work is available.
Grant
Darwin NT
ID: 1937311 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1937312 - Posted: 26 May 2018, 21:16:23 UTC - in response to Message 1937311.  

Tuesday. Monday is a US holiday.

The I hope today's remote fiddling can get the splitters working again.

Just did a manual update, it went through but no work is available.

Eric's already done some poking via remote.
I kinda think we are stuck with the current situation until Tuesday.
And I would be prepared for a long outage on Tuesday as well.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1937312 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1937314 - Posted: 26 May 2018, 21:35:48 UTC - in response to Message 1937312.  
Last modified: 26 May 2018, 21:36:18 UTC

Tuesday. Monday is a US holiday.

The I hope today's remote fiddling can get the splitters working again.

Just did a manual update, it went through but no work is available.

Eric's already done some poking via remote.
I kinda think we are stuck with the current situation until Tuesday.
And I would be prepared for a long outage on Tuesday as well.

Oh well.
A chance for the machines to have a rest.

What ever the problem is, it occurred with/after last weeks outage.
BTW- did you let him know about the file that completely errored out?

blc12_2bit_blc12_guppi_58157_16456_DIAG_PSR_J1024-0719_0006
Grant
Darwin NT
ID: 1937314 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1937316 - Posted: 26 May 2018, 21:37:19 UTC - in response to Message 1937314.  


BTW- did you let him know about the file that completely errored out?

blc12_2bit_blc12_guppi_58157_16456_DIAG_PSR_J1024-0719_0006

No, I didn't.
But, I'm not the only canary in the coal mine.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1937316 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1937317 - Posted: 26 May 2018, 21:38:05 UTC - in response to Message 1937316.  


BTW- did you let him know about the file that completely errored out?

blc12_2bit_blc12_guppi_58157_16456_DIAG_PSR_J1024-0719_0006

No, I didn't.
But, I'm not the only canary in the coal mine.

No worries.
Grant
Darwin NT
ID: 1937317 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1937320 - Posted: 26 May 2018, 22:06:24 UTC

Damn the splitters. Been out of gpu work for over an hour now on the 4 card system.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1937320 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1937325 - Posted: 26 May 2018, 23:05:37 UTC - in response to Message 1937320.  

Damn the splitters. Been out of gpu work for over an hour now on the 4 card system.

I just checked your 4 card system and it had 202 tasks in progress then I rechecked it and it had 225 in progress
ID: 1937325 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1937349 - Posted: 27 May 2018, 1:42:09 UTC - in response to Message 1937325.  

Damn the splitters. Been out of gpu work for over an hour now on the 4 card system.

I just checked your 4 card system and it had 202 tasks in progress then I rechecked it and it had 225 in progress

At the time I posted I had zero gpu tasks. 100 cpu tasks. The gpus were idle. Of course posting about it got me 19 gpu tasks but they were gone in ten minutes. Then back to no tasks upon request.

As I type I have 59 cpu tasks and 175 gpu tasks and no tasks received in the last three requests. I should have 400 gpu tasks in the cache at all times.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1937349 · Report as offensive
Kevin Olley

Send message
Joined: 3 Aug 99
Posts: 906
Credit: 261,085,289
RAC: 572
United Kingdom
Message 1937391 - Posted: 27 May 2018, 15:19:37 UTC - in response to Message 1937349.  

[quote
At the time I posted I had zero gpu tasks. 100 cpu tasks. [/quote]

Its the other way round with the Threadripper, 24 cpu wu's at a time makes short work of 100 wu limit.

What makes it worse is when you are getting a few trickle through they all go to the GPU's.
Kevin


ID: 1937391 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1937403 - Posted: 27 May 2018, 16:41:41 UTC - in response to Message 1937391.  
Last modified: 27 May 2018, 16:42:32 UTC

Yes, I have heard that refrain from RueiKe who has had at TR running the project for quite a while now.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1937403 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1937407 - Posted: 27 May 2018, 17:14:23 UTC

100 WU limit penalizes anyone who has a lot of cores CPU (Intel or AMD) and/or fast GPU's.
ID: 1937407 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1937458 - Posted: 28 May 2018, 5:14:55 UTC

Well, the Ready-to -send buffer has refilled, but how much it is due to the GBT splitters cranking up the output, or the introduction of a good amount of new Arecibo work (MB & AP), remains to be seen.
Grant
Darwin NT
ID: 1937458 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 3187
Credit: 57,163,290
RAC: 0
United States
Message 1937499 - Posted: 28 May 2018, 16:47:38 UTC - in response to Message 1937458.  

Well, the Ready-to -send buffer has refilled, but how much it is due to the GBT splitters cranking up the output, or the introduction of a good amount of new Arecibo work (MB & AP), remains to be seen.


I think it was the Green Bank splitters (at least one of them...) that FUBARed us in the first place...

Seems to be OK now (9:45 AM Berkeley time, May 28...) but that could change... the SETI crew really needs to work on getting the GBT splitters to each work on a separate file, not all try to access the same one!
.

Hello, from Albany, CA!...
ID: 1937499 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1937572 - Posted: 29 May 2018, 5:01:41 UTC

Haveland graphs have flatlined, and the Server Status page no longer has anything there.
Grant
Darwin NT
ID: 1937572 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1937574 - Posted: 29 May 2018, 5:08:47 UTC - in response to Message 1937572.  

The server page has been going blank for a while. The last time I got it to load I saw this weirdness.

Transitioner backlog (hours) 0.0005555555552018:05:28:22:00 21m

I also see that the db purge is not running.

This is at As of 29 May 2018, 5:00:03 UTC and I thought I'd post this in case it helps diagnostically to figure out what is going on.
ID: 1937574 · Report as offensive
Previous · 1 . . . 14 · 15 · 16 · 17 · 18 · 19 · 20 . . . 33 · Next

Message boards : Number crunching : Panic Mode On (112) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.