Panic Mode On (115) Server Problems?

Message boards : Number crunching : Panic Mode On (115) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 31 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982365 - Posted: 27 Feb 2019, 0:55:16 UTC - in response to Message 1982362.  

The setting in under options in cc_config.xml.
<max_tasks_reported>100</max_tasks_reported>
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982365 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1982366 - Posted: 27 Feb 2019, 0:57:38 UTC - in response to Message 1982362.  

If I try to report more than 100 tasks a time, I will get the comms error and a backoff. 100 seems to be my magic number. I'm 95% successful every time.
. . which file is that setting in again? I am sure mine is still over 100 (might be 128) but until now I could not see anyone stating a number which was consistently working ...
Stephen
cc_config
<max_tasks_reported>100</max_tasks_reported>
ID: 1982366 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1982368 - Posted: 27 Feb 2019, 1:05:13 UTC - in response to Message 1982365.  
Last modified: 27 Feb 2019, 1:33:23 UTC

The setting in under options in cc_config.xml.
<max_tasks_reported>100</max_tasks_reported>


. . Well there you have it, mine is now set to 99 and yet still gets 'http internal server error'

. . OK, I set it on the Linux box as well and there EVERY report attempt was successful. I will try another value on the Windows box.

. . I dropped it to 69 but still getting the http internal server error. It seems there is something else at play on this box.

Stephen

<shrug>
ID: 1982368 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982369 - Posted: 27 Feb 2019, 1:09:11 UTC - in response to Message 1982365.  

The setting in under options in cc_config.xml.
<max_tasks_reported>100</max_tasks_reported>


ive tried. still no dice. nothing but http internal server error
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982369 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982372 - Posted: 27 Feb 2019, 1:29:38 UTC

All reported now. Not getting any cpu work. Still crunching through my gpu rescheduled tasks.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982372 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1982373 - Posted: 27 Feb 2019, 1:35:19 UTC - in response to Message 1982372.  

All reported now. Not getting any cpu work. Still crunching through my gpu rescheduled tasks.


. . Now only one box out of 4 is not reporting work so overall things are kind of OK. But this box will be OOW soon if things don't improve.

Stephen

:(
ID: 1982373 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982385 - Posted: 27 Feb 2019, 2:44:51 UTC - in response to Message 1982373.  
Last modified: 27 Feb 2019, 2:45:00 UTC

Downloads are starting but all are hanging up in backoff.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982385 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 1982393 - Posted: 27 Feb 2019, 3:47:05 UTC - in response to Message 1982274.  
Last modified: 27 Feb 2019, 3:49:28 UTC

I was getting worried that they were all stuck in a cluster-f over 05jl06ef, but they seemed to have worked themselves free again.


Look at the size of the file
05jl06ef 272.67 GB

It might be weird to split that one considering the size. usually the files are around 50GB.

Must have been removed. I went to have a look at it but would not see it. It would have been impressive to see it munch through a file that big

Yeah, it finally wrapped up a bit ago. I think all told it took over 24 hours to fully process for both MB and AP.

In passive I wish I had of seen it. I will keep a close eye on the MB work from now on. I hope it wasn't a one off
ID: 1982393 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982395 - Posted: 27 Feb 2019, 3:56:14 UTC

Stuck downloads finally cleared. The logjam is over.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982395 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1982411 - Posted: 27 Feb 2019, 6:18:43 UTC - in response to Message 1982385.  

Downloads are starting but all are hanging up in backoff.

Yeah.
Got home to find work allocated, but both systems in back-off forever mode due to downloads not happening. Retried pending transfers & now things are working once more (at least until the Ready-to-send buffer is empty again anyway).
Grant
Darwin NT
ID: 1982411 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1982417 - Posted: 27 Feb 2019, 6:25:12 UTC - in response to Message 1982411.  

(at least until the Ready-to-send buffer is empty again anyway).

Which may not take too long as I just had 4 noise bombs.
Grant
Darwin NT
ID: 1982417 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1982419 - Posted: 27 Feb 2019, 6:29:27 UTC - in response to Message 1982373.  
Last modified: 27 Feb 2019, 6:33:55 UTC

All reported now. Not getting any cpu work. Still crunching through my gpu rescheduled tasks.

. . Now only one box out of 4 is not reporting work so overall things are kind of OK. But this box will be OOW soon if things don't improve.
Stephen
:(

. . Update, that machine was finally able to report and had a wonderful experience. Without any prompting on my part three consecutive work reports/requests resulted in ghost resends. Within 15 mins the machine has recovered 60 ghosts when it needed them most. Now only 20 outstanding.

. . I also noticed some new tapes have been mounted for splitting, and UnixChick they are still for the same day/night. But a new variety ... blc36's. These will be very slow (relatively speaking).

Stephen

:)
ID: 1982419 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1982421 - Posted: 27 Feb 2019, 6:40:15 UTC

Amazing how one day of data collection can give us enough files to work on for months... I can't remember when we started working on this day of data. It looks like we have at least another 2 or 3 weeks worth of data now.

Thanks Stephen for recovering the ghosts, I'm glad the recovery is going well.
ID: 1982421 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1982423 - Posted: 27 Feb 2019, 6:48:00 UTC - in response to Message 1982421.  

Amazing how one day of data collection can give us enough files to work on for months... I can't remember when we started working on this day of data. It looks like we have at least another 2 or 3 weeks worth of data now.

Thanks Stephen for recovering the ghosts, I'm glad the recovery is going well.


. . I am stickler for ghost recovery, it is only fair to your wingmen and not that hard to do. It's also better for the project generally as it prevents further database bloat.

. . In fairness though, we have processed quite a few Arecibo tapes while we have been gnawing on this particular GBT bone :)

Stephen

:)
ID: 1982423 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982502 - Posted: 27 Feb 2019, 21:21:40 UTC

Unable to contact servers to report work without timeouts.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982502 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982505 - Posted: 27 Feb 2019, 21:32:49 UTC - in response to Message 1982502.  

i was able to report some tasks shortly after it came back, but looks like its down again. no COMMs.

no biggie, i'm sure it'll come back eventually.

someone somewhere posted that this would be normal now right? a second outage on Wednesdays? i could be misremebering.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982505 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982508 - Posted: 27 Feb 2019, 21:41:58 UTC

I wonder if they took the site down to patch a website software vulnerability.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982508 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11358
Credit: 29,581,041
RAC: 66
United States
Message 1982512 - Posted: 27 Feb 2019, 21:50:35 UTC - in response to Message 1982502.  

Unable to contact servers to report work without timeouts.

Yep
ID: 1982512 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1982518 - Posted: 27 Feb 2019, 22:20:08 UTC - in response to Message 1982505.  
Last modified: 27 Feb 2019, 22:36:40 UTC

i was able to report some tasks shortly after it came back, but looks like its down again. no COMMs.

no biggie, i'm sure it'll come back eventually.

someone somewhere posted that this would be normal now right? a second outage on Wednesdays? i could be misremebering.


. . Well if nothing else this project has proven the existence of black holes ... it keeps falling into them :)

{edit}
. . OK, this machine was finally able to report the 69 task limit I set it to but now it is 'no tasks available'. Very typical post outage behaviour ...

Stephen

<shrug>
ID: 1982518 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 1982522 - Posted: 27 Feb 2019, 22:53:26 UTC

Slowly coming back to life.
ID: 1982522 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 31 · Next

Message boards : Number crunching : Panic Mode On (115) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.