The Server Issues / Outages Thread - Panic Mode On! (117)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 39 · 40 · 41 · 42 · 43 · 44 · 45 . . . 52 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13154
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2023525 - Posted: 18 Dec 2019, 2:32:06 UTC

I still have not had any luck in contacting the servers.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2023525 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 2023526 - Posted: 18 Dec 2019, 2:42:15 UTC

All work now completed and reported. No difficulties establishing Scheduler connections, but consistently "Project has no tasks available" even though SSP shows (over)full RTS, database with low load, and backup db caught up.
So, in other words, about how it was 15 hours ago during the pre-outage crash.
ID: 2023526 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 813
Credit: 2,361,516
RAC: 22
United States
Message 2023527 - Posted: 18 Dec 2019, 2:53:20 UTC

I managed to report my completed WUs and I am not requesting any new WUs (I'm good for another day) out of the kindness of my heart as I know my slow machine always manages to get WUs before your fast ones (why?). Good luck and may the WUs flow soon.
ID: 2023527 · Report as offensive
halfempty
Avatar

Send message
Joined: 2 Jun 99
Posts: 97
Credit: 35,236,901
RAC: 114
United States
Message 2023530 - Posted: 18 Dec 2019, 3:02:58 UTC

Had a hard time reporting tasks until I set max_tasks_reported to 64, before that change the connection would just time out. Now I get good connections but the dreaded Project has no tasks available.
ID: 2023530 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13154
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2023531 - Posted: 18 Dec 2019, 3:32:01 UTC - in response to Message 2023530.  

Yes, after Jimbocous reported he was able to report tasks, I figured the old problem of asking for work at the same time as reporting had reared its head. Setting to 64 or even less never works for me. I just set NNT on all the hosts so they can report work. Nobody is getting any work anyway so NNT is a moot point. Finally starting to unload all the work on the hosts.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2023531 · Report as offensive
Larry D. Leach

Send message
Joined: 17 May 99
Posts: 2
Credit: 5,651,524
RAC: 106
United States
Message 2023532 - Posted: 18 Dec 2019, 3:50:39 UTC

I'm only seeing this on one computer (out of 4). Perhaps this one computer just 'got ahead' and some issue is causing no new work to be available?

I've checked every setting possible (and compared those settings to the other computers).

Thanks,

-ldl-
ID: 2023532 · Report as offensive
Larry D. Leach

Send message
Joined: 17 May 99
Posts: 2
Credit: 5,651,524
RAC: 106
United States
Message 2023533 - Posted: 18 Dec 2019, 4:41:01 UTC - in response to Message 2023532.  

Hmm, not sure what changed but after some number of hours (or more), it just got new work and is chewing away. So--ignore me I reckon :)
ID: 2023533 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 2023536 - Posted: 18 Dec 2019, 5:18:49 UTC - in response to Message 2023531.  

Yes, after Jimbocous reported he was able to report tasks, I figured the old problem of asking for work at the same time as reporting had reared its head. Setting to 64 or even less never works for me. I just set NNT on all the hosts so they can report work. Nobody is getting any work anyway so NNT is a moot point. Finally starting to unload all the work on the hosts.

FWIW, my hosts were set to 100,200 and 300 max to report, respectively. Just the luck of the draw on that one, I think ...
Good point on the NNT, will have to try that permutation one of these times. Thanks ...
ID: 2023536 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13154
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2023537 - Posted: 18 Dec 2019, 5:28:42 UTC

I normally run with a max_report value of 100. Wasn't doing anything other than generating scheduler contact errors and multi-hour backoffs. I have tried as low as 40 before and it never works for me. But soon as I set NNT, the very first contact sent my 100 tasks right off and then I was finally back on the standard 305 second scheduler timer. I have reported everything except for the dedicated Seti Xeon server that I had powered off earlier in the day since it was just burning idle power doing nothing. My other hosts have concurrent alternate projects so at least they kept the gpus busy.

Still not getting any work on request.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2023537 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 2023538 - Posted: 18 Dec 2019, 5:30:44 UTC - in response to Message 2023537.  

Still not getting any work on request.
Finally started getting some about an hour ago.
ID: 2023538 · Report as offensive
Profile Sebastian M. Bobrecki
Volunteer tester

Send message
Joined: 7 Feb 02
Posts: 23
Credit: 38,375,443
RAC: 0
Poland
Message 2023542 - Posted: 18 Dec 2019, 6:34:41 UTC

I got some, but still no working downloads for me.
ID: 2023542 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2023543 - Posted: 18 Dec 2019, 6:41:49 UTC - in response to Message 2023538.  

Still not getting any work on request.
Finally started getting some about an hour ago.


. . Well I am getting work now nd it comes through on the first 2 machines back in service but on this beastie nothing will download, it says cannot connect with server.

:(

Stephen

:(
ID: 2023543 · Report as offensive
Profile Rowe Family and Friends

Send message
Joined: 25 Dec 00
Posts: 17
Credit: 38,395,231
RAC: 67
New Zealand
Message 2023544 - Posted: 18 Dec 2019, 6:44:17 UTC
Last modified: 18 Dec 2019, 6:45:56 UTC

and the million odd tasks are now all gone. stuck waiting for new ones.

still had over 200 GPU tasks to process but was out of CPU tasks for about 5 hours. these newer multi core CPU's chew through a heap of WU compared to even a mid range GPU
ID: 2023544 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13368
Credit: 208,696,464
RAC: 304
Australia
Message 2023545 - Posted: 18 Dec 2019, 6:52:23 UTC

Well, a whole bunch of downloads waiting to download when I got home. Lots of clicking later they eventually downloaded, but the Scheduler isn't issuing any new work & the splitters aren't producing any anyway. And then there's still the issue of downloading them if you're lucky enough to score any...
I'm wondering if whatever broke before the outage didn't get sorted, or is just having an excessive impact on the after outage recovery?
Grant
Darwin NT
ID: 2023545 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13368
Credit: 208,696,464
RAC: 304
Australia
Message 2023547 - Posted: 18 Dec 2019, 7:06:34 UTC

Tuesday (in the US) is when they have a weekly outage for database maintenance.
Most days it only last a few hours (although sometimes it can take 8 or more), and today there were some issues going in to the outage, and the recovery from the outage is showing signs of problems as well.

So getting work could be spotty for the next day or so.
Grant
Darwin NT
ID: 2023547 · Report as offensive
Profile Joseph Stateson Project Donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 307
Credit: 70,759,933
RAC: 3
United States
Message 2023549 - Posted: 18 Dec 2019, 7:10:10 UTC

Found other things to do

Discovered that Einstein on my GTX l1060 class miner runs 32c to 37c temps. Far lower than SETI.

MY other mining system, mix of 1660ti, p102-100, 1070 has somehow snagged 4 gpugrid tasks. Temps are significantly higher though both are in the same location in the garage and a box fan is blowing. 58c to 73c

All on risers and seem to be chugging along fairly well. Looks like the gpugrid will finish in about 5 hours, about what they do on my desktop in a fujl slot with the same class of cards
ID: 2023549 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 2023551 - Posted: 18 Dec 2019, 7:13:42 UTC - in response to Message 2023549.  

MY other mining system, mix of 1660ti, p102-100, 1070 has somehow snagged 4 gpugrid tasks. Temps are significantly higher though both are in the same location in the garage and a box fan is blowing. 58c to 73c
GPUGrid does love to cook those GPUs. Made me rethink my cooling the first time I got some work there ... I'd kill for temps that low, however. My SETI crunchers keep their 980s at 75-80c, external on risers. Ugh ...
ID: 2023551 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13368
Credit: 208,696,464
RAC: 304
Australia
Message 2023553 - Posted: 18 Dec 2019, 7:28:12 UTC

Maybe if the Assimilators & Deleters could clear their backlog, then the Purgers could clear their backlog, and then maybe the splitters could produce some work?
Grant
Darwin NT
ID: 2023553 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14504
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2023554 - Posted: 18 Dec 2019, 8:01:26 UTC

I don't think 77/sec is too bad, under the circumstances. I'm getting work, hot from the oven.
ID: 2023554 · Report as offensive
halfempty
Avatar

Send message
Joined: 2 Jun 99
Posts: 97
Credit: 35,236,901
RAC: 114
United States
Message 2023555 - Posted: 18 Dec 2019, 8:01:45 UTC - in response to Message 2023531.  

... Setting to 64 or even less never works for me. I just set NNT on all the hosts so they can report work ...

Wish I had thought of that, NNT is easier and more efficient for me than editing the config files. Learned something new for next time, for now things went back to mostly normal when I stopped staring at the screen :)
ID: 2023555 · Report as offensive
Previous · 1 . . . 39 · 40 · 41 · 42 · 43 · 44 · 45 . . . 52 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)


 
©2022 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.