Panic Mode On (110) Server Problems?

Message boards : Number crunching : Panic Mode On (110) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · 26 · 27 . . . 37 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921557 - Posted: 28 Feb 2018, 3:02:02 UTC

Well, Server Status page is now green, although Scheduler responses (no work available) are taking a minute or more to arrive.
Grant
Darwin NT
ID: 1921557 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1921558 - Posted: 28 Feb 2018, 3:03:32 UTC - in response to Message 1921554.  

Sounds likely. 422,146 hours backlog is something over 48 years, which takes us back to early 1970. Sound familiar? Unix time zero?

I think you have likely deduced the issue Richard.

+1

BTW After the outage the updates are ending on:
Tue 27 Feb 2018 10:00:44 PM EST | SETI@home | Scheduler request failed: HTTP internal server error


ID: 1921558 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921559 - Posted: 28 Feb 2018, 3:03:48 UTC - in response to Message 1921443.  

Glad the staff was able to fix that without more upset. Very helpful to know the creation rate, RTS buffer and tasks returned per hour to know whether the splitters are able to keep up with demand.

There is still something wrong with the transitioner though. Haveland transitioner graph has been flatlined for a day.

And the WU deleters don't seem capable of keeping up any more, so that's probably impacting on splitter output as well.
Grant
Darwin NT
ID: 1921559 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921562 - Posted: 28 Feb 2018, 3:19:14 UTC - in response to Message 1921558.  
Last modified: 28 Feb 2018, 3:19:44 UTC

BTW After the outage the updates are ending on:
Tue 27 Feb 2018 10:00:44 PM EST | SETI@home | Scheduler request failed: HTTP internal server error

I'm still getting Scheduler responses, just taking ages for them to finally arrive. And still no work from any of them.
Grant
Darwin NT
ID: 1921562 · Report as offensive
Iona
Avatar

Send message
Joined: 12 Jul 07
Posts: 790
Credit: 22,438,118
RAC: 0
United Kingdom
Message 1921563 - Posted: 28 Feb 2018, 3:23:34 UTC

I was down to my last GPU task and I've just got some more work, but not a full quota . The downloads are mighty slow, though.
Don't take life too seriously, as you'll never come out of it alive!
ID: 1921563 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921565 - Posted: 28 Feb 2018, 3:27:56 UTC - in response to Message 1921563.  

I was down to my last GPU task and I've just got some more work, but not a full quota .

I think you might be the only person- the Ready-to-send buffer has actually increased (slightly) since the outage. Usually it hits the ground like a stone.
Hopefully the servers will sort them selves out in the not too distant future.

The downloads are mighty slow, though.

I'm finding the forums a bit sluggish at the moment. Takes a while for the page to come up & the thread to start to load.
Grant
Darwin NT
ID: 1921565 · Report as offensive
Profile Chris904395093209d Project Donor
Volunteer tester

Send message
Joined: 1 Jan 01
Posts: 112
Credit: 29,923,129
RAC: 6
United States
Message 1921569 - Posted: 28 Feb 2018, 3:46:20 UTC

Dang it, my fastest PC has been out of CPU tasks for a few hours at least. Normally I can get new tasks as soon as the servers are back and before the pool runs dry. But not tonight. SETI was kind enough to send 1 GPU task for this PC though :-(
~Chris

ID: 1921569 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1921572 - Posted: 28 Feb 2018, 4:03:56 UTC

Lots of task in the download queue. None of them make it down. Backoffs around 3 1/2 hours.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1921572 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921574 - Posted: 28 Feb 2018, 4:07:05 UTC - in response to Message 1921572.  
Last modified: 28 Feb 2018, 4:07:42 UTC

Lots of task in the download queue. None of them make it down. Backoffs around 3 1/2 hours.

Yep.
Gone from not getting any, to not being able to get them to the system once I have got them allocated.

The recovery after outages would appear to be getting worse, not better.
Grant
Darwin NT
ID: 1921574 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921581 - Posted: 28 Feb 2018, 4:53:11 UTC

Still no joy with downloads.
It's times like these I really miss the Cricket graphs.
Grant
Darwin NT
ID: 1921581 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1921583 - Posted: 28 Feb 2018, 5:10:51 UTC - in response to Message 1921574.  

Lots of task in the download queue. None of them make it down. Backoffs around 3 1/2 hours.

Yep.
Gone from not getting any, to not being able to get them to the system once I have got them allocated.

The recovery after outages would appear to be getting worse, not better.


. . Same here allocated 40 tasks (or so) and it took over half an hour to download them :(

Stephen

:(
ID: 1921583 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921585 - Posted: 28 Feb 2018, 5:27:22 UTC - in response to Message 1921583.  

. . Same here allocated 40 tasks (or so) and it took over half an hour to download them :(

Once the download server came back to life they downloaded quickly. Now all we need is for the splitters to start cranking up their output.
Grant
Darwin NT
ID: 1921585 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921587 - Posted: 28 Feb 2018, 6:07:57 UTC - in response to Message 1921534.  

I wonder if the transitioner backlog error was somehow tied to the missing creation rate values for most of yesterday. Those got fixed late in the day but the transitioner reporting is still out to lunch.

Looks like the Transitioner updating is still broken.
Grant
Darwin NT
ID: 1921587 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36871
Credit: 261,360,520
RAC: 489
Australia
Message 1921591 - Posted: 28 Feb 2018, 8:06:01 UTC - in response to Message 1921563.  

I was down to my last GPU task and I've just got some more work, but not a full quota . The downloads are mighty slow, though.

I was out for a few hours before I could get a handful on 1 rig, but I did smash some extra CPU tasks during that time.

But after 2hrs at the pub things here were all good again (proof again that beer will fix everything), but that will hurt the old RAC again.

Cheers.
ID: 1921591 · Report as offensive
Profile Kissagogo27 Special Project $75 donor
Avatar

Send message
Joined: 6 Nov 99
Posts: 716
Credit: 8,032,827
RAC: 62
France
Message 1921616 - Posted: 28 Feb 2018, 10:29:20 UTC

about the same here this morning

28-Feb-2018 09:51:40 [SETI@home] Requesting new tasks for CPU and AMD/ATI GPU
28-Feb-2018 09:51:43 [SETI@home] Scheduler request completed: got 2 new tasks
28-Feb-2018 09:51:46 [SETI@home] Started download of ap_14fe18aa_B6_P1_00286_20180228_07134.wu
28-Feb-2018 09:52:51 [SETI@home] Finished download of ap_14fe18aa_B6_P1_00286_20180228_07134.wu
28-Feb-2018 09:52:51 [SETI@home] Started download of ap_15au16aa_B2_P1_00209_20180228_09870.wu
28-Feb-2018 09:53:48 [SETI@home] Finished download of ap_15au16aa_B2_P1_00209_20180228_09870.wu
28-Feb-2018 09:56:50 [SETI@home] Sending scheduler request: To fetch work.
28-Feb-2018 09:56:50 [SETI@home] Requesting new tasks for CPU and AMD/ATI GPU
28-Feb-2018 09:56:53 [SETI@home] Scheduler request completed: got 1 new tasks
28-Feb-2018 09:56:55 [SETI@home] Started download of ap_17au16ac_B1_P1_00287_20180228_07159.wu
28-Feb-2018 10:01:56 [SETI@home] Temporarily failed download of ap_17au16ac_B1_P1_00287_20180228_07159.wu: transient HTTP error
28-Feb-2018 10:01:56 [SETI@home] Backing off 00:02:18 on download of ap_17au16ac_B1_P1_00287_20180228_07159.wu
28-Feb-2018 10:01:59 [---] Project communication failed: attempting access to reference site
28-Feb-2018 10:02:02 [---] Internet access OK - project servers may be temporarily down.
28-Feb-2018 10:02:57 [SETI@home] Computation for task blc13_2bit_guppi_58137_40992_HIP64241_0041.3597.0.21.44.77.vlar_1 finished
28-Feb-2018 10:02:57 [SETI@home] Starting task 12fe18aa.7416.215582.15.42.231_1
28-Feb-2018 10:03:05 [SETI@home] Started upload of blc13_2bit_guppi_58137_40992_HIP64241_0041.3597.0.21.44.77.vlar_1_r1107740128_0
28-Feb-2018 10:03:21 [SETI@home] Finished upload of blc13_2bit_guppi_58137_40992_HIP64241_0041.3597.0.21.44.77.vlar_1_r1107740128_0
28-Feb-2018 10:04:16 [SETI@home] Started download of ap_17au16ac_B1_P1_00287_20180228_07159.wu
28-Feb-2018 10:04:22 [SETI@home] Sending scheduler request: To fetch work.
28-Feb-2018 10:04:22 [SETI@home] Reporting 1 completed tasks
28-Feb-2018 10:04:22 [SETI@home] Requesting new tasks for CPU and AMD/ATI GPU
28-Feb-2018 10:04:24 [SETI@home] Scheduler request completed: got 0 new tasks
28-Feb-2018 10:04:24 [SETI@home] Project has no tasks available
28-Feb-2018 10:05:01 [SETI@home] Finished download of ap_17au16ac_B1_P1_00287_20180228_07159.wu


transient slow down in data transfers, easy to see with AP wu ^^
ID: 1921616 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1921760 - Posted: 28 Feb 2018, 22:43:25 UTC

File deleters & WU purges still not keeping up, and most likely as a direct result of that is the limited splitter output.
Grant
Darwin NT
ID: 1921760 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1921923 - Posted: 1 Mar 2018, 17:47:32 UTC - in response to Message 1921760.  

Database is very slow now. Can't pull up anybody's tasks, just watch the spinner endlessly.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1921923 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9958
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1921924 - Posted: 1 Mar 2018, 17:51:03 UTC

Well seems to work OK here, just had a look at a couple of your tasks OK
ID: 1921924 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1921926 - Posted: 1 Mar 2018, 17:59:09 UTC - in response to Message 1921924.  
Last modified: 1 Mar 2018, 18:02:14 UTC

I just pulled up some of your tasks and they got displayed quickly. Looking at anyone else with several thousand tasks in any category is painfully slow, if the page ever populates.

[Edit] I'm specifically trying to look at Petri's Titan V results.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1921926 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1921934 - Posted: 1 Mar 2018, 18:37:16 UTC - in response to Message 1921926.  

specifically trying to look at Petri's Titan V results.


While it shows in the computer, I'm not seeing any work being done on that card.
ID: 1921934 · Report as offensive
Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · 26 · 27 . . . 37 · Next

Message boards : Number crunching : Panic Mode On (110) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.