The Server Issues / Outages Thread - Panic Mode On! (119)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (119)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 57 · 58 · 59 · 60 · 61 · 62 · 63 . . . 107 · Next

AuthorMessage
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2042627 - Posted: 2 Apr 2020, 21:10:00 UTC - in response to Message 2042611.  

my wild unsupported guess... Carolyn (the replica) had disk issues last night. When I look at my personal WU list, it is now current, and not 4 days behind like it was a bit ago.It is no longer looking at Carolyn.


and now my view is for March 29 at the latest in my personal view...so we are now back viewing Carolyn
ID: 2042627 · Report as offensive     Reply Quote
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 2042653 - Posted: 2 Apr 2020, 23:36:15 UTC

There is just over 4.3 million results, roughly 8.2 million results waiting administration and over 22 million results awaiting validation. I wonder when we will start to see 8.2 & 22 numbers start to drop? If we are waiting on resends this could be a few months away
ID: 2042653 · Report as offensive     Reply Quote
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 2042712 - Posted: 3 Apr 2020, 4:57:42 UTC
Last modified: 3 Apr 2020, 5:10:45 UTC

I see the Scheduler has been falling over repeatedly & for extended periods for just on 24hrs now.

2/04/2020 14:40:17 | SETI@home | Scheduler request failed: Couldn't connect to server
2/04/2020 14:42:24 | SETI@home | Scheduler request failed: Couldn't connect to server
2/04/2020 14:45:51 | SETI@home | Project has no tasks available
2/04/2020 15:06:21 | SETI@home | Scheduler request failed: Couldn't connect to server
2/04/2020 15:08:03 | SETI@home | Scheduler request failed: Couldn't connect to server
2/04/2020 15:12:05 | SETI@home | Scheduler request failed: Couldn't connect to server
2/04/2020 15:19:03 | SETI@home | Scheduler request failed: Couldn't connect to server
2/04/2020 15:33:29 | SETI@home | Scheduler request failed: HTTP service unavailable
2/04/2020 16:21:41 | SETI@home | Scheduler request failed: Couldn't connect to server
2/04/2020 16:23:48 | SETI@home | Project has no tasks available
3/04/2020 4:01:45 | SETI@home | Scheduler request failed: HTTP service unavailable
3/04/2020 4:03:28 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 4:07:50 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 4:14:39 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 4:24:18 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 4:49:59 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 5:34:43 | SETI@home | Project has no tasks available
3/04/2020 7:22:48 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 7:24:50 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 7:27:28 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 7:32:50 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 7:42:49 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 8:03:02 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 9:00:37 | SETI@home | Project has no tasks available
3/04/2020 12:31:37 | SETI@home | Scheduler request failed: HTTP service unavailable
3/04/2020 12:34:44 | SETI@home | Scheduler request failed: HTTP service unavailable
3/04/2020 12:38:27 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 12:45:34 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 12:56:38 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 13:25:47 | SETI@home | Scheduler request failed: HTTP internal server error
3/04/2020 14:16:20 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 14:24:22 | SETI@home | Scheduler request failed: Couldn't connect to server
3/04/2020 14:39:31 | SETI@home | Scheduler request failed: HTTP service unavailable



Edit- and the forums are barely functioning. Multi-minute page & thread load times.
Grant
Darwin NT
ID: 2042712 · Report as offensive     Reply Quote
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 2042733 - Posted: 3 Apr 2020, 7:17:22 UTC - in response to Message 2042730.  
Last modified: 3 Apr 2020, 8:15:54 UTC

We're doomed!!!
Yep.
Scheduler still MIA.


And for an idea of what a mess the servers are at the moment, take a look at the Graphs.
(and the Scheduler is still MIA).
Grant
Darwin NT
ID: 2042733 · Report as offensive     Reply Quote
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2042743 - Posted: 3 Apr 2020, 10:00:55 UTC

We could declare Panic Mode ON!
ID: 2042743 · Report as offensive     Reply Quote
BetelgeuseFive Project Donor
Volunteer tester

Send message
Joined: 6 Jul 99
Posts: 158
Credit: 17,117,787
RAC: 19
Netherlands
Message 2042744 - Posted: 3 Apr 2020, 10:21:40 UTC

Is the scheduler MIA or Mostly MIA ?

From SSP:

Results received in last hour ** 0 208 5,031 0m

These number change over time, so unless these numbers are completely bogus (always an option) it seems to indicate that there are some scheduler requests coming through ...
ID: 2042744 · Report as offensive     Reply Quote
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2042747 - Posted: 3 Apr 2020, 11:27:09 UTC - in response to Message 2042746.  

Please don't do that. 'They' will hardly notice, but you would be prolonging the agony for 'us' - those tasks would remain, unfinished, in the database for between another three weeks and two months, before being sent out yet again for another cruncher to try, and perhaps fail, to finish them off.

Like you, I loaded all my machines to the maximum possible in the final days, but all my work was both uploaded and reported before I went to bed last night. We know that we, collectively, have put one hell of a strain on the servers since the limits were raised in early December - perhaps, in hindsight, a little too far.

By all means, turn the machines off to save what little electricity they will use while not crunching. But please turn at least one on, at some random time of day, to see if you can snag a delivery slot for your finished work. If that works, move on to the next...
ID: 2042747 · Report as offensive     Reply Quote
Profile Freewill Project Donor
Avatar

Send message
Joined: 19 May 99
Posts: 766
Credit: 354,398,348
RAC: 11,693
United States
Message 2042758 - Posted: 3 Apr 2020, 13:09:36 UTC - in response to Message 2042753.  

You're having better luck than me.
Some serious issue, I think. When I try to look at my hosts I get the following:
Database Error
Warning: Invalid argument supplied for foreach() in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/host.inc on line 792

Warning: Invalid argument supplied for foreach() in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/host.inc on line 799

ID: 2042758 · Report as offensive     Reply Quote
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2042760 - Posted: 3 Apr 2020, 13:14:32 UTC

Carolyn is MIA. But must be 5 AM in CA. Let's them finish a good night of sleep first.
ID: 2042760 · Report as offensive     Reply Quote
Profile Freewill Project Donor
Avatar

Send message
Joined: 19 May 99
Posts: 766
Credit: 354,398,348
RAC: 11,693
United States
Message 2042765 - Posted: 3 Apr 2020, 13:47:32 UTC - in response to Message 2042760.  
Last modified: 3 Apr 2020, 13:48:27 UTC

Carolyn is MIA. But must be 5 AM in CA. Let's them finish a good night of sleep first.
Well, 6:45 AM currently.
They need to get a Finnish staff member. Those people never sleep! :)
ID: 2042765 · Report as offensive     Reply Quote
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2042771 - Posted: 3 Apr 2020, 14:18:40 UTC - in response to Message 2042765.  
Last modified: 3 Apr 2020, 14:20:13 UTC

Carolyn is MIA. But must be 5 AM in CA. Let's them finish a good night of sleep first.
Well, 6:45 AM currently.
They need to get a Finnish staff member. Those people never sleep! :)

You are right, i forget i'm Panama not Brazil. LOL
One who never sleep is a coder friend from Finland.
ID: 2042771 · Report as offensive     Reply Quote
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2042779 - Posted: 3 Apr 2020, 15:12:09 UTC - in response to Message 2042733.  


And for an idea of what a mess the servers are at the moment, take a look at the Graphs.
(and the Scheduler is still MIA).


The graphs are just like the server...a big ole mess, but interesting to see visually, so thanks for the link. My machine did finally manage to connect and report finished work. I'll be around for a few more days finishing up my WUs and hoping to snag some resends. I managed to get one resend about 24 hours ago, but nothing since.

The status page now has some missing fields too.
ID: 2042779 · Report as offensive     Reply Quote
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2042781 - Posted: 3 Apr 2020, 15:26:21 UTC - in response to Message 2042779.  

My machine did finally manage to connect and report finished work.

Nothing form here yet just the usual:
vie 03 abr 2020 10:25:38 EST | SETI@home | Scheduler request failed: Timeout was reached
vie 03 abr 2020 10:25:38 EST | SETI@home | [sched_op] Deferring communication for 00:01:05
vie 03 abr 2020 10:25:38 EST | SETI@home | [sched_op] Reason: Scheduler request failed
vie 03 abr 2020 10:25:44 EST |  | Internet access OK - project servers may be temporarily down.

ID: 2042781 · Report as offensive     Reply Quote
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 2042787 - Posted: 3 Apr 2020, 17:08:15 UTC

03/04/2020 19:06:38 | SETI@home | Reporting 17 completed tasks
03/04/2020 19:06:38 | SETI@home | Requesting new tasks for NVIDIA GPU
03/04/2020 19:06:44 | SETI@home | Scheduler request completed: got 0 new tasks
03/04/2020 19:06:44 | SETI@home | Project has no tasks available
ID: 2042787 · Report as offensive     Reply Quote
Profile Kissagogo27 Special Project $75 donor
Avatar

Send message
Joined: 6 Nov 99
Posts: 715
Credit: 8,032,827
RAC: 62
France
Message 2042788 - Posted: 3 Apr 2020, 17:11:01 UTC
Last modified: 3 Apr 2020, 17:11:23 UTC

Strange behavior sending 1 lost task ... after contacting server few minutes ago ...
UTC+2

03-Apr-2020 19:02:36 [SETI@home] Requesting new tasks for AMD/ATI GPU
03-Apr-2020 19:02:58 [SETI@home] Scheduler request completed: got 1 new tasks
03-Apr-2020 19:02:58 [SETI@home] Resent lost task 09no11ae.25006.110881.10.37.100_2
03-Apr-2020 19:02:58 [SETI@home] Project has no tasks available
03-Apr-2020 19:03:00 [SETI@home] Started download of 09no11ae.25006.110881.10.37.100
03-Apr-2020 19:03:04 [SETI@home] Finished download of 09no11ae.25006.110881.10.37.100


Deadline to 24/04/2020 > 3 weeks
ID: 2042788 · Report as offensive     Reply Quote
Profile Freewill Project Donor
Avatar

Send message
Joined: 19 May 99
Posts: 766
Credit: 354,398,348
RAC: 11,693
United States
Message 2042792 - Posted: 3 Apr 2020, 17:31:56 UTC

I am also able to send tasks now on 3 hosts, so situation is improved.
ID: 2042792 · Report as offensive     Reply Quote
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2042796 - Posted: 3 Apr 2020, 18:33:31 UTC

And I'm starting to get the expected trickle of resent tasks. Trouble is, BOINC doesn't really have a setting for 'tidying up the loose ends of a project that's closing down', so the hosts are backed off and don't ask very often.
ID: 2042796 · Report as offensive     Reply Quote
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2042805 - Posted: 3 Apr 2020, 19:51:14 UTC - in response to Message 2042779.  


And for an idea of what a mess the servers are at the moment, take a look at the Graphs.
(and the Scheduler is still MIA).


The graphs are just like the server...a big ole mess, but interesting to see visually, so thanks for the link. My machine did finally manage to connect and report finished work. I'll be around for a few more days finishing up my WUs and hoping to snag some resends. I managed to get one resend about 24 hours ago, but nothing since.

The status page now has some missing fields too.


. . My slower machines are crunching E@H and checking (all too infrequently) on S@H for resends. I would prefer if they checked at least once an hour not once or twice per day :( They have had no SETI work since about 8 am Thursday UTC.

Stephen

:(
ID: 2042805 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2042806 - Posted: 3 Apr 2020, 19:52:20 UTC

I see that I have also picked up a trickle of resends on the daily driver. First since the project ran out of new work.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2042806 · Report as offensive     Reply Quote
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2042807 - Posted: 3 Apr 2020, 19:53:33 UTC - in response to Message 2042796.  
Last modified: 3 Apr 2020, 19:55:13 UTC

And I'm starting to get the expected trickle of resent tasks. Trouble is, BOINC doesn't really have a setting for 'tidying up the loose ends ... etc


. . In spades, almost never :(

Stephen

:(
ID: 2042807 · Report as offensive     Reply Quote
Previous · 1 . . . 57 · 58 · 59 · 60 · 61 · 62 · 63 . . . 107 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (119)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.