Panic Mode On (79) Server Problems?

Message boards : Number crunching : Panic Mode On (79) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 16 · 17 · 18 · 19 · 20 · 21 · 22 · Next

AuthorMessage
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1318357 - Posted: 21 Dec 2012, 19:42:06 UTC

Hm. Overnight, my AP-only 10-day cache got filled without much fuss.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1318357 · Report as offensive
Profile Donald L. Johnson
Avatar

Send message
Joined: 5 Aug 02
Posts: 8240
Credit: 14,654,533
RAC: 20
United States
Message 1320034 - Posted: 26 Dec 2012, 3:58:57 UTC

Just a bump to keep this thread from falling off the first page. Oh, wow. Last post was before noon PDT on Friday........
Donald
Infernal Optimist / Submariner, retired
ID: 1320034 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1320036 - Posted: 26 Dec 2012, 4:13:25 UTC

I have noticed that in combination of the limits and a reduced number of AP splitters, the download pipe is not fully saturated and allows comms to happen pretty smoothly. For instance, just a few minutes ago, I downloaded an AP in 6 seconds rather than 6 minutes.

Not saying the limits should be kept, but maybe they can be ramped up slowly from like 100/100 to 125/125 and hold it there for a week or so and see how everything reacts (database, network comms), and then maybe +25 again. Do that with small number of AP splitters to try to keep their population controlled at a reasonable amount.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1320036 · Report as offensive
Profile Gatekeeper
Avatar

Send message
Joined: 14 Jul 04
Posts: 887
Credit: 176,479,616
RAC: 0
United States
Message 1320475 - Posted: 27 Dec 2012, 17:45:45 UTC

For the last hour or so, I'm getting:

12/27/2012 10:41:08 AM | SETI@home | Sending scheduler request: Requested by user.
12/27/2012 10:41:08 AM | SETI@home | Reporting 20 completed tasks, requesting new tasks for NVIDIA GPU
12/27/2012 10:41:10 AM | SETI@home | Scheduler request failed: Server returned nothing (no headers, no data)
12/27/2012 10:41:14 AM | | Project communication failed: attempting access to reference site
12/27/2012 10:41:15 AM | | Internet access OK - project servers may be temporarily down.

Status page looks all green. Cricket shows signs of throughput drop though.
ID: 1320475 · Report as offensive
andybutt
Volunteer tester
Avatar

Send message
Joined: 18 Mar 03
Posts: 262
Credit: 164,205,187
RAC: 516
United Kingdom
Message 1320479 - Posted: 27 Dec 2012, 17:55:14 UTC - in response to Message 1320475.  

looks like something just died. i am getting the same
ID: 1320479 · Report as offensive
Mark Fiske

Send message
Joined: 15 Aug 11
Posts: 713
Credit: 7,392,921
RAC: 0
United States
Message 1320480 - Posted: 27 Dec 2012, 18:02:18 UTC

Looks like it's back up:

12/27/2012 9:57:59 AM | SETI@home | update requested by user
12/27/2012 9:58:48 AM | SETI@home | Scheduler request completed: got 13 new tasks
12/27/2012 9:58:48 AM | SETI@home | Resent lost task 01au12ab.23011.21748.140733193388048.10.239_1
...and a bunch more lost ones...
ID: 1320480 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 1320481 - Posted: 27 Dec 2012, 18:08:58 UTC - in response to Message 1320480.  

Looks like it's back up:

12/27/2012 9:57:59 AM | SETI@home | update requested by user
12/27/2012 9:58:48 AM | SETI@home | Scheduler request completed: got 13 new tasks
12/27/2012 9:58:48 AM | SETI@home | Resent lost task 01au12ab.23011.21748.140733193388048.10.239_1
...and a bunch more lost ones...

Nah, it's still borked.
Also getting "HTTP service unavalable" error messages. Every now & then a bit of work comes through.
Grant
Darwin NT
ID: 1320481 · Report as offensive
.clair.

Send message
Joined: 4 Nov 04
Posts: 1300
Credit: 55,390,408
RAC: 69
United Kingdom
Message 1320501 - Posted: 27 Dec 2012, 19:15:48 UTC

More AP splitter shufling going on,
we iz up to fore now,
them boyz in de lab iz playin :¬)
Av a louk a D SSP
ID: 1320501 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1320531 - Posted: 27 Dec 2012, 20:40:40 UTC - in response to Message 1320501.  

There's a lot of VLAR's coming through as well.

Cheers.
ID: 1320531 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1320803 - Posted: 28 Dec 2012, 14:35:56 UTC
Last modified: 28 Dec 2012, 14:41:42 UTC

Scheduler contacts are frequently timing out or returning nothing (no headers, no data), and when i manage to get work they are all Shorties, at which point downloads are terribly slow, sounds as if the internal network is possibly overloaded again.

Claggy
ID: 1320803 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14644
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1320807 - Posted: 28 Dec 2012, 14:50:38 UTC - in response to Message 1320803.  

Scheduler contacts are frequently timing out or returning nothing (no headers, no data), and when i manage to get work they are all Shorties, at which point downloads are terribly slow, sounds as if the internal network is possibly overloaded again.

Claggy

Are you creating ghost results on your scheduler timeouts? (Specifically, the timeouts, not the empty replies)

And similarly, are results which you have attempted, but apparently failed, to report appearing as completed on the web task lists?
ID: 1320807 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1320810 - Posted: 28 Dec 2012, 15:11:07 UTC - in response to Message 1320807.  
Last modified: 28 Dec 2012, 15:14:54 UTC

Scheduler contacts are frequently timing out or returning nothing (no headers, no data), and when i manage to get work they are all Shorties, at which point downloads are terribly slow, sounds as if the internal network is possibly overloaded again.

Claggy

Are you creating ghost results on your scheduler timeouts? (Specifically, the timeouts, not the empty replies)

And similarly, are results which you have attempted, but apparently failed, to report appearing as completed on the web task lists?

Yes for the first question:

28/12/2012 15:00:19 SETI@home [sched_op_debug] Starting scheduler request
28/12/2012 15:00:19 SETI@home Sending scheduler request: To fetch work.
28/12/2012 15:00:19 SETI@home Reporting 3 completed tasks, requesting new tasks for GPU
28/12/2012 15:00:19 SETI@home [sched_op_debug] CPU work request: 0.00 seconds; 0.00 CPUs
28/12/2012 15:00:19 SETI@home [sched_op_debug] NVIDIA GPU work request: 0.00 seconds; 0.00 GPUs
28/12/2012 15:00:19 SETI@home [sched_op_debug] ATI GPU work request: 82261.43 seconds; 0.00 GPUs
28/12/2012 15:05:28 Project communication failed: attempting access to reference site
28/12/2012 15:05:28 SETI@home Scheduler request failed: Timeout was reached
28/12/2012 15:05:28 SETI@home [sched_op_debug] Deferring communication for 1 min 0 sec
28/12/2012 15:05:28 SETI@home [sched_op_debug] Reason: Scheduler request failed
28/12/2012 15:05:29 Internet access OK - project servers may be temporarily down.


there are 20 ATI Ghosts waiting to be resent at the moment (all timed at the moment between 28 Dec 2012 | 15:00:33 UTC and 15:00:38 UTC): All tasks for computer 5427475

For the second question, i think it is yes too, (three tasks were reported, one of them i've seen as being reported at 15:00:31 UTC)

Claggy
ID: 1320810 · Report as offensive
Andre Howard
Volunteer tester
Avatar

Send message
Joined: 16 May 99
Posts: 124
Credit: 217,463,217
RAC: 0
United States
Message 1320812 - Posted: 28 Dec 2012, 15:28:02 UTC - in response to Message 1320807.  

Yes to both those questions as well.

ID: 1320812 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1320814 - Posted: 28 Dec 2012, 15:34:11 UTC - in response to Message 1320810.  
Last modified: 28 Dec 2012, 15:35:49 UTC

Those 20 tasks got resent after about 15 minutes of trying, and were all downloaded at up to 40KBs:

28/12/2012 15:29:41 SETI@home [sched_op_debug] Starting scheduler request
28/12/2012 15:29:41 SETI@home Sending scheduler request: To fetch work.
28/12/2012 15:29:41 SETI@home Reporting 5 completed tasks, requesting new tasks for GPU
28/12/2012 15:29:41 SETI@home [sched_op_debug] CPU work request: 0.00 seconds; 0.00 CPUs
28/12/2012 15:29:41 SETI@home [sched_op_debug] NVIDIA GPU work request: 0.00 seconds; 0.00 GPUs
28/12/2012 15:29:41 SETI@home [sched_op_debug] ATI GPU work request: 85068.53 seconds; 0.00 GPUs
28/12/2012 15:29:45 SETI@home Beta Test [coproc_debug] CUDA instance 0: confirming for 08ja11ab.9886.16190.140733193388037.14.55_1
28/12/2012 15:29:45 SETI@home [coproc_debug] ATI instance 0: confirming for 07oc12ah.26654.67.4.10.136_1
28/12/2012 15:29:57 SETI@home Beta Test [coproc_debug] CUDA instance 0: confirming for 08ja11ab.9886.16190.140733193388037.14.55_1
28/12/2012 15:29:57 SETI@home [coproc_debug] ATI instance 0: confirming for 07oc12ah.26654.67.4.10.136_1
28/12/2012 15:30:24 SETI@home Beta Test [coproc_debug] CUDA instance 0: confirming for 08ja11ab.9886.16190.140733193388037.14.55_1
28/12/2012 15:30:24 SETI@home [coproc_debug] ATI instance 0: confirming for 07oc12ah.26654.67.4.10.136_1
28/12/2012 15:30:38 SETI@home Scheduler request completed: got 20 new tasks
28/12/2012 15:30:38 SETI@home [sched_op_debug] Server version 701
28/12/2012 15:30:38 SETI@home Message from server: Resent lost task 04au12ab.19381.22157.140733193388039.10.83_0
28/12/2012 15:30:38 SETI@home Message from server: Resent lost task 20au12ae.17020.4566.140733193388042.10.215_3
28/12/2012 15:30:38 SETI@home Message from server: Resent lost task 20au12ae.17020.4566.140733193388042.10.221_3
etc
28/12/2012 15:30:38 SETI@home Project requested delay of 303 seconds
28/12/2012 15:30:38 SETI@home [sched_op_debug] estimated total CPU job duration: 0 seconds
28/12/2012 15:30:38 SETI@home [sched_op_debug] estimated total NVIDIA GPU job duration: 0 seconds
28/12/2012 15:30:38 SETI@home [sched_op_debug] estimated total ATI GPU job duration: 9257 seconds
28/12/2012 15:30:38 SETI@home [sched_op_debug] handle_scheduler_reply(): got ack for result 07oc12ah.26654.67.4.10.197_1
28/12/2012 15:30:38 SETI@home [sched_op_debug] handle_scheduler_reply(): got ack for result 07oc12ah.26654.67.4.10.178_1
28/12/2012 15:30:38 SETI@home [sched_op_debug] handle_scheduler_reply(): got ack for result 22no12ab.14275.14791.140733193388048.10.157_1
28/12/2012 15:30:38 SETI@home [sched_op_debug] handle_scheduler_reply(): got ack for result 22no12ab.14275.18881.140733193388048.10.41_1
28/12/2012 15:30:38 SETI@home [sched_op_debug] handle_scheduler_reply(): got ack for result 07oc12ah.26654.67.4.10.138_1
28/12/2012 15:30:38 SETI@home [sched_op_debug] Deferring communication for 5 min 3 sec
28/12/2012 15:30:38 SETI@home [sched_op_debug] Reason: requested by project


Claggy
ID: 1320814 · Report as offensive
KB7RZF
Volunteer tester
Avatar

Send message
Joined: 15 Aug 99
Posts: 9549
Credit: 3,308,926
RAC: 2
United States
Message 1320816 - Posted: 28 Dec 2012, 16:00:06 UTC
Last modified: 28 Dec 2012, 16:01:24 UTC

I got 40 ghost WU's on this host of mine, result of the same issues posted before me:
http://setiathome.berkeley.edu/results.php?hostid=5332132
ID: 1320816 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65689
Credit: 55,293,173
RAC: 49
United States
Message 1320818 - Posted: 28 Dec 2012, 16:09:05 UTC - in response to Message 1320816.  

I got 40 ghost WU's on this host of mine, result of the same issues posted before me:
http://setiathome.berkeley.edu/results.php?hostid=5332132

Maybe Seti could call in these guys... ;)

Me I have no idea if I have any ghosts or not, besides I just woke up.
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 1320818 · Report as offensive
j tramer

Send message
Joined: 6 Oct 03
Posts: 242
Credit: 5,412,368
RAC: 0
Canada
Message 1320819 - Posted: 28 Dec 2012, 16:12:16 UTC

i shut down my second computer.....

not wanted not needed....

soon both computers will not be running seti

more ppl should quit seti
ID: 1320819 · Report as offensive
Keith White
Avatar

Send message
Joined: 29 May 99
Posts: 392
Credit: 13,035,233
RAC: 22
United States
Message 1320825 - Posted: 28 Dec 2012, 16:30:49 UTC - in response to Message 1320819.  

Aye, she's getting a touch cranky again. Flushed my DNS, renewed my IP address and it did get through enough to report and refill. Once. I'll come back in a few hours and see how she's doing.
"Life is just nature's way of keeping meat fresh." - The Doctor
ID: 1320825 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14644
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1320851 - Posted: 28 Dec 2012, 17:34:15 UTC - in response to Message 1320810.  

Scheduler contacts are frequently timing out or returning nothing (no headers, no data), and when i manage to get work they are all Shorties, at which point downloads are terribly slow, sounds as if the internal network is possibly overloaded again.

Claggy

Are you creating ghost results on your scheduler timeouts? (Specifically, the timeouts, not the empty replies)

And similarly, are results which you have attempted, but apparently failed, to report appearing as completed on the web task lists?

Yes for the first question:

28/12/2012 15:00:19 SETI@home [sched_op_debug] Starting scheduler request
28/12/2012 15:00:19 SETI@home Sending scheduler request: To fetch work.
28/12/2012 15:00:19 SETI@home Reporting 3 completed tasks, requesting new tasks for GPU
28/12/2012 15:00:19 SETI@home [sched_op_debug] CPU work request: 0.00 seconds; 0.00 CPUs
28/12/2012 15:00:19 SETI@home [sched_op_debug] NVIDIA GPU work request: 0.00 seconds; 0.00 GPUs
28/12/2012 15:00:19 SETI@home [sched_op_debug] ATI GPU work request: 82261.43 seconds; 0.00 GPUs
28/12/2012 15:05:28 Project communication failed: attempting access to reference site
28/12/2012 15:05:28 SETI@home Scheduler request failed: Timeout was reached
28/12/2012 15:05:28 SETI@home [sched_op_debug] Deferring communication for 1 min 0 sec
28/12/2012 15:05:28 SETI@home [sched_op_debug] Reason: Scheduler request failed
28/12/2012 15:05:29 Internet access OK - project servers may be temporarily down.


there are 20 ATI Ghosts waiting to be resent at the moment (all timed at the moment between 28 Dec 2012 | 15:00:33 UTC and 15:00:38 UTC): All tasks for computer 5427475

For the second question, i think it is yes too, (three tasks were reported, one of them i've seen as being reported at 15:00:31 UTC)

Claggy

I was getting exactly the same thing on Albert (Einstein's test server) before Christmas: the scheduler did everything it was supposed to do in less than a second, then sat there for two minutes twiddling its thumbs until Apache killed it with a SIGTERM (you can see useful things like that in the Einstein family server logs). I tried to convince Bernd and Eric (and David) that the two behaviours might be related (and not just by overwork - Albert was very lightly loaded at the time) - but Christmas holidays intervened. Something to pick up on in the New Year. Until then, zzzzzzzzz...
ID: 1320851 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65689
Credit: 55,293,173
RAC: 49
United States
Message 1320870 - Posted: 28 Dec 2012, 18:00:30 UTC - in response to Message 1320851.  

Just What both projects don't kneed, Senile servers, someone get out the rocking chairs... ;)
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 1320870 · Report as offensive
Previous · 1 . . . 16 · 17 · 18 · 19 · 20 · 21 · 22 · Next

Message boards : Number crunching : Panic Mode On (79) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.