Panic Mode On (51) Server problems?

Message boards : Number crunching : Panic Mode On (51) Server problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 . . . 10 · Next

AuthorMessage
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1130612 - Posted: 22 Jul 2011, 18:55:25 UTC - in response to Message 1130609.  

What the heck have they done to Carolyn, and who is taking her place?


My guess for the reason of this outage was a heatwave, combined with a failed AC-unit in the server closet.

I thought about that yesterday & checked the weather for Berkeley. They are getting highs in the upper 60's °F all week. Not to say something could not have overheated. I am sure their A/C runs year round as it does in my test lab at work.



Well, we did have the a/c go down last year....

Indeed. The A/C in my test lab goes down at least once a year since it runs nearly 24/7/365. Last year they had to put in a new compressor and a new condenser coil.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1130612 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1130669 - Posted: 22 Jul 2011, 21:08:12 UTC

Well....here we go again. Cricket graph headed up.
Scotty, we need more power.

Hope we can make it through the weekend.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1130669 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1130831 - Posted: 23 Jul 2011, 2:17:14 UTC - in response to Message 1130737.  


Once again i'm wondering if both download servers are really working?
Most Work Units time out almost as soon as they start. If they don't it can take 20-45 secs before the download actually starts to happen & then it's only at 3-5kB/s & often times out while downloading.
Other times a download starts, and after 3-10 secs it actually starts downloading & often at anywhere between 15-30kB/s.
Grant
Darwin NT
ID: 1130831 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22202
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1130875 - Posted: 23 Jul 2011, 5:26:45 UTC

Its all a bit bumpy just now.
Uploads have been OK with me most of the time, downloads work when they want to, sometimes.
Looking at the server status page some of the splitters are running, both AP and MB assimilators are playing their own game on a different pitch.
I guess the "Berkeley boys and girls" know what's going on, having spent yesterday (Friday) getting some of it going, so do we run a book on one of them coming in today (Saturday) and doing some unpaid overtime to get the rest going, or will they all take weekend off?
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1130875 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1130877 - Posted: 23 Jul 2011, 5:33:01 UTC - in response to Message 1130875.  

I guess the "Berkeley boys and girls" know what's going on, having spent yesterday (Friday) getting some of it going, so do we run a book on one of them coming in today (Saturday) and doing some unpaid overtime to get the rest going, or will they all take weekend off?

Probably wait & see. If it gets through the weekend without falling over- no problem. If not, then someone may come in to restart things if it can't be done remotely.
Grant
Darwin NT
ID: 1130877 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1130942 - Posted: 23 Jul 2011, 12:34:47 UTC

Ouch......
Somebody please get Vader back on track assimilating MB work, or it's gonna stop up the works.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1130942 · Report as offensive
Ron
Avatar

Send message
Joined: 10 Jan 01
Posts: 2
Credit: 1,823,416
RAC: 0
United States
Message 1131104 - Posted: 23 Jul 2011, 21:12:13 UTC

Since this week no downloads. I cannot connect to servers. The error is project may be down. i can upload finished results fine. but i cannot update requested by user, or download any new work.

basically i am out of Seti work except for 4 units i have left. My cida card is idle. I shouldn't have bought it since there is doing nothing to do.
ID: 1131104 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1131106 - Posted: 23 Jul 2011, 21:14:39 UTC - in response to Message 1130942.  


Yeah, something's stuck somewhere.
Both AP & MB result validation queues continue to grow, yet the work in progress is almost stagnant- even though network traffic has been maxed out for 24hrs. Both assimilator queues also continue to grow, even though all but 3 AP assimilators are shown as running.

And for some reason one of my systems has been unable to get any work for the last 8 hours or so- it requests it & all it gets is a "No tasks sent" message even though there has been no less than 150,000 Work Units ready to send during that period, and my other system has picked up work during that period.
Grant
Darwin NT
ID: 1131106 · Report as offensive
Profile Donald L. Johnson
Avatar

Send message
Joined: 5 Aug 02
Posts: 8240
Credit: 14,654,533
RAC: 20
United States
Message 1131162 - Posted: 23 Jul 2011, 23:47:00 UTC - in response to Message 1131106.  
Last modified: 23 Jul 2011, 23:47:36 UTC

And for some reason one of my systems has been unable to get any work for the last 8 hours or so- it requests it & all it gets is a "No tasks sent" message even though there has been no less than 150,000 Work Units ready to send during that period, and my other system has picked up work during that period.

IIRC, the "No tasks sent" message has nothing to do with the number of Tasks ready to send, or the 100-task download feeder. Rather, it is caused by a server-side limit on the number of tasks that can be sent to your computer in one 24-hour day.

I don't even recall all the things that can affect these limits (there is a thread here where Joe and Richard explained them), but I do recall that there are separate limits for AP and the various MB types (6.03, cuda, cuda23, cuda-fermi, and Anonymous Platform), and that errors and invalid results can reduce the limits.
Donald
Infernal Optimist / Submariner, retired
ID: 1131162 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1131171 - Posted: 24 Jul 2011, 0:11:07 UTC - in response to Message 1131162.  

And for some reason one of my systems has been unable to get any work for the last 8 hours or so- it requests it & all it gets is a "No tasks sent" message even though there has been no less than 150,000 Work Units ready to send during that period, and my other system has picked up work during that period.

IIRC, the "No tasks sent" message has nothing to do with the number of Tasks ready to send, or the 100-task download feeder. Rather, it is caused by a server-side limit on the number of tasks that can be sent to your computer in one 24-hour day.

I don't even recall all the things that can affect these limits (there is a thread here where Joe and Richard explained them), but I do recall that there are separate limits for AP and the various MB types (6.03, cuda, cuda23, cuda-fermi, and Anonymous Platform), and that errors and invalid results can reduce the limits.

"No tasks sent" means what it says on the tin. No tasks were sent. For the reason why, you have to look at the surrounding messages.

The commonest is "not requesting new tasks". If you don't ask, you don't get. You would expect to be seeing "requesting new tasks for CPU" (or NVIDIA GPU, or whatever).

Others, like the maximum number of tasks per day, would be a message including the word "quota". It's been a long time since I've seen one of those, so I can't quote it verbatim. Or a message about a computer which has "reached the maximum number of tasks in progress" (we had one of those on the board recently, so the wording should be fairly close).

In short, read (or post) the complete sequence of messages relating to the scheduler request in question, and the explanation should become clearer.
ID: 1131171 · Report as offensive
Morten Ross
Volunteer tester
Avatar

Send message
Joined: 30 Apr 01
Posts: 183
Credit: 385,664,915
RAC: 0
Norway
Message 1131175 - Posted: 24 Jul 2011, 0:29:30 UTC - in response to Message 1131171.  

Here is one example:

24/07/2011 02:25:53 SETI@home Sending scheduler request: To fetch work.
24/07/2011 02:25:53 SETI@home Reporting 1 completed tasks, requesting new tasks for CPU and GPU
24/07/2011 02:26:01 SETI@home Scheduler request completed: got 0 new tasks
24/07/2011 02:26:01 SETI@home Message from server: No tasks sent
24/07/2011 02:26:01 SETI@home Message from server: No tasks are available for Astropulse v5
24/07/2011 02:26:01 SETI@home Message from server: No tasks are available for Astropulse v505

Morten Ross
ID: 1131175 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1131193 - Posted: 24 Jul 2011, 1:38:15 UTC - in response to Message 1131171.  
Last modified: 24 Jul 2011, 1:47:02 UTC

In short, read (or post) the complete sequence of messages relating to the scheduler request in question, and the explanation should become clearer.

No clearer.
According to my account page, this machine has returned 1147 valid CUD tasks.

24/07/2011 10:48:45 | SETI@home | Sending scheduler request: To fetch work.
24/07/2011 10:48:45 | SETI@home | Reporting 1 completed tasks, requesting new tasks for NVIDIA GPU
24/07/2011 10:48:49 | SETI@home | Scheduler request completed: got 0 new tasks
24/07/2011 10:48:49 | SETI@home | No tasks sent


24/07/2011 10:53:54 | SETI@home | Sending scheduler request: To fetch work.
24/07/2011 10:53:54 | SETI@home | Reporting 2 completed tasks, requesting new tasks for NVIDIA GPU
24/07/2011 10:53:59 | SETI@home | Scheduler request completed: got 0 new tasks
24/07/2011 10:53:59 | SETI@home | No tasks sent


Regardless of CPU or GPU work request, the response is the same- No tasks sent.
About 1 in 20 requests result in work being allocated.
Last night, work was being allocated with almost every request (from 2-20 WU). My other system is also getting No tasks sent messages, but it's about 1 in 3 that get that result.

Both systems are running Lunatics current applications, both are CPU & CUDA crunchers, both are connected to the same router/modem.
The only major difference is BOINC versions.
The one getting work is v6.10.58, the one not getting work is v6.12.33. It was running 6.12.26 & i thought i'd give the new one a go, no improvement.


The one thing i've noticed between the versions is the current version backs off early & backs off a lot on missed requests for work. The work fetch deferred for & interval when no work is allocated on a request immediately go to 10 minutes. Miss again & it can go as high as 30min.

I think there is a problem with work being allocated by the server- for whatever reason it's not available at the rate it was yesterday. And so machines that ask often are more likey to score some than mmachines that don't.



EDIT- the difference in BOINC versions shows up in my account info. I'm set for a 5 day cache.
The older version CUDA turnaround time is 4.5 days.
The current version CUDA turnaround time is 2.2 days.
Grant
Darwin NT
ID: 1131193 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1131194 - Posted: 24 Jul 2011, 1:49:31 UTC - in response to Message 1131193.  


Here're the messages for the last 30 min.

24/07/2011 10:48:45 | SETI@home | Sending scheduler request: To fetch work.
24/07/2011 10:48:45 | SETI@home | Reporting 1 completed tasks, requesting new tasks for NVIDIA GPU
24/07/2011 10:48:49 | SETI@home | Scheduler request completed: got 0 new tasks
24/07/2011 10:48:49 | SETI@home | No tasks sent
24/07/2011 10:51:10 | SETI@home | Computation for task 21mr11ag.31185.16427.15.10.70_0 finished
24/07/2011 10:51:10 | SETI@home | Starting task 21mr11ag.31185.16427.15.10.67_0 using setiathome_enhanced version 610
24/07/2011 10:51:13 | SETI@home | Started upload of 21mr11ag.31185.16427.15.10.70_0_0
24/07/2011 10:51:18 | SETI@home | Finished upload of 21mr11ag.31185.16427.15.10.70_0_0
24/07/2011 10:52:08 | SETI@home | Computation for task 21mr11ag.31185.16427.15.10.68_0 finished
24/07/2011 10:52:08 | SETI@home | Starting task 21mr11ag.31185.16427.15.10.65_0 using setiathome_enhanced version 610
24/07/2011 10:52:11 | SETI@home | Started upload of 21mr11ag.31185.16427.15.10.68_0_0
24/07/2011 10:52:19 | SETI@home | Finished upload of 21mr11ag.31185.16427.15.10.68_0_0
24/07/2011 10:53:54 | SETI@home | Sending scheduler request: To fetch work.
24/07/2011 10:53:54 | SETI@home | Reporting 2 completed tasks, requesting new tasks for NVIDIA GPU
24/07/2011 10:53:59 | SETI@home | Scheduler request completed: got 0 new tasks
24/07/2011 10:53:59 | SETI@home | No tasks sent
24/07/2011 11:05:27 | SETI@home | Computation for task 21mr11ag.31185.16427.15.10.67_0 finished
24/07/2011 11:05:27 | SETI@home | Starting task 21mr11ag.31185.16427.15.10.63_1 using setiathome_enhanced version 610
24/07/2011 11:05:29 | SETI@home | Started upload of 21mr11ag.31185.16427.15.10.67_0_0
24/07/2011 11:05:34 | SETI@home | Finished upload of 21mr11ag.31185.16427.15.10.67_0_0
24/07/2011 11:05:34 | SETI@home | Sending scheduler request: To fetch work.
24/07/2011 11:05:34 | SETI@home | Reporting 1 completed tasks, requesting new tasks for NVIDIA GPU
24/07/2011 11:05:39 | SETI@home | Scheduler request completed: got 0 new tasks
24/07/2011 11:05:39 | SETI@home | No tasks sent
24/07/2011 11:06:51 | SETI@home | Computation for task 21mr11ag.31185.16427.15.10.65_0 finished
24/07/2011 11:06:51 | SETI@home | Starting task 21mr11ag.31185.16427.15.10.60_1 using setiathome_enhanced version 610
24/07/2011 11:06:53 | SETI@home | Started upload of 21mr11ag.31185.16427.15.10.65_0_0
24/07/2011 11:07:02 | SETI@home | Finished upload of 21mr11ag.31185.16427.15.10.65_0_0
24/07/2011 11:10:44 | SETI@home | Sending scheduler request: To fetch work.
24/07/2011 11:10:44 | SETI@home | Reporting 1 completed tasks, requesting new tasks for NVIDIA GPU
24/07/2011 11:10:49 | SETI@home | Scheduler request completed: got 0 new tasks
24/07/2011 11:10:49 | SETI@home | No tasks sent
24/07/2011 11:17:55 | SETI@home | Sending scheduler request: To fetch work.
24/07/2011 11:17:55 | SETI@home | Requesting new tasks for NVIDIA GPU
24/07/2011 11:18:00 | SETI@home | Scheduler request completed: got 0 new tasks
24/07/2011 11:18:00 | SETI@home | No tasks sent
Grant
Darwin NT
ID: 1131194 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34258
Credit: 79,922,639
RAC: 80
Germany
Message 1131265 - Posted: 24 Jul 2011, 6:26:40 UTC


Enable shedop debug with cc_config.xml

It looks like this afterwards.

24.07.2011 08:16:01 SETI@home [sched_op] Starting scheduler request
24.07.2011 08:16:01 SETI@home Sending scheduler request: To fetch work.
24.07.2011 08:16:01 SETI@home Requesting new tasks for ATI GPU
24.07.2011 08:16:01 SETI@home [sched_op] CPU work request: 0.00 seconds; 0.00 CPUs
24.07.2011 08:16:01 SETI@home [sched_op] ATI GPU work request: 1450782.52 seconds; 0.00 GPUs
24.07.2011 08:16:04 SETI@home Scheduler request completed: got 0 new tasks
24.07.2011 08:16:04 SETI@home [sched_op] Server version 613
24.07.2011 08:16:04 SETI@home No tasks sent
24.07.2011 08:16:04 SETI@home No tasks are available for Astropulse v505
24.07.2011 08:16:04 SETI@home No tasks are available for the applications you have selected.
24.07.2011 08:16:04 SETI@home Project requested delay of 303 seconds
24.07.2011 08:16:04 SETI@home [sched_op] Deferring communication for 5 min 3 sec
24.07.2011 08:16:04 SETI@home [sched_op] Reason: requested by project



With each crime and every kindness we birth our future.
ID: 1131265 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1131275 - Posted: 24 Jul 2011, 6:56:05 UTC - in response to Message 1131265.  
Last modified: 24 Jul 2011, 6:57:17 UTC

Whatever it is, it's a problem.
In the last 5 1/2 hours i've only had 11 successfull requests for work on the machine experiencing this issue. While a couple of them resulted in 20+ Work Units, others only got 1 or 2. End result- my cache is shrinking again, not growing.

At the moment there is a dip in network traffic- and it's not because of full caches. Both of my systems have requested work & both have received "No tasks sent" messages in response- that's 5 requests for work from one system & 4 requests fom the other. All while the network traffic is at a lull.
If caches were full, then there would be no demand for work, then there should be plenty of work ready to be dished out- no less than 200,00 Work Units for several houtrs now. So a request for work should result in plenty. Yet it results in none.

Possibly the feeder is blocked? There's work there, i need it, but it's just not getting through. Last night almost every request resulted in work being allocated; now almost every request results in no work being allocated.
Grant
Darwin NT
ID: 1131275 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34258
Credit: 79,922,639
RAC: 80
Germany
Message 1131276 - Posted: 24 Jul 2011, 7:00:05 UTC

Did you try ipconfig /flushdns.

Just to make sure.




With each crime and every kindness we birth our future.
ID: 1131276 · Report as offensive
Profile Slavac
Volunteer tester
Avatar

Send message
Joined: 27 Apr 11
Posts: 1932
Credit: 17,952,639
RAC: 0
United States
Message 1131282 - Posted: 24 Jul 2011, 7:26:29 UTC - in response to Message 1131276.  

I'm having a very hard time getting work from the feeder.

Does anyone know a way to get a substantial number of AP tasks from the feeder. I've got my CPU running AP solely and my GPU churning out AP tasks on occasion, but I can never get more than 100 at a given time (total) in my cache.


Executive Director GPU Users Group Inc. -
brad@gpuug.org
ID: 1131282 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1131286 - Posted: 24 Jul 2011, 7:37:52 UTC - in response to Message 1131276.  

Well my 3 have fed again so get in there people before they want to feed again. :D

Cheers.
ID: 1131286 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1131287 - Posted: 24 Jul 2011, 7:38:15 UTC - in response to Message 1131276.  

Did you try ipconfig /flushdns.

Just to make sure.


Just had to re-boot (updates), still borked.
No problems contacting the servers- just in getting them to allocate work to me when it's requested.
Grant
Darwin NT
ID: 1131287 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34258
Credit: 79,922,639
RAC: 80
Germany
Message 1131288 - Posted: 24 Jul 2011, 7:47:54 UTC - in response to Message 1131282.  

I'm having a very hard time getting work from the feeder.

Does anyone know a way to get a substantial number of AP tasks from the feeder. I've got my CPU running AP solely and my GPU churning out AP tasks on occasion, but I can never get more than 100 at a given time (total) in my cache.


24.07.2011 09:44:03 SETI@home Sending scheduler request: To fetch work.
24.07.2011 09:44:03 SETI@home Reporting 1 completed tasks, requesting new tasks for ATI GPU
24.07.2011 09:44:03 SETI@home [sched_op] CPU work request: 0.00 seconds; 0.00 CPUs
24.07.2011 09:44:03 SETI@home [sched_op] ATI GPU work request: 1461419.62 seconds; 0.00 GPUs
24.07.2011 09:44:40 SETI@home Computation for task 17ja11aa.14923.24607.9.10.97_1 finished
24.07.2011 09:44:40 SETI@home Starting 17ja11aa.14923.24607.9.10.86_0
24.07.2011 09:44:40 SETI@home Starting task 17ja11aa.14923.24607.9.10.86_0 using setiathome_enhanced version 603
24.07.2011 09:44:42 SETI@home Started upload of 17ja11aa.14923.24607.9.10.97_1_0
24.07.2011 09:44:48 SETI@home Finished upload of 17ja11aa.14923.24607.9.10.97_1_0
24.07.2011 09:45:58 SETI@home Scheduler request completed: got 49 new tasks
24.07.2011 09:45:58 SETI@home [sched_op] Server version 613
24.07.2011 09:45:58 SETI@home Project requested delay of 303 seconds
24.07.2011 09:45:58 SETI@home [sched_op] estimated total CPU task duration: 0 seconds
24.07.2011 09:45:58 SETI@home [sched_op] estimated total ATI GPU task duration: 646337 seconds
24.07.2011 09:45:58 SETI@home [sched_op] handle_scheduler_reply(): got ack for task 13mr11af.14318.22562.7.10.227.vlar_1
24.07.2011 09:45:58 SETI@home [sched_op] Deferring communication for 5 min 3 sec
24.07.2011 09:45:58 SETI@home [sched_op] Reason: requested by project
24.07.2011 09:46:00 SETI@home Started download of ap_16ja11ac_B2_P1_00176_20110723_06035.wu



With each crime and every kindness we birth our future.
ID: 1131288 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 . . . 10 · Next

Message boards : Number crunching : Panic Mode On (51) Server problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.