Panic Mode On (56) Server problems?

Message boards : Number crunching : Panic Mode On (56) Server problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · Next

AuthorMessage
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36873
Credit: 261,360,520
RAC: 489
Australia
Message 1157728 - Posted: 1 Oct 2011, 4:14:54 UTC - in response to Message 1157725.  

Yep, the Cricket Graph took another nose dive again. :(

Cheers.
ID: 1157728 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19407
Credit: 40,757,560
RAC: 67
United Kingdom
Message 1157743 - Posted: 1 Oct 2011, 5:27:40 UTC

Comms now back up, well I have uploaded, waiting to see if I can report/request. First attempt failed.

As I type reported ok, but no tasks available.
ID: 1157743 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 66362
Credit: 55,293,173
RAC: 49
United States
Message 1157752 - Posted: 1 Oct 2011, 5:40:08 UTC

I'm full as Seti said the following:

1314 SETI@home 9/30/2011 10:37:12 PM This computer has reached a limit on tasks in progress
Savoir-Faire is everywhere!
The T1 Trust, T1 Class 4-4-4-4 #5550, America's First HST

ID: 1157752 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19407
Credit: 40,757,560
RAC: 67
United Kingdom
Message 1157769 - Posted: 1 Oct 2011, 6:04:37 UTC - in response to Message 1157752.  

I'm full as Seti said the following:

1314 SETI@home 9/30/2011 10:37:12 PM This computer has reached a limit on tasks in progress

Congratulations, now plse, suspend comms, and let me get some, my cpu will start going cold in under 1 hour.
ID: 1157769 · Report as offensive
BetelgeuseFive Project Donor
Volunteer tester

Send message
Joined: 6 Jul 99
Posts: 158
Credit: 17,117,787
RAC: 19
Netherlands
Message 1157794 - Posted: 1 Oct 2011, 7:42:02 UTC

I am receiving some new tasks (although not enough to fill my cache). Problem is the estimated computation time for these new tasks. I'm running Lunatics optimized CUDA on my GT240 and the 'shorties' take slightly less than 5 minutes to complete. The new tasks I am receiving have an estimated computation time of only 58 seconds. This probably means they will error out after twice that time.
I know there have been server side changes and I was able to solve similar problems two weeks ago by removing the FLOPS entry from my app_info file.
Now something seems to have changed once again.
Is there anything I can do to prevent the tasks from being aborted because they take too long to process ? I would hate to waste the small number of tasks I do receive ...

Tom

ID: 1157794 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14679
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1157799 - Posted: 1 Oct 2011, 8:31:02 UTC - in response to Message 1157794.  

I am receiving some new tasks (although not enough to fill my cache). Problem is the estimated computation time for these new tasks. I'm running Lunatics optimized CUDA on my GT240 and the 'shorties' take slightly less than 5 minutes to complete. The new tasks I am receiving have an estimated computation time of only 58 seconds. This probably means they will error out after twice that time.
I know there have been server side changes and I was able to solve similar problems two weeks ago by removing the FLOPS entry from my app_info file.
Now something seems to have changed once again.
Is there anything I can do to prevent the tasks from being aborted because they take too long to process ? I would hate to waste the small number of tasks I do receive ...

Tom

Don't worry. Tasks only error out if they take ten times as long as expected.

The current estimates are, deliberately, five times too short, to avoid reaching that error limit. Just let them run normally. As soon as the first "58 second" task has completed - which will take the normal five minutes - BOINC will realise what's going on and adjust the estimates for the remaining new tasks. You will, temporarily, have a larger cache than you expected, but again - and again deliberately - the quota limit is in place to prevent things getting out of hand.
ID: 1157799 · Report as offensive
Profile S@NL Etienne Dokkum
Volunteer tester
Avatar

Send message
Joined: 11 Jun 99
Posts: 212
Credit: 43,822,095
RAC: 0
Netherlands
Message 1157808 - Posted: 1 Oct 2011, 8:42:31 UTC

well, at least I reported everything now, but as no AP's seem to get validated there are more problems on the way I suppose...

The machine I left running last night didn't report anything on it's own, but had to be "helped" this morning.

Still enough work though as I was so lucky to get almost 30 AP's thursday...
ID: 1157808 · Report as offensive
Profile S@NL Etienne Dokkum
Volunteer tester
Avatar

Send message
Joined: 11 Jun 99
Posts: 212
Credit: 43,822,095
RAC: 0
Netherlands
Message 1157810 - Posted: 1 Oct 2011, 8:46:26 UTC - in response to Message 1157794.  

I am receiving some new tasks (although not enough to fill my cache). Problem is the estimated computation time for these new tasks. I'm running Lunatics optimized CUDA on my GT240 and the 'shorties' take slightly less than 5 minutes to complete. The new tasks I am receiving have an estimated computation time of only 58 seconds. This probably means they will error out after twice that time.
I know there have been server side changes and I was able to solve similar problems two weeks ago by removing the FLOPS entry from my app_info file.
Now something seems to have changed once again.
Is there anything I can do to prevent the tasks from being aborted because they take too long to process ? I would hate to waste the small number of tasks I do receive ...

Tom


sort of similar problem here as I have 1 AP left from when the estimates were made too high. Now this WU has a est. time of 108 hours when it just takes 12 to 13 hours to process on my CPU. At least all the other WU's seem to be "back to normal" now...
ID: 1157810 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1157816 - Posted: 1 Oct 2011, 9:05:09 UTC - in response to Message 1157794.  
Last modified: 1 Oct 2011, 9:08:57 UTC

the 'shorties' take slightly less than 5 minutes to complete. The new tasks I am receiving have an estimated computation time of only 58 seconds.

For me the estimates are all over the place. The DCF is moving around between 0.7 & 1.5. As each GPU tasks complete, their ridiculously long completion times slowly drop, making the almost correct CPU times drop as well. They get down to about half of the actual completion time is when one finally completes & the estimates get bumped up; pushing the GPU task completion times to new heights of ridiculousness.
Hopefully Seti can stay up for the next few days & things will start to settle down.



Although things are still looking bit broken- doesn't look as though AP work is going out. And a lot of the requests for work result in none. Sometimes i get 1 or 2 WUs, occasionally i'll get 20+. But mostly it's "Project has no tasks available".
Grant
Darwin NT
ID: 1157816 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1157819 - Posted: 1 Oct 2011, 9:52:11 UTC


Now i'm not getting any response from the Scheduler.
Grant
Darwin NT
ID: 1157819 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1157826 - Posted: 1 Oct 2011, 10:18:35 UTC

Everything's still working at the moment....
The kitties are in there scrapping for anything the servers can send.
Not getting a lot, but enough to keep things a bit warmer in the crunching den.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1157826 · Report as offensive
BetelgeuseFive Project Donor
Volunteer tester

Send message
Joined: 6 Jul 99
Posts: 158
Credit: 17,117,787
RAC: 19
Netherlands
Message 1157828 - Posted: 1 Oct 2011, 10:22:38 UTC - in response to Message 1157819.  

I had the same problem around the time you posted your message.
Things seem to be working again. I just received 40 (!) new workunits and they downloaded really fast (less than 1.5 minutes for all 40 of them). No big surprise as the cricket graph isn't maxed out, but still nice to see ...



Now i'm not getting any response from the Scheduler.


ID: 1157828 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1157829 - Posted: 1 Oct 2011, 10:25:32 UTC - in response to Message 1157819.  

Now i'm not getting any response from the Scheduler.


Now it's back again.
But for a while there it wasn't.

1/10/2011 19:14:59 SETI@home Sending scheduler request: To fetch work.
1/10/2011 19:14:59 SETI@home Reporting 4 completed tasks, requesting new tasks for CPU and GPU
1/10/2011 19:15:22 Project communication failed: attempting access to reference site
1/10/2011 19:15:22 SETI@home Scheduler request failed: Couldn't connect to server
1/10/2011 19:15:25 Internet access OK - project servers may be temporarily down.
1/10/2011 19:16:22 SETI@home Sending scheduler request: To fetch work.
1/10/2011 19:16:22 SETI@home Reporting 4 completed tasks, requesting new tasks for CPU and GPU
1/10/2011 19:16:44 Project communication failed: attempting access to reference site
1/10/2011 19:16:44 SETI@home Scheduler request failed: Couldn't connect to server
1/10/2011 19:16:46 Internet access OK - project servers may be temporarily down.
1/10/2011 19:17:44 SETI@home Sending scheduler request: To fetch work.
1/10/2011 19:17:44 SETI@home Reporting 6 completed tasks, requesting new tasks for CPU and GPU
1/10/2011 19:18:40 SETI@home Scheduler request failed: HTTP internal server error
1/10/2011 19:19:40 SETI@home Sending scheduler request: To fetch work.
1/10/2011 19:19:40 SETI@home Reporting 6 completed tasks, requesting new tasks for CPU and GPU
1/10/2011 19:20:00 SETI@home Computation for task 17ap11ah.22009.16427.6.10.174_0 finished
1/10/2011 19:20:16 Project communication failed: attempting access to reference site
1/10/2011 19:20:16 SETI@home Scheduler request failed: Failure when receiving data from the peer
1/10/2011 19:20:18 Internet access OK - project servers may be temporarily down.

Now it's mostly "Project has no tasks available"
Grant
Darwin NT
ID: 1157829 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36873
Credit: 261,360,520
RAC: 489
Australia
Message 1157831 - Posted: 1 Oct 2011, 10:47:10 UTC - in response to Message 1157829.  

The main message for my 3 PC's for the last 4-6 hours has been, "This computer has reached a limit on tasks in progress", with the occasional 1-10 tasks being received every 4th or 5th request.

Cheers.

ID: 1157831 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22540
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1157832 - Posted: 1 Oct 2011, 10:52:25 UTC

S@H has been running with a cap on tasks in progress (in other words a limit on the number of tasks you can have on each cruncher) for some time.
Each cruncher is allowed 50 per CPU core, and 400 per GPU.

(My figures might be wrong, I deduced them from the number of tasks on my crunchers.)
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1157832 · Report as offensive
Profile Spectrum
Avatar

Send message
Joined: 14 Jun 99
Posts: 468
Credit: 53,129,336
RAC: 0
Australia
Message 1157833 - Posted: 1 Oct 2011, 10:54:01 UTC

Well after a fair period of no uploads or downloads it seems that the system has settled and all the gripes can be forgotten until the next time, no expectations no regrets lets all do it for the one in a bazillion chance to say we have proven that there is life out there beyond our little blue planet.

Keep on crunching and greetings to all on our little planet called Earth.
ID: 1157833 · Report as offensive
__W__
Avatar

Send message
Joined: 28 Mar 09
Posts: 116
Credit: 5,943,642
RAC: 0
Germany
Message 1157834 - Posted: 1 Oct 2011, 11:14:02 UTC

Someone must have kicked the routers at HE very hard - yiiihhha
Just got 40 WUs and downloaded them in under 2 minutes, in spite of cricket nearly maxed out - and pinging the servers is as fast as never before (from my point of the world) :-) .

__W__
_______________________________________________________________________________
ID: 1157834 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14679
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1157836 - Posted: 1 Oct 2011, 11:18:35 UTC - in response to Message 1157831.  

The main message for my 3 PC's for the last 4-6 hours has been, "This computer has reached a limit on tasks in progress", with the occasional 1-10 tasks being received every 4th or 5th request.

Cheers.

Each one of your three hosts shows either 449 or 450 tasks in progress. That's the current limit for CPU and GPU tasks combined. Subject to the usual caveats about hitting the feeder when it has suitable tasks available, you'll get a fresh task in exchange for each completed task you return.
ID: 1157836 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36873
Credit: 261,360,520
RAC: 489
Australia
Message 1157844 - Posted: 1 Oct 2011, 11:36:17 UTC - in response to Message 1157836.  

The main message for my 3 PC's for the last 4-6 hours has been, "This computer has reached a limit on tasks in progress", with the occasional 1-10 tasks being received every 4th or 5th request.

Cheers.

Each one of your three hosts shows either 449 or 450 tasks in progress. That's the current limit for CPU and GPU tasks combined. Subject to the usual caveats about hitting the feeder when it has suitable tasks available, you'll get a fresh task in exchange for each completed task you return.

Yes it's certainly nowhere near my usual cache capacity but then again I also have quite a bit of CPU work from backup projects for a safety buffer (so far it only seems to be CPU work that I run out of, the GPU work has remained SETI only).

Cheers.
ID: 1157844 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1157908 - Posted: 1 Oct 2011, 15:33:02 UTC

Somebody must be in the lab, the scheduling server is now showing as disabled.


PROUD MEMBER OF Team Starfire World BOINC
ID: 1157908 · Report as offensive
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · Next

Message boards : Number crunching : Panic Mode On (56) Server problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.