Panic Mode On (59) Server problems?

Message boards : Number crunching : Panic Mode On (59) Server problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 12 · Next

AuthorMessage
Blake Bonkofsky
Volunteer tester
Avatar

Send message
Joined: 29 Dec 99
Posts: 617
Credit: 46,383,149
RAC: 0
United States
Message 1162476 - Posted: 15 Oct 2011, 4:20:16 UTC - in response to Message 1162469.  

The crickets are faltering. Still a lot of load, but not maxed. I seriously doubt everyone has hit the limit or gotten their fill. Looks like uploads are tapering off slightly at the same time as downloads.


I'm not so sure. We've been maxxed out for several days now, and have seriously low limits in place. Almost 4 million results in the field. Normally we are around, what, 6 million? I'd say everyone could be cached or limited...
ID: 1162476 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1162504 - Posted: 15 Oct 2011, 4:53:29 UTC - in response to Message 1162469.  

The crickets are faltering. Still a lot of load, but not maxed. I seriously doubt everyone has hit the limit or gotten their fill. Looks like uploads are tapering off slightly at the same time as downloads.

The reason i haven't got my fill is because of the limits, and the fact that the estimated completion times for my GPU work units will never get even remotely close to being accurate till the next stage of the recovery from the borked server code of some weeks ago.
Until the GPU estimates are close to correct there's no way of hitting the server GPU limits, let alone filling the cache. Especially with the hitting of the CPU limits blocking the receiving of any more GPU work.
Grant
Darwin NT
ID: 1162504 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36814
Credit: 261,360,520
RAC: 489
Australia
Message 1162513 - Posted: 15 Oct 2011, 5:05:50 UTC - in response to Message 1162504.  

I'm the opposite way around, mine are screaming for CPU work constantly.

Cheers
ID: 1162513 · Report as offensive
Kevin Olley

Send message
Joined: 3 Aug 99
Posts: 906
Credit: 261,085,289
RAC: 572
United Kingdom
Message 1162550 - Posted: 15 Oct 2011, 6:09:06 UTC - in response to Message 1162504.  

The crickets are faltering. Still a lot of load, but not maxed. I seriously doubt everyone has hit the limit or gotten their fill. Looks like uploads are tapering off slightly at the same time as downloads.

The reason i haven't got my fill is because of the limits, and the fact that the estimated completion times for my GPU work units will never get even remotely close to being accurate till the next stage of the recovery from the borked server code of some weeks ago.
Until the GPU estimates are close to correct there's no way of hitting the server GPU limits, let alone filling the cache. Especially with the hitting of the CPU limits blocking the receiving of any more GPU work.


I am getting up to the GPU limits when my CPU is running other than MB's, either AP's or CPND's.

Whenever I run MB's on CPU my "To Completion" times on GPU rise so high that I stop asking for GPU work even when using a 10 day cache.

The other possible problem I can see is that when the limits are removed there is a high risk of being swamped with CPU WU's, not a major problem unless too many of them are VLAR's, I can process VLAR's on GPU's but they can cause problems if I am also trying to use this machine at the same time.




Kevin


ID: 1162550 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1162552 - Posted: 15 Oct 2011, 6:13:00 UTC - in response to Message 1162513.  

I'm the opposite way around, mine are screaming for CPU work constantly.

Cheers

My i7 is always asking for CPU work, but the limit stops it from getting any until it's completed one.
Grant
Darwin NT
ID: 1162552 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1162567 - Posted: 15 Oct 2011, 7:53:17 UTC

Downloads are definitely screaming along though. Just got two new APs, each only took 18 seconds to download (~450KB/sec).
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1162567 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14679
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1162568 - Posted: 15 Oct 2011, 8:07:43 UTC - in response to Message 1162476.  

The crickets are faltering. Still a lot of load, but not maxed. I seriously doubt everyone has hit the limit or gotten their fill. Looks like uploads are tapering off slightly at the same time as downloads.

I'm not so sure. We've been maxxed out for several days now, and have seriously low limits in place. Almost 4 million results in the field. Normally we are around, what, 6 million? I'd say everyone could be cached or limited...

And we're getting nice crunchy workunits.

I've hardly seen a shorty for days - and the few I have picked up recently have been resends. When we're working on the mid-AR tasks from the September Arecibo recordings, the download pipe can supply them faster than we can crunch them. We became habituated to a flurry of shorty storms over the summer, and we've perhaps forgotten what normal service is like.
ID: 1162568 · Report as offensive
Blake Bonkofsky
Volunteer tester
Avatar

Send message
Joined: 29 Dec 99
Posts: 617
Credit: 46,383,149
RAC: 0
United States
Message 1162588 - Posted: 15 Oct 2011, 8:58:45 UTC - in response to Message 1162568.  

The crickets are faltering. Still a lot of load, but not maxed. I seriously doubt everyone has hit the limit or gotten their fill. Looks like uploads are tapering off slightly at the same time as downloads.

I'm not so sure. We've been maxxed out for several days now, and have seriously low limits in place. Almost 4 million results in the field. Normally we are around, what, 6 million? I'd say everyone could be cached or limited...

And we're getting nice crunchy workunits.

I've hardly seen a shorty for days
- and the few I have picked up recently have been resends. When we're working on the mid-AR tasks from the September Arecibo recordings, the download pipe can supply them faster than we can crunch them. We became habituated to a flurry of shorty storms over the summer, and we've perhaps forgotten what normal service is like.


Indeed. My cache size has nearly tripled, while the number of units has been stuck at the limits for over a week now. My i7 has almost 8 days of CPU, and 3.5 days of GPU work. During the mass shorty storm, that same 400/1200 limit was only good for about 3 days on the CPU, and only about 12-16 hours on the GPU.
ID: 1162588 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1162597 - Posted: 15 Oct 2011, 9:53:15 UTC - in response to Message 1162588.  

My i7 has almost 8 days of CPU, and 3.5 days of GPU work. During the mass shorty storm, that same 400/1200 limit was only good for about 3 days on the CPU, and only about 12-16 hours on the GPU.

If only that were the case for my systems.
With the present limits (and way off the mark GPU computation estimates) my i7 has 2.76 days of CPU work & only 0.79 of GPU. My E6600 has 3.1 days of CPU work & only 0.32 of GPU.
Grant
Darwin NT
ID: 1162597 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34380
Credit: 79,922,639
RAC: 80
Germany
Message 1162598 - Posted: 15 Oct 2011, 10:06:14 UTC
Last modified: 15 Oct 2011, 10:07:19 UTC

I dont even get the limit.
Should be 300 CPU and 400 GPU.

15.10.2011 11:53:28 SETI@home Sending scheduler request: To fetch work.
15.10.2011 11:53:28 SETI@home Reporting 1 completed tasks, requesting new tasks for CPU
15.10.2011 11:53:28 SETI@home [sched_op] CPU work request: 8109434.39 seconds; 0.00 CPUs
15.10.2011 11:53:28 SETI@home [sched_op] ATI GPU work request: 0.00 seconds; 0.00 GPUs
15.10.2011 11:53:42 SETI@home Scheduler request completed: got 0 new tasks
15.10.2011 11:53:42 SETI@home [sched_op] Server version 613
15.10.2011 11:53:42 SETI@home No tasks sent
15.10.2011 11:53:42 SETI@home This computer has reached a limit on tasks in progress
15.10.2011 11:53:42 SETI@home Project requested delay of 303 seconds
15.10.2011 11:53:42 SETI@home [sched_op] handle_scheduler_reply(): got ack for task 27se11af.29341.5798.7.10.116.vlar_1
15.10.2011 11:53:42 SETI@home [sched_op] Deferring communication for 5 min 3 sec
15.10.2011 11:53:42 SETI@home [sched_op] Reason: requested by project
15.10.2011 11:58:47 SETI@home [sched_op] Starting scheduler request
15.10.2011 11:58:47 SETI@home Sending scheduler request: To fetch work.
15.10.2011 11:58:47 SETI@home Requesting new tasks for ATI GPU
15.10.2011 11:58:47 SETI@home [sched_op] CPU work request: 0.00 seconds; 0.00 CPUs
15.10.2011 11:58:47 SETI@home [sched_op] ATI GPU work request: 817634.32 seconds; 0.00 GPUs
15.10.2011 11:59:08 SETI@home Scheduler request completed: got 15 new tasks
15.10.2011 11:59:08 SETI@home [sched_op] Server version 613

No CPU download for today.

I always get what i dont need.

So its not a day limit it seems to be a total limit but i never reach it.


With each crime and every kindness we birth our future.
ID: 1162598 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1162604 - Posted: 15 Oct 2011, 10:45:37 UTC - in response to Message 1162598.  

So its not a day limit it seems to be a total limit but i never reach it.

Ah, you did reach it.

15.10.2011 11:53:42 SETI@home This computer has reached a limit on tasks in progress
It said so right there.
Grant
Darwin NT
ID: 1162604 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34380
Credit: 79,922,639
RAC: 80
Germany
Message 1162614 - Posted: 15 Oct 2011, 11:44:20 UTC - in response to Message 1162604.  
Last modified: 15 Oct 2011, 11:45:11 UTC

So its not a day limit it seems to be a total limit but i never reach it.

Ah, you did reach it.

15.10.2011 11:53:42 SETI@home This computer has reached a limit on tasks in progress
It said so right there.


But its not true.

Only a little later.

15.10.2011 12:50:34 SETI@home Requesting new tasks for ATI GPU
15.10.2011 12:50:34 SETI@home [sched_op] CPU work request: 0.00 seconds; 0.00 CPUs
15.10.2011 12:50:34 SETI@home [sched_op] ATI GPU work request: 662931.66 seconds; 0.00 GPUs
15.10.2011 12:50:52 SETI@home Scheduler request completed: got 28 new tasks
15.10.2011 12:50:52 SETI@home [sched_op] Server version 613
15.10.2011 12:50:52 SETI@home Project requested delay of 303 seconds
15.10.2011 12:50:52 SETI@home [sched_op] estimated total CPU task duration: 0 seconds
15.10.2011 12:50:52 SETI@home [sched_op] estimated total ATI GPU task duration: 228716 seconds
15.10.2011 12:50:52 SETI@home [sched_op] Deferring communication for 5 min 3 sec
15.10.2011 12:50:52 SETI@home [sched_op] Reason: requested by project


With each crime and every kindness we birth our future.
ID: 1162614 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14679
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1162615 - Posted: 15 Oct 2011, 11:54:47 UTC - in response to Message 1162598.  

Read it in two sections.

15.10.2011 11:53:28 SETI@home [sched_op] CPU work request: 8109434.39 seconds; 0.00 CPUs
15.10.2011 11:53:42 SETI@home No tasks sent
15.10.2011 11:53:42 SETI@home This computer has reached a limit on tasks in progress

15.10.2011 11:58:47 SETI@home [sched_op] ATI GPU work request: 817634.32 seconds; 0.00 GPUs
15.10.2011 11:59:08 SETI@home Scheduler request completed: got 15 new tasks

There are separate limits for CPU (50 per core) and GPU (400 per card), as you yourself said.

I don't know what the specific limits for ATI GPUs are, and they may be different, since the project doesn't yet natively support ATI.

The question here is possibly: how many CPU tasks are currently cached on your machine?
ID: 1162615 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34380
Credit: 79,922,639
RAC: 80
Germany
Message 1162628 - Posted: 15 Oct 2011, 12:58:12 UTC - in response to Message 1162615.  

Read it in two sections.

15.10.2011 11:53:28 SETI@home [sched_op] CPU work request: 8109434.39 seconds; 0.00 CPUs
15.10.2011 11:53:42 SETI@home No tasks sent
15.10.2011 11:53:42 SETI@home This computer has reached a limit on tasks in progress

15.10.2011 11:58:47 SETI@home [sched_op] ATI GPU work request: 817634.32 seconds; 0.00 GPUs
15.10.2011 11:59:08 SETI@home Scheduler request completed: got 15 new tasks

There are separate limits for CPU (50 per core) and GPU (400 per card), as you yourself said.

I don't know what the specific limits for ATI GPUs are, and they may be different, since the project doesn't yet natively support ATI.

The question here is possibly: how many CPU tasks are currently cached on your machine?


257 CPU tasks atm.
Not 300.
And i never get 400 for the GPU.
Mostly between 300 and 330.
I´ve seen other ATI hosts with more than 800 units in cache with only a quad core and single GPU.

I´m almost certain the limits doesn´t work like it should.
Nothing new but worth to mention IMHO.



With each crime and every kindness we birth our future.
ID: 1162628 · Report as offensive
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 1162652 - Posted: 15 Oct 2011, 15:57:04 UTC

Wonder what happened with the MB & AP validators on the status page, at least now they're not showing in the list...
ID: 1162652 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1162658 - Posted: 15 Oct 2011, 16:15:15 UTC - in response to Message 1162652.  

Wonder what happened with the MB & AP validators on the status page, at least now they're not showing in the list...

Huh?
Last update I see is 16:00 and they're all there. AP 3 not running unfortunately, but it is there.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1162658 · Report as offensive
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 1162663 - Posted: 15 Oct 2011, 16:31:25 UTC - in response to Message 1162658.  

Wonder what happened with the MB & AP validators on the status page, at least now they're not showing in the list...

Huh?
Last update I see is 16:00 and they're all there. AP 3 not running unfortunately, but it is there.

Yes they're back, but I'm not the only one noticing they weren't for a time :)
ID: 1162663 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1162664 - Posted: 15 Oct 2011, 16:32:47 UTC - in response to Message 1162663.  

Wonder what happened with the MB & AP validators on the status page, at least now they're not showing in the list...

Huh?
Last update I see is 16:00 and they're all there. AP 3 not running unfortunately, but it is there.

Yes they're back, but I'm not the only one noticing they weren't for a time :)

Saw that....
Don't recall that ever happening before.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1162664 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1162679 - Posted: 15 Oct 2011, 17:26:23 UTC - in response to Message 1162677.  

Wonder what happened with the MB & AP validators on the status page, at least now they're not showing in the list...

Huh?
Last update I see is 16:00 and they're all there. AP 3 not running unfortunately, but it is there.

Yes they're back, but I'm not the only one noticing they weren't for a time :)

Saw that....
Don't recall that ever happening before.


And now AP 3 is running finally, at least it is green. Let's see if it actually is working too at the next update.


And as of 17:20 it is showing down again.
Meowsigh.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1162679 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1162710 - Posted: 15 Oct 2011, 18:50:27 UTC

Time to raise the kibble limits!

Meow meow meow!
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1162710 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 12 · Next

Message boards : Number crunching : Panic Mode On (59) Server problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.