Panic Mode On (83) Server Problems?

Message boards : Number crunching : Panic Mode On (83) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 21 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1357852 - Posted: 17 Apr 2013, 18:06:16 UTC - in response to Message 1357827.  

Right now, there is AP work available to split, but for some reason all the AP splitters are in disabled status.

As are the ap_assimilators. Only the ap-validator4 seems to be running. This has been the case since Tuesday afternoon. Maybe a problem with the AstroPulse science database. And since Matt did not post his usual post-outage update yesterday, they may still be working on solving that problem....

AP assimilators are still down- things are really building up there.
As it is, the MB assimilators are only slowly getting that backlog under control.
Grant
Darwin NT
ID: 1357852 · Report as offensive
Profile Donald L. Johnson
Avatar

Send message
Joined: 5 Aug 02
Posts: 8240
Credit: 14,654,533
RAC: 20
United States
Message 1357931 - Posted: 18 Apr 2013, 3:16:27 UTC - in response to Message 1357852.  

Right now, there is AP work available to split, but for some reason all the AP splitters are in disabled status.

As are the ap_assimilators. Only the ap-validator4 seems to be running. This has been the case since Tuesday afternoon. Maybe a problem with the AstroPulse science database. And since Matt did not post his usual post-outage update yesterday, they may still be working on solving that problem....

AP assimilators are still down- things are really building up there.
As it is, the MB assimilators are only slowly getting that backlog under control.

Looks like Astropulse is back in business.
[As of 18 Apr 2013, 3:00:05 UTC] all AP functions are running, 7 AP splitters splitting, and about 500 results ready to send.


Donald
Infernal Optimist / Submariner, retired
ID: 1357931 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1358171 - Posted: 18 Apr 2013, 17:58:12 UTC - in response to Message 1357931.  
Last modified: 18 Apr 2013, 17:59:03 UTC

Ready-to-send buffer is shrinking, no new MB work being split.
Server Status page shows all MB splitters offline.
Grant
Darwin NT
ID: 1358171 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1358182 - Posted: 18 Apr 2013, 18:10:05 UTC - in response to Message 1358171.  
Last modified: 18 Apr 2013, 18:10:23 UTC

Ready-to-send buffer is shrinking, no new MB work being split.
Server Status page shows all MB splitters offline.

I hope that doesn't mean all my GPUs will be sucking wind whilst I am gone to work today.....
Hopefully da boyz in da lab are just playing about with some thingys and nothing's broken.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1358182 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1358193 - Posted: 18 Apr 2013, 18:24:21 UTC
Last modified: 18 Apr 2013, 18:24:36 UTC

And I see that the last server status page update shows some MB splitters back at work.

Meowsmile.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1358193 · Report as offensive
Tom*

Send message
Joined: 12 Aug 11
Posts: 127
Credit: 20,769,223
RAC: 9
United States
Message 1358194 - Posted: 18 Apr 2013, 18:24:23 UTC
Last modified: 18 Apr 2013, 18:24:40 UTC

Since all MB splitters are disabled but the pfb splitter is running
and the stats show
Results ready to send 217,822 258 8m
Current result creation rate 2.2091/sec 1.6656/sec 5m

Are we currently processing the more sensitive MB's?

if not who is splitting MB's
ID: 1358194 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1358210 - Posted: 18 Apr 2013, 18:52:37 UTC

Maybe this is some of that new splitter code that was talked about months ago going into place. At the moment all but 3 splitters are online and they are being generated at 36.9398/sec & 1.7282/sec.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1358210 · Report as offensive
Profile RottenMutt
Avatar

Send message
Joined: 15 Mar 01
Posts: 1011
Credit: 230,314,058
RAC: 0
United States
Message 1358379 - Posted: 19 Apr 2013, 5:13:25 UTC - in response to Message 1358210.  

queue's are draining, what's up with that...
ID: 1358379 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1358382 - Posted: 19 Apr 2013, 5:26:17 UTC

The only "production" splitters running are on Lando and Vader, I guess one of those two is having a bit of a problem :-(

Oh well, reserve projects here I come...
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1358382 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1358394 - Posted: 19 Apr 2013, 6:28:51 UTC - in response to Message 1358382.  
Last modified: 19 Apr 2013, 6:29:18 UTC

Been getting lots of "Project has no tasks" messages. Notice the splitters just aren't cranking up the output as the number that are ready-to-send drops.
Also been getting a few downloads timing out, or taking a while to start downloading.
Grant
Darwin NT
ID: 1358394 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1358400 - Posted: 19 Apr 2013, 6:38:00 UTC - in response to Message 1358394.  

Been getting lots of "Project has no tasks" messages. Notice the splitters just aren't cranking up the output as the number that are ready-to-send drops.
Also been getting a few downloads timing out, or taking a while to start downloading.

Yup...
The MB splitters are not keeping up at all for some reason.
Hit and miss here..
My top rig totally ran out of GPU work, which might have taken an hour or so, given the shorties goin' on. And then, it had the fortune of getting a 98 WU hit, which will shore it back up for a while.



"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1358400 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1358401 - Posted: 19 Apr 2013, 6:38:03 UTC

Currently, uploads and reporting failing. Downloads get "projects has no tasks available"
ID: 1358401 · Report as offensive
Lionel

Send message
Joined: 25 Mar 00
Posts: 680
Credit: 563,640,304
RAC: 597
Australia
Message 1358432 - Posted: 19 Apr 2013, 7:12:10 UTC - in response to Message 1358401.  

we're dry ... ready to send was at 115 ...

this is why we need to lift the limits so that we can continue to work through while they sort out issues that arise ...
ID: 1358432 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1358438 - Posted: 19 Apr 2013, 7:17:03 UTC - in response to Message 1358432.  

we're dry ... ready to send was at 115 ...

this is why we need to lift the limits so that we can continue to work through while they sort out issues that arise ...

Yeah, I know....
I'd truly love it if they could.
Unless it would cause the DB to crash.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1358438 · Report as offensive
alan
Avatar

Send message
Joined: 18 Feb 00
Posts: 131
Credit: 401,606
RAC: 0
United Kingdom
Message 1358446 - Posted: 19 Apr 2013, 7:47:28 UTC

The limits are there to prevent issues from arising in the first place. I fail to see how raising them would help the project at all.

The project scope is clearly stated, to use spare cpu cycles that would otherwise be "wasted" running screensavers or other idle tasks. It was never intended to cope with dedicated computers or specially built crunching farms with multiple GPU's and demands for massive caches to sustain them running 24x7.

By constantly demanding that the limits be raised you are making your need to execute as many SETI tasks as possible more important than the well-being of the project itself.

The fix to this problem is larger workunits which will reduce the number of "in progress" entries in the database.
ID: 1358446 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1358452 - Posted: 19 Apr 2013, 8:00:11 UTC - in response to Message 1358446.  

The limits are there to prevent issues from arising in the first place. I fail to see how raising them would help the project at all.

The project scope is clearly stated, to use spare cpu cycles that would otherwise be "wasted" running screensavers or other idle tasks. It was never intended to cope with dedicated computers or specially built crunching farms with multiple GPU's and demands for massive caches to sustain them running 24x7.

By constantly demanding that the limits be raised you are making your need to execute as many SETI tasks as possible more important than the well-being of the project itself.

The fix to this problem is larger workunits which will reduce the number of "in progress" entries in the database.

+1
ID: 1358452 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1358454 - Posted: 19 Apr 2013, 8:05:13 UTC - in response to Message 1358446.  

The limits are there to prevent issues from arising in the first place. I fail to see how raising them would help the project at all.

The project scope is clearly stated, to use spare cpu cycles that would otherwise be "wasted" running screensavers or other idle tasks. It was never intended to cope with dedicated computers or specially built crunching farms with multiple GPU's and demands for massive caches to sustain them running 24x7.

By constantly demanding that the limits be raised you are making your need to execute as many SETI tasks as possible more important than the well-being of the project itself.

The fix to this problem is larger workunits which will reduce the number of "in progress" entries in the database.


Alan...
Please don't try to remind some of us 'what the original scope of the project' was.
We know that. And we all (some of us, anyway,) know that the capacity of the project has thankfully grown far beyond the original concept.
No, in it's inception, it was never envisioned that vast numbers of like minded individuals would flock to the calling. But we have.
And the project has tried to it's best ability to respond.
They have done rather well, considering the meager funding...in most latter years, by it's own participants.

Matt has indicated that since the move to the colocation facility, we may well be outstripping the capacity to gather data by our massive ability to process it.

I see this as a great thing for the Seti project. And proof that the whole Boinc distributed computing concept has indeed come to fruition. We, as small users, have amassed such an enormous amount of computing power that the project might have some trouble supplying enough data for us to process......who would have dreamed?
Not Eric or Matt, I assure you.

As I have suggested, but am not in the direct loop right now, I suspect that since the relocation of the servers and the ability of the project to supply data to the crunching base, one of the upshots is that the GPUUG fundraisers upcoming might possibly be for a dedicated server to process the accumulated results.

The kitties would still like to get and process as much work as possible to further the initial goal of this project. Which is, of course, to prove that we are not alone.

I believe that every day we get closer to proving that reality.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1358454 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1358456 - Posted: 19 Apr 2013, 8:10:30 UTC

All task now reported and new downloaded! Now idea why, just lucky I guess
ID: 1358456 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1358459 - Posted: 19 Apr 2013, 8:13:28 UTC - in response to Message 1358456.  

All task now reported and new downloaded! Now idea why, just lucky I guess

Yeah...dicey right now. Servers are doing all they can with the dilithium crystals available.....
I have my best cruncher sucking GPU fumes right now, but the kitties shall keep sniffing for more WUs.

I got 9 rigs asking, so one or more of them is bound to hit Seti paydirt.

LOL...
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1358459 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1358464 - Posted: 19 Apr 2013, 8:28:56 UTC - in response to Message 1358454.  
Last modified: 19 Apr 2013, 8:29:38 UTC

The limits are there to prevent issues from arising in the first place. I fail to see how raising them would help the project at all.

The project scope is clearly stated, to use spare cpu cycles that would otherwise be "wasted" running screensavers or other idle tasks. It was never intended to cope with dedicated computers or specially built crunching farms with multiple GPU's and demands for massive caches to sustain them running 24x7.

By constantly demanding that the limits be raised you are making your need to execute as many SETI tasks as possible more important than the well-being of the project itself.

The fix to this problem is larger workunits which will reduce the number of "in progress" entries in the database.


Alan...
Please don't try to remind some of us 'what the original scope of the project' was.
We know that. And we all (some of us, anyway,) know that the capacity of the project has thankfully grown far beyond the original concept.
No, in it's inception, it was never envisioned that vast numbers of like minded individuals would flock to the calling. But we have.
And the project has tried to it's best ability to respond.
They have done rather well, considering the meager funding...in most latter years, by it's own participants.

Matt has indicated that since the move to the colocation facility, we may well be outstripping the capacity to gather data by our massive ability to process it.

I see this as a great thing for the Seti project. And proof that the whole Boinc distributed computing concept has indeed come to fruition. We, as small users, have amassed such an enormous amount of computing power that the project might have some trouble supplying enough data for us to process......who would have dreamed?
Not Eric or Matt, I assure you.

As I have suggested, but am not in the direct loop right now, I suspect that since the relocation of the servers and the ability of the project to supply data to the crunching base, one of the upshots is that the GPUUG fundraisers upcoming might possibly be for a dedicated server to process the accumulated results.

The kitties would still like to get and process as much work as possible to further the initial goal of this project. Which is, of course, to prove that we are not alone.

I believe that every day we get closer to proving that reality.


Do you understand what I am saying, Alan?
I am not trying to put you down.
Just a little history from one who has been here a loooooong time.
And I know you have been, as well.

It's a success story, not a problem, as I see it.
Or perhaps a little of both.

Meow.
Mark.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1358464 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 21 · Next

Message boards : Number crunching : Panic Mode On (83) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.