Panic Mode On (108) Server Problems?

Message boards : Number crunching : Panic Mode On (108) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 24 · 25 · 26 · 27 · 28 · 29 · Next

AuthorMessage
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1905216 - Posted: 6 Dec 2017, 23:13:23 UTC - in response to Message 1905214.  

True there is the reset button ... if the client cache is empty, a reset would do it.
ID: 1905216 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1905219 - Posted: 6 Dec 2017, 23:26:43 UTC - in response to Message 1905212.  

Wed 06 Dec 2017 10:52:17 PM EET | SETI@home | Project has no tasks available
Seem like none have been splut to my machine. (split, splat splut :) )
If you let loose your pile of ghosted tasks you would have a better chance of getting more :D


Manually releasing ghosts 20 at a time. Yeah.
Give me a release ghosts button.


. . I'll vote for one of those ......

Stephen

:)
ID: 1905219 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1905221 - Posted: 6 Dec 2017, 23:28:22 UTC - in response to Message 1905216.  

True there is the reset button ... if the client cache is empty, a reset would do it.


. . You would think ... but no :(

Stephen

:(
ID: 1905221 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1905244 - Posted: 7 Dec 2017, 0:57:50 UTC

Two rigs, 400 WUs "my cache runneth over......"

"Sour Grapes make a bitter Whine." <(0)>
ID: 1905244 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36774
Credit: 261,360,520
RAC: 489
Australia
Message 1905260 - Posted: 7 Dec 2017, 2:33:39 UTC

Damn there's some noisy guppies getting about ATM. :-(

Cheers.
ID: 1905260 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1905267 - Posted: 7 Dec 2017, 3:16:04 UTC - in response to Message 1905260.  

Not just guppies. Arecibo VLAR's also.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1905267 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1668
Credit: 623,086,772
RAC: 156
Finland
Message 1905280 - Posted: 7 Dec 2017, 4:43:40 UTC - in response to Message 1905213.  

Manually releasing ghosts 20 at a time. Yeah.
Give me a release ghosts button.
It's not that bad :) It recovers 20 if it can. It releases what it can't recover i.e. old ones.

EDIT: I have wondered how you ghost so many ... is it because of corrupt client_state because of the large amount of tasks it is trying to handle?


Sometimes one of the GPUs goes to an error state and all started apps begin to say 'can not determine number of CPUs' and the tasks error out. If I hit reset project then before they are uploaded they become ghosts. That is my explanation.
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1905280 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1905296 - Posted: 7 Dec 2017, 6:45:11 UTC - in response to Message 1905260.  

Damn there's some noisy guppies getting about ATM. :-(

Cheers.


. . Quite a few in fact ...

Stephen

:)
ID: 1905296 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 1905298 - Posted: 7 Dec 2017, 6:53:24 UTC
Last modified: 7 Dec 2017, 6:55:06 UTC

MB received-in-the-last-hour has hit 115,000.
That is incredibly high.

Is it just a matter of noisy WUs, or has some configuration value for the splitters gotten messed up?
Grant
Darwin NT
ID: 1905298 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22527
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1905300 - Posted: 7 Dec 2017, 7:50:04 UTC

I had a whole pile of "crash and burn" tasks just after the splitters came back. From memory they were all GUPPI from one target star. It looked to me that it was a very noisy data set going through.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1905300 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 1905302 - Posted: 7 Dec 2017, 8:55:01 UTC
Last modified: 7 Dec 2017, 9:51:29 UTC

Hmm.
Splitters were going along nicely for a while there, but appear to have gotten bogged down. Gone from 60+/s to around 35.
Ready-to-send buffer is emptying out again.

Received-last-hour appears to have leveled off at 118,000.
Edit-
Make that paused at 118,000. It's now hit 123,000 and the splitters are now down to 30/s.
Grant
Darwin NT
ID: 1905302 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 1905310 - Posted: 7 Dec 2017, 10:24:28 UTC - in response to Message 1905302.  
Last modified: 7 Dec 2017, 10:27:31 UTC

Make that paused at 118,000. It's now hit 123,000 and the splitters are now down to 30/s.

Splitters now down to <13/s.
Ready-to-send buffer has now gone from draining slowly, to almost free fall.
Hopefully things will sort themselves out over the next couple of hours, but if not then get 'em while you can.

Edit- splitters just hit 40/s, so panic delayed (returned per hour still climbing, but at a much slower rate than it is has been).
Grant
Darwin NT
ID: 1905310 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1905390 - Posted: 7 Dec 2017, 20:12:30 UTC - in response to Message 1905354.  

We need WU's on Beta.
Give us work, or I'll be forced to take this issue to the UN security council.

Edit: That did it. After many days of no work on Beta, work flows again :-)


. . OK, now I am getting excited about that Lamborghini ...

Stephen

:)
ID: 1905390 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 1905500 - Posted: 8 Dec 2017, 6:28:32 UTC

Well, before the Server Status information died, it looks like the Ready to send buffer still hadn't fully recovered, but there was a drop in the work being returned per hour that allowed it to recover more than it was. Splitter output was still poor.
Add to that, WUs Awaiting deletion for both AP & MB are backing up- something there is having issues of it's own (and that started around the same time splitter output dropped away) As to whether there's any relation between the two, no idea.
Grant
Darwin NT
ID: 1905500 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22527
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1905502 - Posted: 8 Dec 2017, 6:32:06 UTC

...but there's work coming out over on Beta, so maybe they've grabbed the bits and bytes needed ;-)
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1905502 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36774
Credit: 261,360,520
RAC: 489
Australia
Message 1905512 - Posted: 8 Dec 2017, 7:30:47 UTC - in response to Message 1905503.  

...but there's work coming out over on Beta, so maybe they've grabbed the bits and bytes needed ;-)

And not "any" kind of work, it's an endless stream of APs :-)

GBT type AP's?

Cheers.
ID: 1905512 · Report as offensive
Wild6-NJ
Volunteer tester

Send message
Joined: 4 Aug 99
Posts: 43
Credit: 100,336,791
RAC: 140
Message 1905588 - Posted: 8 Dec 2017, 14:12:31 UTC

uh oh, assimilators are down
ID: 1905588 · Report as offensive
Gilles Dupont

Send message
Joined: 28 Mar 08
Posts: 1
Credit: 3,712,892
RAC: 0
Canada
Message 1905652 - Posted: 8 Dec 2017, 18:38:20 UTC - in response to Message 1898224.  

Astropulse for me For Gilles Dupont
ID: 1905652 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1905702 - Posted: 8 Dec 2017, 20:39:29 UTC

SSP is current but doesn't show anything out of the ordinary except for the assimilators being off. But I am getting nothing but no work is available messages on all machines and the caches are dropping fast. Supposedly ~600,000 tasks available. Why aren't they going out?
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1905702 · Report as offensive
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 1905707 - Posted: 8 Dec 2017, 20:45:18 UTC

I have full cache on all my 3 hosts.
ID: 1905707 · Report as offensive
Previous · 1 . . . 24 · 25 · 26 · 27 · 28 · 29 · Next

Message boards : Number crunching : Panic Mode On (108) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.