Panic Mode On (107) Server Problems?

Message boards : Number crunching : Panic Mode On (107) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 24 · 25 · 26 · 27 · 28 · 29 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1896221 - Posted: 19 Oct 2017, 22:16:13 UTC - in response to Message 1896220.  

The Linux machine is in the process of getting kicked. I'm down 300 tasks on it. Been getting the "no work is available" message all day.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1896221 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1896265 - Posted: 20 Oct 2017, 1:47:19 UTC - in response to Message 1896221.  

The Linux machine is in the process of getting kicked. I'm down 300 tasks on it. Been getting the "no work is available" message all day.


. . Apart from needing to do lots of server kicking, I am loving these new Blc04 WUs. They only take as long to run on GPUs as a normal Arecibo task. And despite that they are even quicker to run on CPUs than any other Blc task. Only 40 mins each on my i5-6600 :) I'll take all they can give me :)

Stephen

:)
ID: 1896265 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1896267 - Posted: 20 Oct 2017, 1:56:31 UTC - in response to Message 1896265.  

And my luck would have it that I have received nothing but Arecibo shorties as I recover the 200 ghosts the server created when I had the server communication issue.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1896267 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1896311 - Posted: 20 Oct 2017, 10:20:57 UTC - in response to Message 1896267.  

And my luck would have it that I have received nothing but Arecibo shorties as I recover the 200 ghosts the server created when I had the server communication issue.


. . There seems to be plenty of them at the moment so hang in there :)

Stephen

:)
ID: 1896311 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1896325 - Posted: 20 Oct 2017, 13:25:46 UTC - in response to Message 1896311.  

And my luck would have it that I have received nothing but Arecibo shorties as I recover the 200 ghosts the server created when I had the server communication issue.


. . There seems to be plenty of them at the moment so hang in there :)

Stephen

:)


. . Damn, I spoke too soon, they seem to have dried up :(

Stephen

:(
ID: 1896325 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1896351 - Posted: 20 Oct 2017, 15:55:37 UTC - in response to Message 1896311.  

And my luck would have it that I have received nothing but Arecibo shorties as I recover the 200 ghosts the server created when I had the server communication issue.


. . There seems to be plenty of them at the moment so hang in there :)

Stephen

:)

Well it turned out I had about 260 ghosts. Every recovery session to clear them netted nothing but Arecibo shorties. Spent most of the afternoon and early evening doing that. And it was nothing to do on my part just the internal server error communications on several work requests.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1896351 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1896420 - Posted: 20 Oct 2017, 20:31:08 UTC - in response to Message 1896351.  


Well it turned out I had about 260 ghosts. Every recovery session to clear them netted nothing but Arecibo shorties. Spent most of the afternoon and early evening doing that. And it was nothing to do on my part just the internal server error communications on several work requests.


. . I do not envy going through that, juggling half a dozen windows and holding your tongue just right to provoke the resends is tedious and getting only 20 at a time makes it all the more so. I have 48 of them on this rig but I lack the spirit to tackle that ATM :(

Stephen

:(
ID: 1896420 · Report as offensive
Iona
Avatar

Send message
Joined: 12 Jul 07
Posts: 790
Credit: 22,438,118
RAC: 0
United Kingdom
Message 1896481 - Posted: 21 Oct 2017, 0:35:21 UTC
Last modified: 21 Oct 2017, 0:38:07 UTC

For the last 30 mins or so, I've been getting that dreaded message about task availability. Maybe it'll sort itself out, now that I've posted!
Don't take life too seriously, as you'll never come out of it alive!
ID: 1896481 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1896512 - Posted: 21 Oct 2017, 3:35:17 UTC

I've never been so lucky. Always had to give the machines the old recovery kick. In the process of 5 minute countdown now.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1896512 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1896554 - Posted: 21 Oct 2017, 6:36:58 UTC - in response to Message 1896481.  

For the last 30 mins or so, I've been getting that dreaded message about task availability. Maybe it'll sort itself out, now that I've posted!


. . You can almost bet on that :)

Stephen

:)
ID: 1896554 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1897116 - Posted: 23 Oct 2017, 15:47:33 UTC

Woke up this morning to the Linux machine with no GPU tasks onboard or running and only running on the CPU. Have "no work is available" messages in the log for the past 5 hours. Time to kick the servers.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1897116 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1897133 - Posted: 23 Oct 2017, 18:34:33 UTC - in response to Message 1897116.  

I can't actually remember running out of tasks other than during maintenance periods, or server outages affecting everyone. I'm sure their was times, but not in recent memory ... shhh, about age now!
I have seen the cache run down very low, 25% maybe, then recover, and repeat, but not actually run out.
A kick usually makes it act normal again, for me any ways ....
ID: 1897133 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1897137 - Posted: 23 Oct 2017, 19:04:26 UTC - in response to Message 1897133.  

This has happened to me twice now on the Linux cruncher. It takes less than 3 hours to mow through my 300 task GPU allotment. So 3 hours of no work is available responses from the servers and I am running only my CPU cache. When I wake up in the morning the first thing I do is pull up BoincTasks and see what is running. If I see 8 cpu tasks running on the Linux machine I know I ran out of gpu tasks overnight. I've had lots of instances like yourself where I get really low on gpu cache, maybe 25 tasks, but it has either pulled them out of the servers on its own, or I have been around to kick the servers.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1897137 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1897207 - Posted: 24 Oct 2017, 0:42:10 UTC - in response to Message 1897137.  

This has happened to me twice now on the Linux cruncher. It takes less than 3 hours to mow through my 300 task GPU allotment. So 3 hours of no work is available responses from the servers and I am running only my CPU cache. When I wake up in the morning the first thing I do is pull up BoincTasks and see what is running. If I see 8 cpu tasks running on the Linux machine I know I ran out of gpu tasks overnight. I've had lots of instances like yourself where I get really low on gpu cache, maybe 25 tasks, but it has either pulled them out of the servers on its own, or I have been around to kick the servers.


. . For me on the Linux big hitter I routinely do the "kick the server in the pants" thing. Any time the cache drops more than a dozen or so tasks below normal I do it. Usually it is right for a few hours after that. I don't want it running out whenever my back is turned.

Stephen

:)
ID: 1897207 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 20899
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1897257 - Posted: 24 Oct 2017, 10:22:15 UTC

...getting ready for this week's outrage in a few hours time.

Not that there's anything I can do as I'm at work, miles from my crunchers.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1897257 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1897262 - Posted: 24 Oct 2017, 11:42:29 UTC - in response to Message 1897257.  

...getting ready for this week's outrage in a few hours time.

Not that there's anything I can do as I'm at work, miles from my crunchers.


. . Work is the curse of the crunching man ...

. .If you'll pardon the bad paraphrase ...

Stephen

:(
ID: 1897262 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1897292 - Posted: 25 Oct 2017, 0:30:12 UTC

The splitters don't seem to be running properly. The output is only ~3 tasks per second when it is usually 60-70 tasks per second after the resumption from outage.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1897292 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 1897297 - Posted: 25 Oct 2017, 1:04:41 UTC - in response to Message 1897292.  

The splitters don't seem to be running properly. The output is only ~3 tasks per second when it is usually 60-70 tasks per second after the resumption from outage.

Back up to >56 tasks/sec. This seems to happen each week until the transitioner backlog is cleared.
ID: 1897297 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1897298 - Posted: 25 Oct 2017, 1:09:26 UTC

Well I was looking at the Haveland/Munin page and it spiked to 70 per second as the project came back but promptly cratered. And if you look at the Munin page by week, last weeks splitters output didn't show that. It had a good solid sustained 6 hours of 65 tasks per second output.

seti@home Server Statistics
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1897298 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1897575 - Posted: 26 Oct 2017, 16:05:59 UTC

Time to kick the servers. Only half my allotted cache on all machines.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1897575 · Report as offensive
Previous · 1 . . . 24 · 25 · 26 · 27 · 28 · 29 · Next

Message boards : Number crunching : Panic Mode On (107) Server Problems?


 
©2022 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.