Panic Mode On (114) Server Problems?

Message boards : Number crunching : Panic Mode On (114) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 45 · Next

AuthorMessage
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1967090 - Posted: 25 Nov 2018, 21:29:42 UTC - in response to Message 1967088.  

Thanks Rob, that is the one I have been trying to monitor mainly. It is doing that a lot in the last week. Still can't get any of the tasks to show though. Not sure how you were able to pull them up. Thanks for the heads up.
I'm not having any problems viewing your computers' tasks lists here Keith.

Cheers.
ID: 1967090 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1967097 - Posted: 25 Nov 2018, 21:39:42 UTC - in response to Message 1967090.  

Well I can now look at my other hosts with something more reasonable for the spinner, they eventually populate. That host 6279633 still won't show me any tasks. I've cleared the browsing history, restarted the browser, used Firefox and other computers. Still won't show me the tasks. I think the high number of errors that Rob says are there are slowing me down somehow.

Ok, after watching the spinner for 5 minutes, the tasks are finally populating. Doesn't look like those 32 were all in one slug. Looks like it is throwing one every hour. Rebooted that host and looked things over again. Reverted to an older BIOS profile and will see if that changes anything. Thanks for the help guys.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1967097 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1967105 - Posted: 25 Nov 2018, 21:45:31 UTC - in response to Message 1967088.  
Last modified: 25 Nov 2018, 21:46:25 UTC

Thanks Rob, that is the one I have been trying to monitor mainly. It is doing that a lot in the last week. Still can't get any of the tasks to show though. Not sure how you were able to pull them up. Thanks for the heads up.

. . Like Rob and Wiggo I can view your machines without difficulty. The slacker that isn't pulling it's weight is the one with the errors :). I did find it slow to view the error on the big hitter, but with 10,000 tasks to keep track of I think I can understand that. Is it possible that your ISP is choking your bandwidth??

Stephen

??
ID: 1967105 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1967111 - Posted: 25 Nov 2018, 21:51:30 UTC

Wouldn't be an issue with the ISP bandwidth, more the issue that the database has to pull up and sort through 53,000 tasks on my account. My download bandwidth seems to handle 5 hosts simultaneously requesting work from the project at the same time when the project just comes back from outage and I am one of the first to hit the schedulers. I consistently get 16Mbs download rates.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1967111 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1967122 - Posted: 25 Nov 2018, 22:13:45 UTC

I wonder if my GPUGrid task which is running on high priority now are taking away too much resources from the CUDA Seti tasks. I always get tagged with EDF on the GPUGrid tasks one day before their deadline. The other hosts seem to handle that common situation though. But the troublesome host has had a GPUGrid task running in EDF while the Seti tasks were erroring out. If it wasn't for the overwhelming credit disparity between GPUGrid and Seti, the project debts would be more reasonable and I wouldn't be caught in EDF all the time. I only have 6 GPUGrid tasks onboard any host at any time which equates to 10 hours of computing. You would think that I should be able to dispatch 10 hours of computing time in the typical 4 day deadlines for GPUGrid.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1967122 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22190
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1967123 - Posted: 25 Nov 2018, 22:14:08 UTC

I think you are facing in the right direction with your comments about the number of errors, but even so, it only took about 15sec to see the "head count", and another few seconds to get one on the screen.
Hope you get to the bottom of the problem with that cruncher quickly.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1967123 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1967124 - Posted: 25 Nov 2018, 22:15:27 UTC - in response to Message 1967111.  

Wouldn't be an issue with the ISP bandwidth, more the issue that the database has to pull up and sort through 53,000 tasks on my account. My download bandwidth seems to handle 5 hosts simultaneously requesting work from the project at the same time when the project just comes back from outage and I am one of the first to hit the schedulers. I consistently get 16Mbs download rates.


. . OK, but it seems unlikely to be a SETI database issue when others can access your details without significant issues. To me that says the database is coping even with the humungous size of your account details.

Stephen

??
ID: 1967124 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1967141 - Posted: 25 Nov 2018, 22:47:49 UTC - in response to Message 1967124.  

Well there was a little blip in the database master queries in the Haveland graphs around the time I was trying to view tasks. I can see tasks now with no issues. So transcendental issue for me it seems.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1967141 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1967172 - Posted: 26 Nov 2018, 0:53:22 UTC - in response to Message 1967141.  
Last modified: 26 Nov 2018, 0:54:27 UTC

Well there was a little blip in the database master queries in the Haveland graphs around the time I was trying to view tasks. I can see tasks now with no issues. So transcendental issue for me it seems.


. . Cool, as long as it is no longer a problem. Though I am concerned that with the noisy tapes at the tail end of each GBT series lately we are looking at a lot of noise bombs from all 3 series of tapes that are still mounted. This could see the remaining data vanish in all too short a time. I hope they have some more tapes ready to mount shortly. Parkes maybe ? Just kidding ...

Stephen

:).
ID: 1967172 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1967198 - Posted: 26 Nov 2018, 3:09:59 UTC
Last modified: 26 Nov 2018, 3:42:10 UTC

I am looking forward to seeing the four 10.47 GB tapes get processed they have been there for a long time.
    blc05_2bit_guppi_58406_33654_DIAG_3C249_1_0120 blc05_2bit_guppi_58406_33734_DIAG_3C249_1_0121 blc06_2bit_guppi_58406_33654_DIAG_3C249_1_0120
    blc06_2bit_guppi_58406_33734_DIAG_3C249_1_0121


Looks like they will be some time away from being processed because new work has been added as mentioned above


ID: 1967198 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1967206 - Posted: 26 Nov 2018, 3:36:27 UTC

More datafiles have been added, so no need to panic until Tuesday's outage (hopefully a normal one).
ID: 1967206 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1967212 - Posted: 26 Nov 2018, 4:23:27 UTC - in response to Message 1967206.  

More datafiles have been added, so no need to panic until Tuesday's outage (hopefully a normal one).

A good thing. Will have to wait till Tuesday for that zombie Arecibo 14no18aa file to get expunged. Website has slowed down for me in the last 5 minutes compared to this afternoon. Maybe the adding of the new files is consuming server resources.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1967212 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1967357 - Posted: 27 Nov 2018, 7:11:59 UTC - in response to Message 1967212.  

More datafiles have been added, so no need to panic until Tuesday's outage (hopefully a normal one).

A good thing. Will have to wait till Tuesday for that zombie Arecibo 14no18aa file to get expunged. Website has slowed down for me in the last 5 minutes compared to this afternoon. Maybe the adding of the new files is consuming server resources.


. . Ruh Roh!

. . It looks like maybe they reduced the overall task limit to about 500 cos I am getting "reached the daily limit" messages...

. . :(

Stephen

? ?
ID: 1967357 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13731
Credit: 208,696,464
RAC: 304
Australia
Message 1967358 - Posted: 27 Nov 2018, 7:22:16 UTC - in response to Message 1967357.  

. . Ruh Roh!

. . It looks like maybe they reduced the overall task limit to about 500 cos I am getting "reached the daily limit" messages...

Your i5 had had a GPU compute error, so it will be limited in the number of WUs it can get till more WUs have validated.


SETI@home v8 (anonymous platform, NVIDIA GPU. Max tasks per day 70.
Grant
Darwin NT
ID: 1967358 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1967359 - Posted: 27 Nov 2018, 7:28:04 UTC - in response to Message 1967357.  

The reason you would be getting a "daily task limit" is you errored out a bunch of work. The project automatically reduces your tasks until you start returning valids again.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1967359 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1967360 - Posted: 27 Nov 2018, 7:33:14 UTC - in response to Message 1967358.  

. . Ruh Roh!

. . It looks like maybe they reduced the overall task limit to about 500 cos I am getting "reached the daily limit" messages...

Your i5 had had a GPU compute error, so it will be limited in the number of WUs it can get till more WUs have validated.


SETI@home v8 (anonymous platform, NVIDIA GPU. Max tasks per day 70.
And that's not to mention that if your rig requests both CPU & GPU work and your CPU has its full quota and no work is available then you will also get that same message. ;-)

Cheers.
ID: 1967360 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1967401 - Posted: 27 Nov 2018, 16:10:22 UTC - in response to Message 1967359.  

The reason you would be getting a "daily task limit" is you errored out a bunch of work. The project automatically reduces your tasks until you start returning valids again.


. . Yep, I had a single compute error task, but I had returned over 80 successful tasks after than yet continued to get the 'task limit' message for over an hour until the cache ran dry. Talk about getting harsh :(

Stephen

:(
ID: 1967401 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1967410 - Posted: 27 Nov 2018, 22:19:24 UTC

. . Hi people,

. . Does anyone remember where the setting is for the maximum number of completed tasks reported at one time?

. . Getting nothing but http errors so I think I need to reduce it a bit.

Stephen

? ?
ID: 1967410 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1967412 - Posted: 27 Nov 2018, 22:21:08 UTC - in response to Message 1967409.  

Nop, 14no18aa is still stuck, even after the outage.


. . Soon it will have squatter's rights and we'll never get rid of it ...

Stephen

:)
ID: 1967412 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1967413 - Posted: 27 Nov 2018, 22:25:33 UTC - in response to Message 1967410.  

From the manual:

<max_tasks_reported>N</max_tasks_reported>
Report at most N tasks per scheduler RPC. Try N=1000 if your computer has lots of tasks to report and is having trouble completing a scheduler RPC.
Take no notice of the 1000 suggestion - you'd be better off starting with say 64.

If you've been setting and unsetting log flags, you'll have a fully-populated cc_config.xml file already - modify the existing line, don't add a duplicate.
ID: 1967413 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 45 · Next

Message boards : Number crunching : Panic Mode On (114) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.