Panic Mode On (106) Server Problems?

Message boards : Number crunching : Panic Mode On (106) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 13 · 14 · 15 · 16 · 17 · 18 · 19 . . . 29 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1873524 - Posted: 17 Jun 2017, 5:15:45 UTC - in response to Message 1873522.  


It used to be you would get a fairly even split between then (1/3 Arecibo & 2/3 GBT when they had most of the Arecibo splitters disabled) these days I seem to get all Arecibo or all GBT a lot of the time, particularly when getting work is an issue and it finally arrives. Then at other times when the scheduler is behaving I usually get more of a mix of both.


I think you have also hit on something I too have observed. When the schedulers are behaving, you get an equal mix of Arecibo and GBT work, as long as the splitters have something to chew on. When the schedulers do start screwing up, there is usually only one type being delivered.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1873524 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13351
Credit: 208,696,464
RAC: 304
Australia
Message 1873699 - Posted: 17 Jun 2017, 23:33:35 UTC - in response to Message 1873524.  
Last modified: 17 Jun 2017, 23:51:20 UTC

Woke up to find my cache about 1/3 full.
Tried flipping the Use Nvidia GPU option. No joy. Tried flipping the application settings again, no joy. Did end up getting some work, but it appears there's a whole bunch of Arecibo VLARs coming out at the moment, no GBT work in sight. Even if things were working properly it'd be difficult to get GPU work at present.

EDIT- finally started getting some work, and it's all GBT.
Grant
Darwin NT
ID: 1873699 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1873704 - Posted: 17 Jun 2017, 23:56:39 UTC - in response to Message 1873699.  

I was in the same boat. Flipping Nvidia type did nothing for me. Flipping AP/MB/Other Work did nothing for me. Set work_fetch_debug in the logs for all crunchers and let it run through a few requests and I started getting work again. Noticed nothing but VLARs initially and then started seeing GBT tasks again. Full caches now on MB minus the AP tasks deductions.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1873704 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1873705 - Posted: 18 Jun 2017, 0:01:20 UTC - in response to Message 1873699.  

Woke up to find my cache about 1/3 full.
Tried flipping the Use Nvidia GPU option. No joy. Tried flipping the application settings again, no joy. Did end up getting some work, but it appears there's a whole bunch of Arecibo VLARs coming out at the moment, no GBT work in sight. Even if things were working properly it'd be difficult to get GPU work at present.

EDIT- finally started getting some work, and it's all GBT.


. . One machine ran out of work completely during the night and was crunching for E@H for a while but it now has a partially filled cache.

. . The others have been getting some work and right now, all GBT.

Stephen

..
ID: 1873705 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1873706 - Posted: 18 Jun 2017, 0:04:05 UTC - in response to Message 1873704.  
Last modified: 18 Jun 2017, 0:04:31 UTC

I was in the same boat. Flipping Nvidia type did nothing for me. Flipping AP/MB/Other Work did nothing for me. Set work_fetch_debug in the logs for all crunchers and let it run through a few requests and I started getting work again. Noticed nothing but VLARs initially and then started seeing GBT tasks again. Full caches now on MB minus the AP tasks deductions.


. . funnily. On the subject of APs, all my rigs have the same settings but while one rig has had over 40 APs, another has had only maybe 4 and the other 2 rigs zip.

. . It really is a bit of a lucky dip getting work from the servers ...

Stephen

..
ID: 1873706 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1873709 - Posted: 18 Jun 2017, 0:10:27 UTC - in response to Message 1873706.  


. . funnily. On the subject of APs, all my rigs have the same settings but while one rig has had over 40 APs, another has had only maybe 4 and the other 2 rigs zip.

. . It really is a bit of a lucky dip getting work from the servers ...

Stephen

..

For most of the afternoon, two systems were regularly getting AP work and the other system, - NIL. Then within the last half hour, the lagging system made up for the lack of work and got several slugs of AP tasks and now all systems have just about equal amounts of AP work.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1873709 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13351
Credit: 208,696,464
RAC: 304
Australia
Message 1873775 - Posted: 18 Jun 2017, 8:40:51 UTC

And back to random allocation of work on requests for more...
Grant
Darwin NT
ID: 1873775 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1873803 - Posted: 18 Jun 2017, 13:05:17 UTC - in response to Message 1873775.  

And back to random allocation of work on requests for more...


. . The Pentium-D system keeps running out of work and getting E@H tasks. It has happened several times in the last 24/36 hours. But one thing toggling preferences has shown me is that this very rig does not have an app to crunch AP work :(. I am wondering if that might be responsible for the flakiness of getting new work ...

Stephen

?
ID: 1873803 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13141
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1873846 - Posted: 18 Jun 2017, 15:36:03 UTC - in response to Message 1873803.  

No, I have been getting the " No AP work is available", "No tasks sent" messages when the splitters are in fact shoveling out AP work and I DO have the AP app on all machines. Woke up this morning and one machine is down by half in work. Other machines are fine. Trying my work_fetch_debug and preferences toggling.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1873846 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1874122 - Posted: 20 Jun 2017, 4:45:00 UTC

Is anyone getting work for NVIDIA?
I can't for some time now, and no work around is working for me
..
No tasks sent
No tasks available for AP v7
Computer has reached it's limit
ID: 1874122 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 20907
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1874123 - Posted: 20 Jun 2017, 4:51:07 UTC

Loads :-)
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1874123 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1874125 - Posted: 20 Jun 2017, 5:03:44 UTC - in response to Message 1874122.  

Is anyone getting work for NVIDIA?
I can't for some time now, and no work around is working for me
..
No tasks sent
No tasks available for AP v7
Computer has reached it's limit


. . Same here, I hve tried many variation and have had spurious and limited success. In fact on this i5 rig I have only succeeded in filling the CPU queue with Arecibo VLARs. It won't send me anything else ATM. The Linux machines are getting low, the little one is OK at about 80% but the other is below 40% and nothing with provoke new downloads. Either they are playing with some new settings for the servers that are still kind of flaky, or there is a growing problem with the download servers themselves.

. . But the symptoms vary from rig to rig, even at the one location.

Stephen

??
ID: 1874125 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1874127 - Posted: 20 Jun 2017, 5:17:47 UTC

Maybe I will get lucky and the AP limit (5 I think) will break and I will get a load of 270 Astropulse tasks :)
ID: 1874127 · Report as offensive
Iona
Avatar

Send message
Joined: 12 Jul 07
Posts: 790
Credit: 22,438,118
RAC: 0
United Kingdom
Message 1874130 - Posted: 20 Jun 2017, 5:58:30 UTC - in response to Message 1874122.  

I've not been getting anything for my 970, for several hours - I've got maybe, an hours work left. Tons of CPU work and 1 AP, though.
Don't take life too seriously, as you'll never come out of it alive!
ID: 1874130 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13351
Credit: 208,696,464
RAC: 304
Australia
Message 1874131 - Posted: 20 Jun 2017, 6:12:01 UTC

Got home from work to find my cache down to 50WUs left between 2 GPUs.
Grant
Darwin NT
ID: 1874131 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1874139 - Posted: 20 Jun 2017, 7:06:09 UTC

The system does seem to be full of Arecibo vlar's.

I started a pre-maintenance move of ~ 350 CPU to GPU tasks and not one of those fresh downloads of CPU tasks were non-Arecibo vlars.

However I did pick up 29 other GPU tasks while restarting BOINC in the process - strange since none of my other computers got anything.
ID: 1874139 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13351
Credit: 208,696,464
RAC: 304
Australia
Message 1874142 - Posted: 20 Jun 2017, 7:33:36 UTC - in response to Message 1874139.  
Last modified: 20 Jun 2017, 7:35:17 UTC

The system does seem to be full of Arecibo vlar's.

I started a pre-maintenance move of ~ 350 CPU to GPU tasks and not one of those fresh downloads of CPU tasks were non-Arecibo vlars.

However I did pick up 29 other GPU tasks while restarting BOINC in the process - strange since none of my other computers got anything.

I just picked up a batch of AP work.
But no matter how much Application or Use Nvidia GPU flipping I do I can't get any MB work for either CPU or GPU.
Probably got about 60min worth of GPU MB work left, then i'll be out of GPU work just in time for the weekly outage. And probably close to out of work for the CPU as well.


EDIT- Looks like it's not just us having issues, plenty of others are as well- look at the Graphs.
In progress has been steadily declining for almost 9 hours.
Grant
Darwin NT
ID: 1874142 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1874147 - Posted: 20 Jun 2017, 8:43:31 UTC - in response to Message 1874127.  
Last modified: 20 Jun 2017, 8:49:52 UTC

Maybe I will get lucky and the AP limit (5 I think) will break and I will get a load of 270 Astropulse tasks :)


. . If there is an AP limit I don't know what it is, I have had dozens on one machine, but very few on the other three.

. . One of my machines is getting work but this one has absolutely none, it has run out and is crunching E@H tasks. It has been for the last hour. I just changed the host location and that produced some downloads. OH well, it looks like it will be a very long outage from here.

Stephen

??
ID: 1874147 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1874149 - Posted: 20 Jun 2017, 8:56:32 UTC - in response to Message 1874147.  

I have been having better luck by just restarting BOINC ... no bloody idea why ....
ID: 1874149 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1874168 - Posted: 20 Jun 2017, 12:35:58 UTC

Right now it appears to be a VLAR storm. Why that keeps the Server from sending GUPPIES is a Mystery. The Only task I can get to download is an Arecibo VLAR. If you move those to the GPU the Server will send more CPU VLARs. Maybe someone should checkout why an Arecibo VLAR storm Stops the Server from sending BLC VLARs.
ID: 1874168 · Report as offensive
Previous · 1 . . . 13 · 14 · 15 · 16 · 17 · 18 · 19 . . . 29 · Next

Message boards : Number crunching : Panic Mode On (106) Server Problems?


 
©2022 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.