It might actually be a good plan for those of us running Anonymous to set No New Tasks until this gets figured out, in order to quit hammering the servers for those running stock who could actually get some work.
Personally, Einstein is running fine here until it's squared away and I just can't see breaking what works here to work around a far-end issue ... Just a thought.

Just setting the No New Tasks doesn't seem to stop my hitting the Server. I probably have to increase the "get additional tasks" to something like 0.25 or 0.3
Tom

. . Or if you have the motivation (and time) suspend the network until you have a page or 3 of completed tasks then let them upload before suspending the network again. But that is very fiddly.

Stephen

. .

. . Or you could do what I have done with the machines with no work, turn them off and save on power bills.
:)

Ah, but without all the lovely heat of those 980s rising up the basement stairwell, I might actually have to turn the house's heat on :| Though it would be cheaper ...

---

Any oldtimers here remember that one Christmas season that seti was very down... was it a month?? two??

Was that the 1 where it all came crashing down in early December and didn't get fixed until late January, or something like that?

It might actually be a good plan for those of us running Anonymous to set No New Tasks until this gets figured out, in order to quit hammering the servers for those running stock who could actually get some work.
Personally, Einstein is running fine here until it's squared away and I just can't see breaking what works here to work around a far-end issue ... Just a thought.

. . Or you could do what I have done with the machines with no work, turn them off and save on power bills.
Stephen :)

The lack of heat being produced here is very helpful ATM with the very hot and smokey conditions we're experiencing, but I'd imagine that those in the other half of the world would prefer to have the heat on. ;-)

Cheers.

I well remember the 'very extended maintenance' of a few years ago and had no problem with that (I was one of the many who advocated it), but this latest 'episode' is something else. It may well be, make or break, as you suggest. For me, that moment appears to have arrived. Best of luck to you all.

. . Considering the time of year it may be circumspect to wait until the new year and see if it is all resolved. Considering the role the BOINC project management appears to have had in this debacle it may not be resolved until then. So relax and have a Merry Christmas for the interim. And a pleasant and Happy New Year!

Stephen

:)

Any oldtimers here remember that one Christmas season that seti was very down... was it a month?? two??

Was that the 1 where it all came crashing down in early December and didn't get fixed until late January, or something like that?

It might actually be a good plan for those of us running Anonymous to set No New Tasks until this gets figured out, in order to quit hammering the servers for those running stock who could actually get some work.
Personally, Einstein is running fine here until it's squared away and I just can't see breaking what works here to work around a far-end issue ... Just a thought.

. . Or you could do what I have done with the machines with no work, turn them off and save on power bills.
Stephen :)

The lack of heat being produced here is very helpful ATM with the very hot and smokey conditions we're experiencing, but I'd imagine that those in the other half of the world would prefer to have the heat on. ;-)

Cheers.

Instead of burning my GPUs with electricity I'll burn some wood in the fireplace and some rubber with spikes on the icy roads with my V60 D6.
Now I have time to sleep and replace my Seasonic 1250-X with a 1600W bla bla and throw out those two 1080's and one 1080Ti to make room for my presents.

Those old ones I'd hope go to East. Not south.

---

To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones

Brilliant workaround... applying to every machine stat and thank you! :^)
Edit: I tried it first on a 2x2080ti machine and it was doing two at once fine... watched it complete half a dozen like that. Hrm.

I'm confused is stated on the workaround:

(this workaround will work only on single GPU hosts)

And you talk about a 2 GPU host, so it's safe to use on a multi GPU build or no?

---

but this latest 'episode' is something else. It may well be, make or break, as you suggest. For me, that moment appears to have arrived. Best of luck to you all.

Actually Iona your failed tasks are all from "device 1" which should be your 2nd GPU as "device 0" is completing its tasks. If a restart doesn't clear the problem (it may have just had a driver crash and hasn't recovered) then just remove that offending GPU.
Cheers.

. . Hi Wiggo,

. . The machine I see only has the one GTX970, and looking at the stderr info even on the successful tasks it is having massive problems initialising the card with constant restarts before it actually processes the task. The errored tasks are when it gets one too many restarts.

. . I agree that maybe shutting down the machine, lifting the lid and blowing or sucking out the dust bunnies might be a good idea, and checking the PCIe power cables as well. I prefer the vacuum cleaner to the canned air approach as it does not simply move the dust, but actually removes the dust. If that and the restart does not restore normal behaviour then serious hardware issues sound likely.

Stephen

<fingers crossed>

:)

Brilliant workaround... applying to every machine stat and thank you! :^)
Edit: I tried it first on a 2x2080ti machine and it was doing two at once fine... watched it complete half a dozen like that. Hrm.

I'm confused is stated on the workaround:

(this workaround will work only on single GPU hosts)

And you talk about a 2 GPU host, so it's safe to use on a multi GPU build or no?

This is a warning, as I can't test it on multi GPU systems.
Judging by the stderr output, the BOINC manager assigns the CUDA device number in a different way than the OpenCL device number, so the CUDA client will not know which device it should use. Perhaps it will use only device 0, regardless that another instance is already using it.

---

but this latest 'episode' is something else. It may well be, make or break, as you suggest. For me, that moment appears to have arrived. Best of luck to you all.

Actually Iona your failed tasks are all from "device 1" which should be your 2nd GPU as "device 0" is completing its tasks. If a restart doesn't clear the problem (it may have just had a driver crash and hasn't recovered) then just remove that offending GPU.
Cheers.

. . Hi Wiggo,
. . The machine I see only has the one GTX970, and looking at the stderr info even on the successful tasks it is having massive problems initialising the card with constant restarts before it actually processes the task. The errored tasks are when it gets one too many restarts.
. . I agree that maybe shutting down the machine, lifting the lid and blowing or sucking out the dust bunnies might be a good idea, and checking the PCIe power cables as well. I prefer the vacuum cleaner to the canned air approach as it does not simply move the dust, but actually removes the dust. If that and the restart does not restore normal behaviour then serious hardware issues sound likely.
Stephen
<fingers crossed>
:)

Yes it does only show a single card now so it either didn't reinitialise after a reboot and/or Iona removed it, but with the replica still being a good 11hrs behind only Iona can tell us sooner what her solution was and if it's working (and I didn't waste my time going through a large random selection of her errored and validated tasks Stderr outputs for nothing). ;-)

Cheers.

Thanks... I think that explains the poor performance I am having on the 4-GPU system... they are all using the same card! The 2 GPU systems are still an improvement over stock. The only issue I had was that on first launch a resumed task may stall; abort it and all the rest are fine.

---

Mr. Kevvy are you using BoincTasks to monitor the work units when they start? Should tell you which GPUs the work units are running on.

---

Thanks... I think that explains the poor performance I am having on the 4-GPU system... they are all using the same card! The 2 GPU systems are still an improvement over stock. The only issue I had was that on first launch a resumed task may stall; abort it and all the rest are fine.

Perhaps you could try to uninstall opencl, forcing your system to ask only for CUDA tasks.
I think the special app can fully replace the original CUDA60 app, so the BOINC manager can correctly set the device id to use.

---

the issue is likely the task trying to restart a task that was previously running a different app. it was running on an openCL app, and then when you renamed it to get the CUDA app in there, it tries to pick up where it left off, but everything is different and it just hangs.

I just aborted any tasks running on GPUs that showed 0% GPU utilization and it successfully picks up new tasks.

running fine on my 10-GPU host, when it can get tasks.

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

I can't figure out how to pass the -nobs parameter with my workaround.

---

try putting it in the cmdline text file?

mb_cmdline-8.22-opencl_nvidia_SoG.txt
mb_cmdline-8.22-opencl_nvidia_sah.txt

etc

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

I did it. It has no effect.

---

Interesting. Well it looks like I've been out of work for two days now. Just noticed it. It's been a lot of HTTP errors over the past 48 hours when trying to contact scheduler, and the times that it is successful, the response is well over 60 seconds from request to reply... and it comes back with "no tasks available."

But looking here on the website.. I apparently have 19 in progress. Yay ghosts.

What was the procedure for that again? Send a work request, then suspend network before a reply, and then give it a few minutes and allow network again and do another request?

---

Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)

Interesting. Well it looks like I've been out of work for two days now. Just noticed it. It's been a lot of HTTP errors over the past 48 hours when trying to contact scheduler, and the times that it is successful, the response is well over 60 seconds from request to reply... and it comes back with "no tasks available."

But looking here on the website.. I apparently have 19 in progress. Yay ghosts.

What was the procedure for that again? Send a work request, then suspend network before a reply, and then give it a few minutes and allow network again and do another request?

The replica is still 11 hours behind so it's likely that you have no ghosts at all. ;-)

Cheers.

it should work. I'm running it in my text file. and I'm seeing the right cpu use % for my setup.

7 GPUs running the special app on a 12 thread system sees ~60% CPU use
10 GPUs running the special app on a 40 thread system sees about 25% CPU use.

you can also add it to your app_config file. add this line:

<cmdline>-nobs</cmdline>

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

the issue is likely the task trying to restart a task that was previously running a different app. it was running on an openCL app, and then when you renamed it to get the CUDA app in there, it tries to pick up where it left off, but everything is different and it just hangs.

I just aborted any tasks running on GPUs that showed 0% GPU utilization and it successfully picks up new tasks.

running fine on my 10-GPU host, when it can get tasks.

How often do you get tasks? Even with stock and reset, I got only like 50 WUs yesterday when I got lucky. I just tried again and I got either no new work or HTTP internal error. I feel it's probably wiser for me to just stop sending requests to server until this is resolved.

---