There is an ongoing problem with CUDA tasks and the Clock Rates being dropped in Nvidia cards I've written it up and the problem is with Vista and Win7 both.

What happens, is the clock rate gets dropped to conserve power/heat etc. and never returns to high speed again. This always happens with DUAL Nvidia cards installed and seems only magic prevents it from happening on it's own most of the time. Doesn't matter what power settings are set, performance mode seems to help, but not totally correct it. Snoozing or Suspending tasks is a 95% guarantee the clocks with drop and never regain full speed again.

I've informed Nvidia tech support and the forums.


http://forums.nvidia.com/index.php?s=9f29a996e0ac9d6ea44a506f6631f805&showtopic=200414&pid=1237460&st=0&#entry1237460

8-)

Tex1954

This problem has recently been drawn to the attention of the SETI CUDA developers, and is in the process of being reported onwards to other CUDA-enabled BOINC projects. It seems to be related specifically to the release of nVidia drivers which will support the forthcoming v4 release of the CUDA run-time support files. The central BOINC library code isn't yet fully compatible with CUDA v4: the problem has been overcome with test programs, and should go away with the next round of application releases.

Unfortunately, v3 and earlier nVidia drivers aren't available for the very latest generation of nVidia cards, but if you have a card which can run with nVidia driver 266.58, that should avoid the problem.

This problem has recently been drawn to the attention of the SETI CUDA developers, and is in the process of being reported onwards to other CUDA-enabled BOINC projects. It seems to be related specifically to the release of nVidia drivers which will support the forthcoming v4 release of the CUDA run-time support files. The central BOINC library code isn't yet fully compatible with CUDA v4: the problem has been overcome with test programs, and should go away with the next round of application releases.

Unfortunately, v3 and earlier nVidia drivers aren't available for the very latest generation of nVidia cards, but if you have a card which can run with nVidia driver 266.58, that should avoid the problem.

Interesting, Richard do you know if this is only effecting the 5 series cards or is it effecting others running the 275.61's? I downgraded to the 275.51's and don't seem to be having these issues.

---

Traveling through space at ~67,000mph!

This problem has recently been drawn to the attention of the SETI CUDA developers, and is in the process of being reported onwards to other CUDA-enabled BOINC projects. It seems to be related specifically to the release of nVidia drivers which will support the forthcoming v4 release of the CUDA run-time support files. The central BOINC library code isn't yet fully compatible with CUDA v4: the problem has been overcome with test programs, and should go away with the next round of application releases.

Unfortunately, v3 and earlier nVidia drivers aren't available for the very latest generation of nVidia cards, but if you have a card which can run with nVidia driver 266.58, that should avoid the problem.

Interesting, Richard do you know if this is only effecting the 5 series cards or is it effecting others running the 275.61's? I downgraded to the 275.51's and don't seem to be having these issues.

I'm not involved in the detailled technicalities, just a messenger. I'll try and find out - or someone who knows might post.

This problem has recently been drawn to the attention of the SETI CUDA developers, and is in the process of being reported onwards to other CUDA-enabled BOINC projects. It seems to be related specifically to the release of nVidia drivers which will support the forthcoming v4 release of the CUDA run-time support files. The central BOINC library code isn't yet fully compatible with CUDA v4: the problem has been overcome with test programs, and should go away with the next round of application releases.

Unfortunately, v3 and earlier nVidia drivers aren't available for the very latest generation of nVidia cards, but if you have a card which can run with nVidia driver 266.58, that should avoid the problem.

Interesting, Richard do you know if this is only effecting the 5 series cards or is it effecting others running the 275.61's? I downgraded to the 275.51's and don't seem to be having these issues.

I'm not involved in the detailled technicalities, just a messenger. I'll try and find out - or someone who knows might post.

OK I will. It's quite involved, but I'll try detail first then explain further if needed.

Certain new methods that Cuda4 drivers deal with memory & Cuda transfers are sensitive to being abrubtly terminated without warning. All Windows-Boinc-Cuda app releases to date use boincApi code for their exit code, given that Boinc needs to tell applications through this channel when to snooze/resume/exit etc, as well as when the worker needs to exit normally.

Symptoms directly pertaining to effects using Cuda 4 drivers with current Boinc-Cuda applications are primarily the 'sticky downclock' problem, but also other forms of unexplained erroring out.

There are other non-Cuda related symptoms visible across non-Cuda (CPU) applications as well, most visible being truncation or erasure of the stderr.txt contents, and less visible possibly checkpoint & result files as well.

These sorts of symptoms, being apparently related to how 'nicely' the program treats the active buffer transfers when the application shuts down, seemed to be statisically more common on lower bus/memory speed systems, probably as a result of the transfers etc taking longer (i.e. higher contention).

The trial solution in testing is to install exit code within boincAPI that 'asks' the worker thread (that feeds the Cuda device etc) to shut down 'nicely', so that it can quickly finish what it is doing & tidyup before being 'killed'. At present this seems effective at preventing the downclock problem & possibly the stderr/etc truncation symptoms as well, though we're poking at it to look for unexpected issues at this time. I've relayed as much information as I can to Berkeley & will leave it in their hands.

If you experience the downclock problems, there are currently 2 options I'm aware of:
- Downgrade to driver 266.58 which is not as senstivie to its tasks being summarily executed, or
- Determine if it's a situation where you absolutely need the fix now: That would only be a possiblity for this Project (Other projects don't have the fix yet & may not be even aware of the issue), and only under special circumstances, as it would involve pre-alpha testing unproven code. We are a bit overworked at the moment with V7 & other development considerations, So please don't expect a rush release of this uproven code.

In any case, high throughput hosts are statisically less susceptible to this problem, so It is quite possible many hosts don't see the symptoms appear even with newer drivers & existing applications.

HTH, Jason

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

Thanks to all the great folks for addressing this issue!

I am sure it is in capable hands now!!

8-)

Tex1954

Thanks for the information guys. I never noticed the clock slow down per say on my 480, hence the question. I was seeing other differences mainly in the gaming arena that ultimately caused me to swap drivers again.

---

Traveling through space at ~67,000mph!

Making it so you can click on this link from first post.

http://forums.nvidia.com/index.php?s=9f29a996e0ac9d6ea44a506f6631f805&showtopic=200414&pid=1237460&st=0&#entry1237460

Dual cards only?

My GTS450 was dropping to about half clock speed. I was using the Nvidia 270.61 driver. I have the onboard 8300 disabled in the bios, so I don't think that the BM or the driver can see it.

I have reverted to 267.59 and haven't seen the problem again as of yet.

---

Just got notified by the Nvidia Updater that 275.33 driver was available. WHQL certified and not beta. Downloading as I type. Will give it a shot. I've not been bothered at all by any downclocking with the 270.61 and 275.27 drivers on my two GTX 460's for some reason.

Keith

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

So far so good here with the 580's. I have them running in SLI of course. I also have them overclocked slightly to 825mhz for the core clock and 2104mhz for the memory clock. I have been watching them using a program called MSI Afterburner. Even with repeated stops and starts, no downclocking issues have been noticed with driver 270.61.

Core just dropped to 405 mhz when I suspended computation and didn't go back up. It still happens with the 275.33 WHQL drivers.

Card is:
GeForce GTS 450

I just looked to see what version I have:
NVIDIA UNIX x86_64 Kernel Module 260.19.36

No problems here with clock speed. :)

I wonder if I should upgrade anyway?

No issues with it on my 480. Haven't bothered with them on the 250 yet.

---

Traveling through space at ~67,000mph!

Just had a huge slowdown on my 560ti. CUDA apps that were taking at most 24 mins were taking almost 2hrs. Restart fixed that.

---

Executive Director GPU Users Group Inc. -
brad@gpuug.org