@Pre-FERMI nVidia GPU users: Important warning

Message boards : Number crunching : @Pre-FERMI nVidia GPU users: Important warning
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · 10 · 11 . . . 13 · Next

AuthorMessage
mrchips

Send message
Joined: 12 Dec 04
Posts: 7
Credit: 8,657,947
RAC: 7,004
United States
Message 1610483 - Posted: 7 Dec 2014, 17:24:39 UTC

Thanks Richard, I downloaded an older driver 337.88, I just got 90 WU's for the gpu. I'm running on all cylinders now.
ID: 1610483 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11516
Credit: 106,371,489
RAC: 71,874
United Kingdom
Message 1610485 - Posted: 7 Dec 2014, 17:32:44 UTC - in response to Message 1610483.  

Thanks Richard, I downloaded an older driver 337.88, I just got 90 WU's for the gpu. I'm running on all cylinders now.

No problem. It's interesting, though, because it does imply that the project's pre-emptive measures to block the faulty driver may have been applied rather too liberally.
ID: 1610485 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 50
Credit: 3,973,364
RAC: 8,865
United States
Message 1611735 - Posted: 10 Dec 2014, 13:24:23 UTC - in response to Message 1609710.  
Last modified: 10 Dec 2014, 13:25:14 UTC

I have another minor update from NVIDIA.

5 November 2014 9:32 pm -- Kevin Kang
Hi Jacob, thanks for the reporting. We have reproduced this issue and have assigned it to the appropriate developer team for reviewing. Thanks!


12 November 2014 11:28 am -- JacobKlein
Hi Kevin / Team: It has been a week without a status update. Could you kindly please consider answering the questions that I asked in the bug details? Thanks, Jacob Klein


14 November 2014 1:21 am -- Kevin Kang
Hi Jacob, Sorry for update on this issue late. As noted in release notes, the R340 drivers will continue to support the Tesla generation of NVIDIA GPUs until April 1, 2016, and the NVIDIA support team will continue to address driver issues for these products in driver branches up to and including Release 340. However, future driver enhancements and optimizations in driver releases after Release 340 will not support these products. Our developer team is working on this issue actively for the future R340 driver release, we'll keep you posted once it has been fixed. Sorry for any inconvenience! Thanks, Kevin


5 December 2014 8:40 pm -- JacobKlein
[CUDA RegDev Program [DevZone]]
Hi Kevin / Team:
I noticed that new R340 drivers (341.21) were release today for my FX 3800M GPU. I excitedly read the release notes, only to find no mention of this issue. I then spent over an hour thoroughly retesting. To my surprise, these 6 failing test cases are still failing in this latest release. I thought you easily reproduced this issue the same day I reported the bug, 4 weeks ago? Can you please give me an update on the timetable for a solution?
Thanks, Jacob Klein


[NEW]
10 December 2015 2:54 am
The following items have been modified for this Bug:
- Status changed from "Open - in progress" to "Open - Fix being tested"
ID: 1611735 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11516
Credit: 106,371,489
RAC: 71,874
United Kingdom
Message 1611754 - Posted: 10 Dec 2014, 14:31:54 UTC - in response to Message 1611735.  

I have another minor update from NVIDIA.

- Status changed from "Open - in progress" to "Open - Fix being tested"

I presume you've volunteered your rig for the test programme? Give me a shout if you see a GeForce (9800GT) version being tested.
ID: 1611754 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 50
Credit: 3,973,364
RAC: 8,865
United States
Message 1611757 - Posted: 10 Dec 2014, 14:44:04 UTC - in response to Message 1611754.  

My main rig is always a test ground for everything :)
But yes, if NVIDIA requests me to test anything, I'll see if I can get you access to test against your 9800GT. Thanks for volunteering.
ID: 1611757 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 50
Credit: 3,973,364
RAC: 8,865
United States
Message 1612803 - Posted: 12 Dec 2014, 14:13:22 UTC - in response to Message 1611735.  
Last modified: 12 Dec 2014, 14:13:49 UTC

I have another update (significant) from NVIDIA.

5 November 2014 9:32 pm -- Kevin Kang
Hi Jacob, thanks for the reporting. We have reproduced this issue and have assigned it to the appropriate developer team for reviewing. Thanks!


12 November 2014 11:28 am -- JacobKlein
Hi Kevin / Team: It has been a week without a status update. Could you kindly please consider answering the questions that I asked in the bug details? Thanks, Jacob Klein


14 November 2014 1:21 am -- Kevin Kang
Hi Jacob, Sorry for update on this issue late. As noted in release notes, the R340 drivers will continue to support the Tesla generation of NVIDIA GPUs until April 1, 2016, and the NVIDIA support team will continue to address driver issues for these products in driver branches up to and including Release 340. However, future driver enhancements and optimizations in driver releases after Release 340 will not support these products. Our developer team is working on this issue actively for the future R340 driver release, we'll keep you posted once it has been fixed. Sorry for any inconvenience! Thanks, Kevin


5 December 2014 8:40 pm -- JacobKlein
[CUDA RegDev Program [DevZone]]
Hi Kevin / Team:
I noticed that new R340 drivers (341.21) were release today for my FX 3800M GPU. I excitedly read the release notes, only to find no mention of this issue. I then spent over an hour thoroughly retesting. To my surprise, these 6 failing test cases are still failing in this latest release. I thought you easily reproduced this issue the same day I reported the bug, 4 weeks ago? Can you please give me an update on the timetable for a solution?
Thanks, Jacob Klein


10 December 2015 2:54 am
The following items have been modified for this Bug:
- Status changed from "Open - in progress" to "Open - Fix being tested"


[NEW]
The following items have been modified for this Bug:
- Status changed from "Open - Fix being tested" to "Closed - Fixed"


[NEW]
Last comment from NVIDIA (12/12/2014 6:07 AM):
Hi Jacob,

You're right, due to time frame, the new R340 driver release - 341.21 doesn't contain this bug's fix yet. This bug has been fixed now in our development versions of R340 driver, it would be available for you in the next R340 driver release. I can't tell you the release date of the next R340 driver for now, but we'll keep you posted once it's ready. Sorry for any inconvenience!

Thanks,
Kevin
ID: 1612803 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 50
Credit: 3,973,364
RAC: 8,865
United States
Message 1612827 - Posted: 12 Dec 2014, 15:29:32 UTC - in response to Message 1612803.  

and more...

5 November 2014 9:32 pm -- Kevin Kang
Hi Jacob, thanks for the reporting. We have reproduced this issue and have assigned it to the appropriate developer team for reviewing. Thanks!


12 November 2014 11:28 am -- JacobKlein
Hi Kevin / Team: It has been a week without a status update. Could you kindly please consider answering the questions that I asked in the bug details? Thanks, Jacob Klein


14 November 2014 1:21 am -- Kevin Kang
Hi Jacob, Sorry for update on this issue late. As noted in release notes, the R340 drivers will continue to support the Tesla generation of NVIDIA GPUs until April 1, 2016, and the NVIDIA support team will continue to address driver issues for these products in driver branches up to and including Release 340. However, future driver enhancements and optimizations in driver releases after Release 340 will not support these products. Our developer team is working on this issue actively for the future R340 driver release, we'll keep you posted once it has been fixed. Sorry for any inconvenience! Thanks, Kevin


5 December 2014 8:40 pm -- JacobKlein
[CUDA RegDev Program [DevZone]]
Hi Kevin / Team:
I noticed that new R340 drivers (341.21) were release today for my FX 3800M GPU. I excitedly read the release notes, only to find no mention of this issue. I then spent over an hour thoroughly retesting. To my surprise, these 6 failing test cases are still failing in this latest release. I thought you easily reproduced this issue the same day I reported the bug, 4 weeks ago? Can you please give me an update on the timetable for a solution?
Thanks, Jacob Klein


10 December 2015 2:54 am
The following items have been modified for this Bug:
- Status changed from "Open - in progress" to "Open - Fix being tested"


[NEW]
The following items have been modified for this Bug:
- Status changed from "Open - Fix being tested" to "Closed - Fixed"


[NEW]
Last comment from NVIDIA (12/12/2014 6:07 AM):
Hi Jacob,

You're right, due to time frame, the new R340 driver release - 341.21 doesn't contain this bug's fix yet. This bug has been fixed now in our development versions of R340 driver, it would be available for you in the next R340 driver release. I can't tell you the release date of the next R340 driver for now, but we'll keep you posted once it's ready. Sorry for any inconvenience!

Thanks,
Kevin


[NEW]
12 December 2014 6:11 am JacobKlein
[CUDA RegDev Program [DevZone]] Hi Kevin! Thanks for getting back to me. I was wondering - does the fix correct all 6 of the API issues that I identified?


[NEW]
12 December 2014 6:22 am Kevin Kang
Yes, I have verified the fix on my Windows7SP1-64Bit/GTX 260 (sm_13) setup, all the 6 OpenCL samples which mentioned in repro steps are passed now.


[NEW]
12 December 2014 7:26 am JacobKlein
[CUDA RegDev Program [DevZone]] Excellent - Thank you so much!
ID: 1612827 · Report as offensive
Wedge009
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 448
Credit: 285,616,564
RAC: 153,194
Australia
Message 1612953 - Posted: 12 Dec 2014, 20:39:24 UTC

Wonderful news, thanks for your persistence.

From what I understand, this means that driver revisions up to and including 340.x will allow pre-Fermi generation NV GPUs to correctly process the OpenCL applications, is that correct? Future revisions drop pre-Fermi support altogether, so this should avoid broken NV AP results in the future, right?
Soli Deo Gloria
ID: 1612953 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 50
Credit: 3,973,364
RAC: 8,865
United States
Message 1612954 - Posted: 12 Dec 2014, 20:43:42 UTC
Last modified: 12 Dec 2014, 20:46:39 UTC

Well... let's not jump to conclusions.

What it means is:
- R337 drivers were processing OpenCL correctly for pre-Fermi GPUs.
- R340 drivers up to and including 341.21 were processing come OpenCL incorrectly for pre-Fermi GPUs.
- The next R340 driver after 341.21 will correct some (maybe all? not sure.) of the OpenCL problems.
- The solution may or may not fix the SETI AstroPulse problems that were identified
- If the solution does fix the SETI AstroPulse problems, then the admins can change the scheduler and the validator to start sending/accepting results from R340 driver that is later than a certain version.
ID: 1612954 · Report as offensive
Wedge009
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 448
Credit: 285,616,564
RAC: 153,194
Australia
Message 1612986 - Posted: 12 Dec 2014, 21:35:08 UTC - in response to Message 1612954.  

I see. Well, still, it is progress in the right direction, no?
Soli Deo Gloria
ID: 1612986 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 50
Credit: 3,973,364
RAC: 8,865
United States
Message 1612988 - Posted: 12 Dec 2014, 21:42:54 UTC

Oh hell yeah, definitely the right direction! People were worried NVIDIA wouldn't do a single thing. I spent several hours going through API testing to figure out a way to delicately show NVIDIA the problem, and now they claim to have fixed it.

I have my fingers crossed that it fixes the AstroPulse issues. No guarantees on that, but I do have hope.
ID: 1612988 · Report as offensive
Profile RottenMutt
Avatar

Send message
Joined: 15 Mar 01
Posts: 1011
Credit: 230,274,184
RAC: 0
United States
Message 1616392 - Posted: 19 Dec 2014, 19:05:15 UTC

Version 347.09 - BETA
Release Date Wed Dec 17, 2014
Operating System Windows 7 64-bit
Windows 8.1 64-bit
Windows 8 64-bit
Windows Vista 64-bit
Language English (US)
File Size 293.12 MB


anyone try the beta
ID: 1616392 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11516
Credit: 106,371,489
RAC: 71,874
United Kingdom
Message 1616410 - Posted: 19 Dec 2014, 19:44:28 UTC - in response to Message 1616392.  

Version 347.09 - BETA
Release Date Wed Dec 17, 2014
Operating System Windows 7 64-bit
Windows 8.1 64-bit
Windows 8 64-bit
Windows Vista 64-bit
Language English (US)
File Size 293.12 MB


anyone try the beta

Won't help anyone in this thread. No new release for either GeForce or Quadro FX pre-Fermi models.
ID: 1616410 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 50
Credit: 3,973,364
RAC: 8,865
United States
Message 1616414 - Posted: 19 Dec 2014, 20:02:53 UTC
Last modified: 19 Dec 2014, 20:04:23 UTC

To clarify, the driver version 347.09 is an R346 (Release 346) driver. It is applicable to GeForce GPUs that are Fermi or later.

The problem in this thread deals with pre-Fermi GPUs and some/all Quadro series GPUs, which NVIDIA has stopped developing new-feature-drivers for, meaning there will never be an R343 or R346 driver. However, NVIDIA has pledged to continue to support pre-Fermi devices by issuing bug-fixes to the R340 (Release 340) drivers.

The current R340 driver for my Quadro FX3800M GPU, is 341.21, dated 12/5/2014. And it does not solve the OpenCL problems.

NVIDIA says that the next R340 release should have the OpenCL fixes to make at least the OpenCL SDK examples work correctly.

Happy holidays!
Jacob

PS: I've had no problems with the new 347.09 driver on my Kepler GPUs. BOINC reports it as Cuda 7.0, and GPUGrid works great with them.
ID: 1616414 · Report as offensive
Greg Tippitt
Volunteer tester

Send message
Joined: 19 Apr 04
Posts: 24
Credit: 9,817,904
RAC: 7,413
United States
Message 1617175 - Posted: 22 Dec 2014, 4:11:17 UTC - in response to Message 1579025.  

TBar,
When you say driver version 340.xx do you mean only 340.52 on Windows or does this apply to 340.32 on Linux as well?

I have Telsa M1060 GPU cards on several Ubuntu Linux systems. I got the systems running just before the SETI work unit shortage, so I aborted all the SETI tasks in my work queues so others could have them, since I run work for several projects. I've been running Einstein and GPUGrid task on my GPUs and have just started them working on SETI again today. These systems each have 24 CPU cores which have started running SETI work again as well. I have gotten a couple of SETI BETA AP work units for CUDA, but have not gotten any production CUDA tasks yet.

Should I only run only CPU tasks for SETI or will my GPUs work correctly?

When you make this type of "Chicken Little - The sky is falling" statement in huge fonts, it would be helpful if you were a bit more specific where the sky is falling.

Greg


That's probably still too vague for most people. Let's see, how about;

Attention, if you have a nVidia card that's 4 years old or older, and have updated to Driver 340.xx,
you are now Flooding SETI with Bad AstroPulse Science. This includes just about any nVidia
card that's not at least a 400 series or around 4 years old or newer.


How about that?
ID: 1617175 · Report as offensive
Wedge009
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 448
Credit: 285,616,564
RAC: 153,194
Australia
Message 1617179 - Posted: 22 Dec 2014, 4:25:43 UTC

This is a problem at the driver level - it is universal, not operating system specific. ie Yes, this warning applies to Linux as well as Windows systems.

Your M1060 GPUs are pre-Fermi generation and so they will be affected by the OpenCL bug in the 340+ drivers. I believe you should be able to run MultiBeam tasks on them, just not AstroPulse tasks. At least, not without producing erroneous results.

Also, I'm quite sure there are no CUDA applications for AstroPulse, only MultiBeam. The plan class may have CUDA in the name but I believe they still run the OpenCL programs.
Soli Deo Gloria
ID: 1617179 · Report as offensive
Profile Wiggo "Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 12607
Credit: 169,485,680
RAC: 86,775
Australia
Message 1617210 - Posted: 22 Dec 2014, 6:33:23 UTC

This is a problem at the driver level - it is universal, not operating system specific. ie Yes, this warning applies to Linux as well as Windows systems.

Your M1060 GPUs are pre-Fermi generation and so they will be affected by the OpenCL bug in the 340+ drivers. I believe you should be able to run MultiBeam tasks on them, just not AstroPulse tasks. At least, not without producing erroneous results.

Also, I'm quite sure there are no CUDA applications for AstroPulse, only MultiBeam. The plan class may have CUDA in the name but I believe they still run the OpenCL programs.

If you have any pre-Fermi GPU with any current 34x.xx drivers you will not get any GPU tasks and that includes Multibeam also, so you must revert to a pre-34x.xx driver to get work. ;-)

Cheers.
ID: 1617210 · Report as offensive
Wedge009
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 448
Credit: 285,616,564
RAC: 153,194
Australia
Message 1617238 - Posted: 22 Dec 2014, 7:15:18 UTC

Ah, so the server restriction is still in place (my oldest NV GPU in use is Fermi generation, so I haven't been keeping track). But it is still the case that MultiBeam applications are not affected by the bug, correct?
Soli Deo Gloria
ID: 1617238 · Report as offensive
Profile Wiggo "Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 12607
Credit: 169,485,680
RAC: 86,775
Australia
Message 1617279 - Posted: 22 Dec 2014, 8:08:21 UTC

Ah, so the server restriction is still in place (my oldest NV GPU in use is Fermi generation, so I haven't been keeping track). But it is still the case that MultiBeam applications are not affected by the bug, correct?

No it doesn't effect Cuda processing, only OpenCL processing, but the servers' restriction is a blanket job by "pre-Fermi GPU + driver version" only on whether you get any type of GPU work or not.

Cheers.
ID: 1617279 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 50
Credit: 3,973,364
RAC: 8,865
United States
Message 1617659 - Posted: 23 Dec 2014, 3:21:41 UTC

Hmm...
When searching for my Quadro FX3800M drivers today on NVIDIA's website, it returned that R346 v347.09 was available.
http://www.nvidia.com/download/driverResults.aspx/81137/en-us

My GPU is even on the "Supported Products" listing, too.

However, the Release Notes say:
Quadro FX 3800M: Not supported

I guess NVIDIA is doing a not-so-great job of filtering out applicable drivers when the user is trying to download them. Nevertheless, I will attempt installation of this driver over the next week, but I don't expect it to work.

Still waiting on the next R340 release to hopefully fix those OpenCL woes... :)
ID: 1617659 · Report as offensive
Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · 10 · 11 . . . 13 · Next

Message boards : Number crunching : @Pre-FERMI nVidia GPU users: Important warning


 
©2017 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.