@Pre-FERMI nVidia GPU users: Important warning

Message boards : Number crunching : @Pre-FERMI nVidia GPU users: Important warning
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 13 · Next

AuthorMessage
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1597716 - Posted: 6 Nov 2014, 16:56:53 UTC - in response to Message 1597684.  

I have received the following updates to my NVIDIA bug 1574543:
https://developer.nvidia.com/nvbugs/cuda/edit/1574543
Status changed from "Open - pending review" to "Open - in progress"

5 November 2014 9:32 pm Kevin Kang
Hi Jacob, thanks for the reporting. We have reproduced this issue and have assigned it to the appropriate developer team for reviewing. Thanks!

Well, at least you got a named contact out of it - that's more than Raistmer's rather less specific version of the same report got. I do think that the identified failure of NVidia's own sample code on professional hardware with enterprise drivers stands a better chance of being fixed than a third-party application on the consumer platform. Best of luck.

Richard, I received exactly same message from exactly same named person. Just FYI.
But, yep, best of luck...
ID: 1597716 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1597722 - Posted: 6 Nov 2014, 17:04:46 UTC
Last modified: 6 Nov 2014, 17:05:42 UTC

2 November 2014 11:31 pmRaistmer _
Any chance to get bugfix included for any subsequent driver for CC 1.x cards? Currently we had to exclude CC1.1 cards support from BOINC. And already receiving reclamations from users that will be rerouted of course.
11 October 2014 2:36 amRaistmer _
product field filled
6 October 2014 4:10 pmRaistmer _
Could some explanation be given why "will not fix" ?
16 September 2014 3:03 amKevin Kang
Thanks for the reporting and providing the repro case. We have reproduced this issue and has assigned it to the appropriate developer team for investigation, we'll keep you posted once the further information becomes available. Thanks!


Good luck
ID: 1597722 · Report as offensive
BEParker

Send message
Joined: 24 Jun 99
Posts: 17
Credit: 10,317,641
RAC: 18
United States
Message 1599004 - Posted: 9 Nov 2014, 16:08:50 UTC - in response to Message 1597374.  

Thomas Arnold wrote:
Hello, I need your insight and help.
I am using this Video card, NVIDIA GeForce GTX 260 (896MB) driver: 311.06 OpenCL: 1.0

In the past we have never had a problem but now we are receiving
Computation error running seti@homev77.00 (cuda22)

We are not too familiar with much of the program but we support the efforts to run the data sets. Can you please tell me if we need to change something with our setup or will these errors clear themselves or just continue to build up in the task tab?

The driver is old enough so it doesn't have the issue which started this thread. I don't know why all SETI@home v7 7.00 windows_intelx86 (cuda22) and (cuda23) tasks are failing on your host 6648399, but it does very well on (cuda32), (cuda42), and (cuda50). Perhaps one of the CUDA experts here can figure out why the servers aren't sending tasks for the plan classes which work well.
                                                                   Joe


Will likely be digging out the scheduler code again on the weekend, if someone doesn't beat me to it. No accumulated data for the app versions, plus a logic hole with respect to systematically issuing to all app versions, ignoring the error count & quota, seems to be along the lines of what's happening. [I'll need to start by looking if that server code's been changed since a couple of months ago]

For the host side, FWIW the application (2.2 & 2.3 planclasses) appears to not even be making it to device initialisation. That would seem to me the DLLs are somehow damaged, or the driver install has gone awry. I'd imagine a clean driver install (of a suitable known good older version for this GPU) and a project reset may be in order.


I am having the same kind of problem with a GeForce 9500 GT card. It was working fine until the middle October. I checked the message boards and I, obviously, fall into the pre-Fermi category, but no combination of older or newer drivers allows a CUDA data file to be used for any more than about 15 seconds, then resets to 0, starts over, and eventually reports "Computational Error". I just did an uninstall and a re-install of BOINC and a disconnect and re-connect for the SETI project.

I realize I need newer hardware to be able to do newer things. The only reason I had a video card with a GPU was to use it for SETI. I don't do gaming, so old is not an issue for that. I'm just trying to figure out if my system is the reason the GPU stopped working of if something changed in BOING that rendered my GPU useless.
ID: 1599004 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1599007 - Posted: 9 Nov 2014, 16:29:38 UTC - in response to Message 1599004.  
Last modified: 9 Nov 2014, 16:48:21 UTC

I am having the same kind of problem with a GeForce 9500 GT card. It was working fine until the middle October. I checked the message boards and I, obviously, fall into the pre-Fermi category, but no combination of older or newer drivers allows a CUDA data file to be used for any more than about 15 seconds, then resets to 0, starts over, and eventually reports "Computational Error". I just did an uninstall and a re-install of BOINC and a disconnect and re-connect for the SETI project.

I realize I need newer hardware to be able to do newer things. The only reason I had a video card with a GPU was to use it for SETI. I don't do gaming, so old is not an issue for that. I'm just trying to figure out if my system is the reason the GPU stopped working of if something changed in BOING that rendered my GPU useless.

Can you do the Remove Seti and add Seti again, I'd have expected to see a lot of abandoned work, But there is none,
try and complete as many CPU tasks as possible before you do it. (Set NNT)

The scheduler did once send you Cuda32, Cuda42 and Cuda50 work, and have high Max tasks per day for all three, But insists in only sending Cuda22 and Cuda23 work now, strange.

Claggy
ID: 1599007 · Report as offensive
BEParker

Send message
Joined: 24 Jun 99
Posts: 17
Credit: 10,317,641
RAC: 18
United States
Message 1599026 - Posted: 9 Nov 2014, 17:56:38 UTC - in response to Message 1599007.  

I am having the same kind of problem with a GeForce 9500 GT card. It was working fine until the middle October. I checked the message boards and I, obviously, fall into the pre-Fermi category, but no combination of older or newer drivers allows a CUDA data file to be used for any more than about 15 seconds, then resets to 0, starts over, and eventually reports "Computational Error". I just did an uninstall and a re-install of BOINC and a disconnect and re-connect for the SETI project.

I realize I need newer hardware to be able to do newer things. The only reason I had a video card with a GPU was to use it for SETI. I don't do gaming, so old is not an issue for that. I'm just trying to figure out if my system is the reason the GPU stopped working of if something changed in BOING that rendered my GPU useless.

Can you do the Remove Seti and add Seti again, I'd have expected to see a lot of abandoned work, But there is none,
try and complete as many CPU tasks as possible before you do it. (Set NNT)

The scheduler did once send you Cuda32, Cuda42 and Cuda50 work, and have high Max tasks per day for all three, But insists in only sending Cuda22 and Cuda23 work now, strange.

Claggy


Removed and re-added Seti. Also made sure the newer PhysX package was removed and clean-installed NVIDIA driver 266.58 (as recommended in an earlier message in this thread).

A CUDA23 file started up and had been running for 5 minutes, now, with no problems noted. Download a whole bunch more CUDA23 files after that.

A "contaminated" file may have been the issue. If the current CUDA23 file successfully finishes (in about 2 hours) I will know, I guess.

As for not getting any CUDA32, CUDA42, or CUDA50 work, I remember getting those in the past, but can't remember when they stopped.

Thank You for your help.

Bruce Parker
Dumfries, VA
ID: 1599026 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1599028 - Posted: 9 Nov 2014, 18:08:43 UTC - in response to Message 1599004.  

About terminology:
The only reason I had a video card with a GPU was to use it for SETI.

Every videocard and videochip out there has or is a GPU. This abbreviation stands for graphics processing unit and is used to render anything on the monitor. Without a GPU, no images on the screen, not 2D and not 3D.

So a GPU is not something awfully new.

The awfully new thing about these is that since a couple of years they've become so powerful that they can be used for doing calculations or their own. This in general is called a GPGPU, or general purpose computing on graphics processing units.
ID: 1599028 · Report as offensive
BEParker

Send message
Joined: 24 Jun 99
Posts: 17
Credit: 10,317,641
RAC: 18
United States
Message 1599120 - Posted: 9 Nov 2014, 22:10:09 UTC - in response to Message 1599026.  

Everything seems to be back running normally. I have processed a couple of CUDA23 work units with no apparent issues.

Bruce
ID: 1599120 · Report as offensive
DanHansen@Denmark
Volunteer tester
Avatar

Send message
Joined: 14 Nov 12
Posts: 194
Credit: 5,881,465
RAC: 0
Denmark
Message 1599170 - Posted: 10 Nov 2014, 0:02:18 UTC

Hi Claggy,

Maybe you can help me. Is these cards affected?:
OS: Win7 32bit/64bit - Asus GeForce GTX770
OS: Linux 64 bit - Asus GeForce GT640

No, none of those GPUs are pre-Fermi's.


Thanks my friend ;) Thank you for letting me know ;)
Project Headless CLI Linux Multiple GPU Boinc Servers
Ubuntu Server 14.04.1 64bit
Kernel 3.13.0-32-generic
CPU's i5-4690K
GPU's GT640/GTX750TI
Nvidia v.340.29
BOINC v.7.2.42

ID: 1599170 · Report as offensive
styxdogg

Send message
Joined: 7 Mar 01
Posts: 3
Credit: 1,540,836
RAC: 0
United States
Message 1599433 - Posted: 10 Nov 2014, 15:55:32 UTC - in response to Message 1599026.  
Last modified: 10 Nov 2014, 15:56:28 UTC

I am having the same kind of problem with a GeForce 9500 GT card. It was working fine until the middle October. I checked the message boards and I, obviously, fall into the pre-Fermi category, but no combination of older or newer drivers allows a CUDA data file to be used for any more than about 15 seconds, then resets to 0, starts over, and eventually reports "Computational Error". I just did an uninstall and a re-install of BOINC and a disconnect and re-connect for the SETI project.

I realize I need newer hardware to be able to do newer things. The only reason I had a video card with a GPU was to use it for SETI. I don't do gaming, so old is not an issue for that. I'm just trying to figure out if my system is the reason the GPU stopped working of if something changed in BOING that rendered my GPU useless.

Can you do the Remove Seti and add Seti again, I'd have expected to see a lot of abandoned work, But there is none,
try and complete as many CPU tasks as possible before you do it. (Set NNT)

The scheduler did once send you Cuda32, Cuda42 and Cuda50 work, and have high Max tasks per day for all three, But insists in only sending Cuda22 and Cuda23 work now, strange.

Claggy


Removed and re-added Seti. Also made sure the newer PhysX package was removed and clean-installed NVIDIA driver 266.58 (as recommended in an earlier message in this thread).

A CUDA23 file started up and had been running for 5 minutes, now, with no problems noted. Download a whole bunch more CUDA23 files after that.

A "contaminated" file may have been the issue. If the current CUDA23 file successfully finishes (in about 2 hours) I will know, I guess.

As for not getting any CUDA32, CUDA42, or CUDA50 work, I remember getting those in the past, but can't remember when they stopped.

Thank You for your help.

Bruce Parker
Dumfries, VA



I posted about a problem with cuda23 getting errors previously, but I didn't get a reply. I just did what you did here.. hope it fixes it. I haven't got any new work yet (says there is none) so I can't check. -Matt

NVIDIA GeForce GT 330 (1024MB) driver: 337.88 OpenCL: 1.0
ID: 1599433 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1599506 - Posted: 10 Nov 2014, 20:35:22 UTC - in response to Message 1599433.  
Last modified: 10 Nov 2014, 20:36:12 UTC

I posted about a problem with cuda23 getting errors previously, but I didn't get a reply. I just did what you did here.. hope it fixes it. I haven't got any new work yet (says there is none) so I can't check. -Matt

NVIDIA GeForce GT 330 (1024MB) driver: 337.88 OpenCL: 1.0

Your problem is the same as Thomas Arnold's and BEParker's problem,
As i responded to Both of them, If you do a Remove and Add Seti it should fix your problem,
It did for BEParker, Thomas Arnold didn't respond.

Claggy
ID: 1599506 · Report as offensive
styxdogg

Send message
Joined: 7 Mar 01
Posts: 3
Credit: 1,540,836
RAC: 0
United States
Message 1600239 - Posted: 12 Nov 2014, 15:19:05 UTC - in response to Message 1599506.  

I posted about a problem with cuda23 getting errors previously, but I didn't get a reply. I just did what you did here.. hope it fixes it. I haven't got any new work yet (says there is none) so I can't check. -Matt

NVIDIA GeForce GT 330 (1024MB) driver: 337.88 OpenCL: 1.0

Your problem is the same as Thomas Arnold's and BEParker's problem,
As i responded to Both of them, If you do a Remove and Add Seti it should fix your problem,
It did for BEParker, Thomas Arnold didn't respond.

Claggy


Thanks. Just a note it, looking at my results again, it was both cuda22 and cuda23 that showed as having an 'error while computing'. I see now in my tasks I have a cuda 22 at about 60% done and I forced a cuda 23 which is also now at about 60%. I assume this means all is well now. -Matt
ID: 1600239 · Report as offensive
triplemmm

Send message
Joined: 21 Nov 00
Posts: 27
Credit: 8,764,139
RAC: 13
Australia
Message 1600954 - Posted: 14 Nov 2014, 6:51:20 UTC

i have recently upgraded my pc to AMD a8-5600k with radeon hd graphic 3.60Ghz and also a Nvidia geforce 210 card. my concern is how do i know if my GPU is working and calculating my Seti WU's? As i have noticed that i have dropped from 4 WU at one time to three and wonder if this is the reason?
M MM
ID: 1600954 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 149
Credit: 9,783,406
RAC: 9
United States
Message 1601045 - Posted: 14 Nov 2014, 12:10:37 UTC
Last modified: 14 Nov 2014, 12:10:54 UTC

I have an update, and it's good news I believe, in regards to my NVIDIA bug 1574543:
https://developer.nvidia.com/nvbugs/cuda/edit/1574543

5 November 2014 9:32 pm -- Kevin Kang
Hi Jacob, thanks for the reporting. We have reproduced this issue and have assigned it to the appropriate developer team for reviewing. Thanks!


12 November 2014 11:28 am -- JacobKlein
Hi Kevin / Team: It has been a week without a status update. Could you kindly please consider answering the questions that I asked in the bug details? Thanks, Jacob Klein


[NEW]
14 November 2014 1:21 am -- Kevin Kang
Hi Jacob, Sorry for update on this issue late. As noted in release notes, the R340 drivers will continue to support the Tesla generation of NVIDIA GPUs until April 1, 2016, and the NVIDIA support team will continue to address driver issues for these products in driver branches up to and including Release 340. However, future driver enhancements and optimizations in driver releases after Release 340 will not support these products. Our developer team is working on this issue actively for the future R340 driver release, we'll keep you posted once it has been fixed. Sorry for any inconvenience! Thanks, Kevin
ID: 1601045 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1601157 - Posted: 14 Nov 2014, 19:05:42 UTC - in response to Message 1601045.  
Last modified: 14 Nov 2014, 19:07:41 UTC

[NEW]
14 November 2014 1:21 am -- Kevin Kang
Hi Jacob, Sorry for update on this issue late. As noted in release notes, the R340 drivers will continue to support the Tesla generation of NVIDIA GPUs until April 1, 2016, and the NVIDIA support team will continue to address driver issues for these products in driver branches up to and including Release 340. However, future driver enhancements and optimizations in driver releases after Release 340 will not support these products. Our developer team is working on this issue actively for the future R340 driver release, we'll keep you posted once it has been fixed. Sorry for any inconvenience! Thanks, Kevin


This certainly concurs with the way I interpreted the existing Cuda documentation (not in an pure OpenCL context). We're at an inevitable juncture where upcoming OSes will require hardware features only available in Fermi+, mainly 64 bit addressing, and the emulations having taken place to bring the older gens along so far have become unwieldly (making the older cards crunch slower with each driver iteration).

The line makes practical sense for nVidia. It's just really unfortunate that the timing of these moves was near Maxwell release, resulting in the usual driver maturation problems converging with unrelated major changes. Could definitely have been done cleaner IMO, though I suspect there'll be some more growing pains yet.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1601157 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 149
Credit: 9,783,406
RAC: 9
United States
Message 1601163 - Posted: 14 Nov 2014, 19:12:38 UTC - in response to Message 1601157.  

It was known, by me, since March, that R340 would be the "end of the line" for pre-Fermi GPUs, per http://nvidia.custhelp.com/app/answers/detail/a_id/3473. I chatted with an NVIDIA agent, then, and confirmed that pre-Fermi GPUs would not be recognized by R343+ drivers. So, that prompted me to pull my GTS 240 GPUs out of my main rig, several months ago.

It was NOT known to me, that the R343 drivers running on my laptop's FX 3800M GPU... were corrupting BOINC data.

I am happy that it seems NVIDIA will honor their R340 support to fix the problem for the OpenCL SDK examples that I proved were failing. And I'm crossing my fingers that this also resolves the R340 data integrity issues that BOINC OpenCl applications are having. I feel pretty confident that it will!

Regards,
Jacob
ID: 1601163 · Report as offensive
triplemmm

Send message
Joined: 21 Nov 00
Posts: 27
Credit: 8,764,139
RAC: 13
Australia
Message 1603303 - Posted: 20 Nov 2014, 22:45:57 UTC - in response to Message 1600954.  

I have done all that is mentioned in this board to get my GPU to start working seti WU's but still no joy. Is there something I have missed or not done to get my GPU to start working thru the seti WU's. The only reason i upgraded my Nvidiu card was so I could do more seti WU's. Can anyone help me to get my system working at it best for seti WU's.
ID: 1603303 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1603316 - Posted: 20 Nov 2014, 23:03:36 UTC

Apart from there being very few tasks around just now for everyone you are still running the wrong version of the Nvidia drivers - you should be using an older set, 337.88 appears to be about as new as you can go - which can be downloaded from the Nvidia website by looking for the older drivers. Do a an advanced/clean installation otherwise the drivers won't be properly replaced.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1603316 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1603322 - Posted: 20 Nov 2014, 23:18:08 UTC - in response to Message 1603303.  
Last modified: 20 Nov 2014, 23:56:32 UTC

I have done all that is mentioned in this board to get my GPU to start working seti WU's but still no joy. Is there something I have missed or not done to get my GPU to start working thru the seti WU's. The only reason i upgraded my Nvidiu card was so I could do more seti WU's. Can anyone help me to get my system working at it best for seti WU's.

I'm starting to think that the project has limited Pre-Fermi GPUs to the MBv7 Cuda23 and Cuda22 apps,
and put in a Max driver limit in too, Inspite of the MBv7 CUDA apps NOT being effected by the 340.52 OpenCL Pre-Fermi driver Bug.

The way round the driver limit is to install the Lunatics apps, But just install the Cuda23 MBv7 GPU app, DON'T install the NV OpenCL AP GPU app,
it won't work properly on Pre-Fermi GPUs (Edit) on 340.xx drivers.

Claggy
ID: 1603322 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1603332 - Posted: 20 Nov 2014, 23:53:09 UTC - in response to Message 1603322.  

I have done all that is mentioned in this board to get my GPU to start working seti WU's but still no joy. Is there something I have missed or not done to get my GPU to start working thru the seti WU's. The only reason i upgraded my Nvidiu card was so I could do more seti WU's. Can anyone help me to get my system working at it best for seti WU's.

I'm starting to think that the project has limited Pre-Fermi GPUs to the MBv7 Cuda23 and Cuda22 apps,
and put in a Max driver limit in too, Inspite of the MBv7 CUDA apps NOT being effected by the 340.52 OpenCL Pre-Fermi driver Bug.

The way round the driver limit is to install the Lunatics apps, But just install the Cuda23 MBv7 GPU app, DON'T install the NV OpenCL AP GPU app,
it won't work properly on Pre-Fermi GPUs.

Claggy

The OpenCL AP app is fine, provided you haven't installed the 340.xx drivers, or if you have, you uninstall them again first, and revert to 337.xx or earlier.
ID: 1603332 · Report as offensive
Jacob Klein
Volunteer tester

Send message
Joined: 15 Apr 11
Posts: 149
Credit: 9,783,406
RAC: 9
United States
Message 1603349 - Posted: 21 Nov 2014, 0:14:12 UTC - in response to Message 1603332.  

Just a little more clarification, since 341.05 was released as an R340 driver.

R340 drivers, like 340.xx and 341.xx, aren't *yet* working right for all OpenCL apps... until NVIDIA hopefully fixes things for these R340 drivers. I still have hope! :)
ID: 1603349 · Report as offensive
Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 13 · Next

Message boards : Number crunching : @Pre-FERMI nVidia GPU users: Important warning


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.