"Sleepy" NV GPU - how to "wake up" ?

Message boards : Number crunching : "Sleepy" NV GPU - how to "wake up" ?
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1308252 - Posted: 21 Nov 2012, 6:06:29 UTC
Last modified: 21 Nov 2012, 6:13:45 UTC

My NV GTX260 based host with Windows Server 2003 x64 worked more than week w/o failures now so I paid not much attention to it.
But recently, looking into host results I discovered very big elapsed times for some results (like 7k seconds, ~2h). Comparison with another tasks of same AR showed that such times not normal for this host indeed. Same (and it was midrange AR) AR GPU can complete in about 800 seconds... So, slowdown is really huge.
What was different in config:
usually I logon on that host locally, with attached console. This results in fan becoming noiser. I attributed this to usage of RiveTuner some years ago, when host contained GSO9600. Fan was set to manual with high percentage so, perhaps, those old settings loaded only when user logon locally(it's important).
But looks like failures of this host correlate with attached keyboard so last week I disconnected keyboard and do logon only remotely. Cause it's Win2003 GPU is visible to BOINC (BOINC as service, of course) even w/o logon at all.
But apparently w/o local logon GPU enters in some very slow state...
The question is: can GTX260 automatically lower its frequency to some very low level (much lower than stock freq) in case of overheating? Can silent fan (and fan is silent if no local logon was made) be the reason of such slowdown?
And if yes - how to make it rotate faster on true "system startup" (RivaTuner states so) and not on user local logon (as it happens in reality)?
I use RivaTuner 2.21. Maybe some other tools can speedup fan w/o user logon ?

ADDON:
Example of "slow" task
Anonymous platform (NVIDIA GPU)
2685038681 1102466023 27 Oct 2012 | 20:15:00 UTC 20 Nov 2012 | 8:36:21 UTC Completed and validated 7,136.00 96.63 151.43 SETI@home Enhanced
Anonymous platform (NVIDIA GPU)

Multibeam x41g Preview, Cuda 3.20

Legacy setiathome_enhanced V6 mode.
Work Unit Info:
...............
WU true angle range is : 0.365709
Device 1: GeForce GTX 260, 895 MiB, regsPerBlock 16384
computeCap 1.3, multiProcs 27
clockRate = 1242000

Example of "usual" fast task:

2685130193 1102507523 27 Oct 2012 | 21:37:02 UTC 20 Nov 2012 | 12:46:19 UTC Completed and validated 828.03 77.61 120.76 SETI@home Enhanced
Anonymous platform (NVIDIA GPU)

Multibeam x41g Preview, Cuda 3.20

Legacy setiathome_enhanced V6 mode.
Work Unit Info:
...............
WU true angle range is : 0.368613
Device 1: GeForce GTX 260, 895 MiB, regsPerBlock 16384
computeCap 1.3, multiProcs 27
clockRate = 1242000

(same "clock" values, but it's possible that clockRate reported is maximal one or dynamically lowered through task execution).[/i]
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1308252 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1308260 - Posted: 21 Nov 2012, 6:52:14 UTC - in response to Message 1308252.  

According to http://www.tomshardware.com/reviews/radeon-geforce-comparison,2007-8.html the card can throttle due to temperature issues.

If temperature caused by a slow fan is your problem, MSI Afterburner (also EVGA Precision X, but I don't know if it will work on your card) has a check-box to start the program with your settings when Windows starts.

I notice that you are using an older driver.

There has been a recent Windows Update that forced me to reinstall my NVIDIA drivers on two of my computers, but none of them are running your cards or OS.

However, if I recall correctly, these were updates that included your OS.

I have never had exactly the problem you are having.

I have no idea how any of this works running BOINC as a service.





ID: 1308260 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1308294 - Posted: 21 Nov 2012, 10:14:01 UTC
Last modified: 21 Nov 2012, 10:57:19 UTC

I have some similar problem in the past with a host with XP and running 590+2x580 and Boinc running as a service.

At least in my case the problem is EVGA Precision does not run until i log to the host. Direct log or remote log. Of course remote log only with VNC or you will loose the GPU driver.

If no log is done, that makes the GPUs reach very fast high temperatures and an internal sensor automatic slow down the clock (i belive is a feature builded in the GPU).

The only solution i find was make an autolog-on on the host with a secondary user and start the EVGA precision on that user, so the EVGA automaticaly rise the fan speed and cool the GPU to a confortable level.

The only thing is, as the secondary user is loged, teoricaly your computer could be exposed if anyone access him localy, i solve the problem by making the secondary user a very limiter user, practicaly i only could fire the EVGA program, but anyway that is cleary a possible point of weekness, and if some one acess localy the host by using the secondary already loged user, it could stop by a mistake the EVGA program and the problem will return, i avoid it by removing the local keyboard/mouse access on this host, but of course thats is a very week security measure.

Other compleate different thing than eventualy make something like that (produce a random very slow to process WU) as find by Jason´s in some host the CR of the memory goes automaticaly to 1T, but the correct setting is 2T, at least on my MB, for some misterious reason in some WU that´s makes the GPU slow down the clock rate. If this happening and you do a reset everything back to normal, even the problematic WU when returns to work fast.

Solution? go to bios set-up and manualy fix the CR of the memory to 1T.

Please understand me, i just try to show two similar situation, i have not your technical skills and cleary don´t understand why some things happens, just know this 2 situations happens with me, and whow i fix them, few months ago.

Hope helps

BTW - If you find any program that speed up the fan without need to log or how to run EVGA precision as a service, please share with me your findings, i will be very grateful, still autolog my XT host and if the EVGA program not running, it will go to overheat very fast.
ID: 1308294 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1308349 - Posted: 21 Nov 2012, 13:52:31 UTC

FWIW: I noticed a slowdown on one of my GPUs a few days ago - WUs that take around 15-25 minutes were dragging on for upwards of 2 hours, for no apparent reason. EVGA Precision showed the card running at full frequency. I couldn't figure out the problem, so in best old preogrammer style, I restarted the machine - and that fixed my problem (which has not recurred so far).

So I assume something in the card got confused - memory leak, maybe?
ID: 1308349 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1308350 - Posted: 21 Nov 2012, 14:02:16 UTC

Have you tried running RivaTuner as a service as well? I don't recall the full syntax at the moment, but using the command line "sc create" I think is where you would want to start.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1308350 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1308351 - Posted: 21 Nov 2012, 14:09:21 UTC

Thanks all a lot!
As I understood this problem worth some consideration indeed. Auto-logon would be not good cause it's not home PC indeed.
Is it possible to flash required fan settings directly in GPU own BIOS memory ?
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1308351 · Report as offensive
Horacio

Send message
Joined: 14 Jan 00
Posts: 536
Credit: 75,967,266
RAC: 0
Argentina
Message 1308362 - Posted: 21 Nov 2012, 14:47:35 UTC - in response to Message 1308351.  

Thanks all a lot!
As I understood this problem worth some consideration indeed. Auto-logon would be not good cause it's not home PC indeed.
Is it possible to flash required fan settings directly in GPU own BIOS memory ?

It seems rather difficult... You will need a pre-made BIOS for the specific model/vendor with that FAN speed changed, or you will need to read the current BIOS and rebuild/hack it and then overrite it on the harware...

I guess it will be more easy to find how to control fan through software and then write your own little service... (Not an easy fix for everybody, but I guess this wont be too difficult for you Raistmer)

ID: 1308362 · Report as offensive
Profile S@NL - eFMer - efmer.com/boinc
Volunteer tester
Avatar

Send message
Joined: 7 Jun 99
Posts: 512
Credit: 148,746,305
RAC: 0
United States
Message 1308366 - Posted: 21 Nov 2012, 14:52:13 UTC - in response to Message 1308351.  

Thanks all a lot!
As I understood this problem worth some consideration indeed. Auto-logon would be not good cause it's not home PC indeed.
Is it possible to flash required fan settings directly in GPU own BIOS memory ?

You should be able to use manual control with the slider. (Disable the auto mode)
This should stick, at least it did on my system.

The last driver seems to down clock some cards. Seems to happen close to 80C.
Probably the keep the temps in a safe zone.
TThrottle Control your temperatures. BoincTasks The best way to view BOINC. Anza Borrego Desert hiking.
ID: 1308366 · Report as offensive
Profile shizaru
Volunteer tester
Avatar

Send message
Joined: 14 Jun 04
Posts: 1130
Credit: 1,967,904
RAC: 0
Greece
Message 1308371 - Posted: 21 Nov 2012, 15:07:10 UTC - in response to Message 1308351.  

Is it possible to flash required fan settings directly in GPU own BIOS memory?


Maybe you can try this instead:
http://www.nvidia.com/object/nvidia_system_tools_6.02.html

It's got a fan speed setting and the tools are tied into the NVIDIA Control Panel and can be made to work at startup, but I'm not sure about your Windows version.
ID: 1308371 · Report as offensive
Profile S@NL - eFMer - efmer.com/boinc
Volunteer tester
Avatar

Send message
Joined: 7 Jun 99
Posts: 512
Credit: 148,746,305
RAC: 0
United States
Message 1308376 - Posted: 21 Nov 2012, 15:25:11 UTC - in response to Message 1308371.  

Is it possible to flash required fan settings directly in GPU own BIOS memory?


Maybe you can try this instead:
http://www.nvidia.com/object/nvidia_system_tools_6.02.html

It's got a fan speed setting and the tools are tied into the NVIDIA Control Panel and can be made to work at startup, but I'm not sure about your Windows version.

Works nice on older cards.
It's a legacy program, they stopped developing a while.
TThrottle Control your temperatures. BoincTasks The best way to view BOINC. Anza Borrego Desert hiking.
ID: 1308376 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1308431 - Posted: 21 Nov 2012, 18:24:09 UTC - in response to Message 1308349.  

FWIW: I noticed a slowdown on one of my GPUs a few days ago - WUs that take around 15-25 minutes were dragging on for upwards of 2 hours, for no apparent reason. EVGA Precision showed the card running at full frequency. I couldn't figure out the problem, so in best old preogrammer style, I restarted the machine - and that fixed my problem (which has not recurred so far).

So I assume something in the card got confused - memory leak, maybe?


Check you memory timings, look the post i made before, thats exactly what happens with me months ago due the memory CR problem.

ID: 1308431 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1309182 - Posted: 23 Nov 2012, 9:18:12 UTC - in response to Message 1308431.  

FWIW: I noticed a slowdown on one of my GPUs a few days ago - WUs that take around 15-25 minutes were dragging on for upwards of 2 hours, for no apparent reason. EVGA Precision showed the card running at full frequency. I couldn't figure out the problem, so in best old preogrammer style, I restarted the machine - and that fixed my problem (which has not recurred so far).

So I assume something in the card got confused - memory leak, maybe?


Check you memory timings, look the post i made before, thats exactly what happens with me months ago due the memory CR problem.

Thanks. GPU restored its performance w/o reboot, just after logon and fan speed increase. Currently it runs few days already w/o slowdown (with fan at increased rotation speed/noise level). So I think the problem in GPU overheating rather than BIOS settings (was no reboot) but will check BIOS settings too when reboot will required/scheduled.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1309182 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1309183 - Posted: 23 Nov 2012, 9:19:43 UTC - in response to Message 1308376.  
Last modified: 23 Nov 2012, 9:49:37 UTC

Is it possible to flash required fan settings directly in GPU own BIOS memory?


Maybe you can try this instead:
http://www.nvidia.com/object/nvidia_system_tools_6.02.html

It's got a fan speed setting and the tools are tied into the NVIDIA Control Panel and can be made to work at startup, but I'm not sure about your Windows version.

Works nice on older cards.
It's a legacy program, they stopped developing a while.


EDIT: here the survived link on binary from some mirror:
Unfortunately, binary not available for download. only description is available.
http://nvworld.ru/utilities/nvsystools/
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1309183 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1309184 - Posted: 23 Nov 2012, 9:21:12 UTC - in response to Message 1308366.  


You should be able to use manual control with the slider. (Disable the auto mode)
This should stick, at least it did on my system.

I did not find any fan/OC related options in NV control panel of 263.06 driver. Did you mean NV own control utility or slider in RivaTuner app ?

SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1309184 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1309191 - Posted: 23 Nov 2012, 9:41:11 UTC - in response to Message 1308362.  

Thanks all a lot!
As I understood this problem worth some consideration indeed. Auto-logon would be not good cause it's not home PC indeed.
Is it possible to flash required fan settings directly in GPU own BIOS memory ?

It seems rather difficult... You will need a pre-made BIOS for the specific model/vendor with that FAN speed changed, or you will need to read the current BIOS and rebuild/hack it and then overrite it on the harware...



Just recalled some discussion about leaving OCed values in software and apply them on each boot or flashing them onto GPU. Maybe it was about ATi GPUs, not NV ones...
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1309191 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34256
Credit: 79,922,639
RAC: 80
Germany
Message 1309194 - Posted: 23 Nov 2012, 9:46:35 UTC

I`m using MSI Afterburner.
Works on nvidia also.



With each crime and every kindness we birth our future.
ID: 1309194 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1309196 - Posted: 23 Nov 2012, 9:50:28 UTC - in response to Message 1309194.  
Last modified: 23 Nov 2012, 9:52:48 UTC

I`m using MSI Afterburner.
Works on nvidia also.

Thanks, Mike. Downloading...
EDIT: looks like this utility is successor of RivaTuner too.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1309196 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1309198 - Posted: 23 Nov 2012, 10:00:18 UTC - in response to Message 1309196.  
Last modified: 23 Nov 2012, 10:00:37 UTC

I`m using MSI Afterburner.
Works on nvidia also.

Thanks, Mike. Downloading...
EDIT: looks like this utility is successor of RivaTuner too.

MSI Afterburner works as a service? With out need to log on any user?
ID: 1309198 · Report as offensive
Profile S@NL - eFMer - efmer.com/boinc
Volunteer tester
Avatar

Send message
Joined: 7 Jun 99
Posts: 512
Credit: 148,746,305
RAC: 0
United States
Message 1309209 - Posted: 23 Nov 2012, 10:48:57 UTC - in response to Message 1309198.  

I`m using MSI Afterburner.
Works on nvidia also.

Thanks, Mike. Downloading...
EDIT: looks like this utility is successor of RivaTuner too.

MSI Afterburner works as a service? With out need to log on any user?

You need to log in.
GPU access is only allowed for a logged in user. So no service.
TThrottle Control your temperatures. BoincTasks The best way to view BOINC. Anza Borrego Desert hiking.
ID: 1309209 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1309217 - Posted: 23 Nov 2012, 11:25:33 UTC - in response to Message 1309209.  

I`m using MSI Afterburner.
Works on nvidia also.

Thanks, Mike. Downloading...
EDIT: looks like this utility is successor of RivaTuner too.

MSI Afterburner works as a service? With out need to log on any user?

You need to log in.
GPU access is only allowed for a logged in user. So no service.

Windows Server 2003 (see opening post) is comparable to Windows XP - and Windows XP allows services to access the GPU, even when the service logs on using a different account than the foreground user.
ID: 1309217 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : "Sleepy" NV GPU - how to "wake up" ?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.