"Sleepy" NV GPU - how to "wake up" ?


log in

Advanced search

Message boards : Number crunching : "Sleepy" NV GPU - how to "wake up" ?

1 · 2 · Next
Author Message
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3420
Credit: 46,535,924
RAC: 9,270
Russia
Message 1308252 - Posted: 21 Nov 2012, 6:06:29 UTC
Last modified: 21 Nov 2012, 6:13:45 UTC

My NV GTX260 based host with Windows Server 2003 x64 worked more than week w/o failures now so I paid not much attention to it.
But recently, looking into host results I discovered very big elapsed times for some results (like 7k seconds, ~2h). Comparison with another tasks of same AR showed that such times not normal for this host indeed. Same (and it was midrange AR) AR GPU can complete in about 800 seconds... So, slowdown is really huge.
What was different in config:
usually I logon on that host locally, with attached console. This results in fan becoming noiser. I attributed this to usage of RiveTuner some years ago, when host contained GSO9600. Fan was set to manual with high percentage so, perhaps, those old settings loaded only when user logon locally(it's important).
But looks like failures of this host correlate with attached keyboard so last week I disconnected keyboard and do logon only remotely. Cause it's Win2003 GPU is visible to BOINC (BOINC as service, of course) even w/o logon at all.
But apparently w/o local logon GPU enters in some very slow state...
The question is: can GTX260 automatically lower its frequency to some very low level (much lower than stock freq) in case of overheating? Can silent fan (and fan is silent if no local logon was made) be the reason of such slowdown?
And if yes - how to make it rotate faster on true "system startup" (RivaTuner states so) and not on user local logon (as it happens in reality)?
I use RivaTuner 2.21. Maybe some other tools can speedup fan w/o user logon ?

ADDON:
Example of "slow" task
Anonymous platform (NVIDIA GPU)
2685038681 1102466023 27 Oct 2012 | 20:15:00 UTC 20 Nov 2012 | 8:36:21 UTC Completed and validated 7,136.00 96.63 151.43 SETI@home Enhanced
Anonymous platform (NVIDIA GPU)

Multibeam x41g Preview, Cuda 3.20

Legacy setiathome_enhanced V6 mode.
Work Unit Info:
...............
WU true angle range is : 0.365709
Device 1: GeForce GTX 260, 895 MiB, regsPerBlock 16384
computeCap 1.3, multiProcs 27
clockRate = 1242000

Example of "usual" fast task:

2685130193 1102507523 27 Oct 2012 | 21:37:02 UTC 20 Nov 2012 | 12:46:19 UTC Completed and validated 828.03 77.61 120.76 SETI@home Enhanced
Anonymous platform (NVIDIA GPU)

Multibeam x41g Preview, Cuda 3.20

Legacy setiathome_enhanced V6 mode.
Work Unit Info:
...............
WU true angle range is : 0.368613
Device 1: GeForce GTX 260, 895 MiB, regsPerBlock 16384
computeCap 1.3, multiProcs 27
clockRate = 1242000

(same "clock" values, but it's possible that clockRate reported is maximal one or dynamically lowered through task execution).[/i]
____________

tbretProject donor
Volunteer tester
Avatar
Send message
Joined: 28 May 99
Posts: 2795
Credit: 209,825,442
RAC: 109,772
United States
Message 1308260 - Posted: 21 Nov 2012, 6:52:14 UTC - in response to Message 1308252.

According to http://www.tomshardware.com/reviews/radeon-geforce-comparison,2007-8.html the card can throttle due to temperature issues.

If temperature caused by a slow fan is your problem, MSI Afterburner (also EVGA Precision X, but I don't know if it will work on your card) has a check-box to start the program with your settings when Windows starts.

I notice that you are using an older driver.

There has been a recent Windows Update that forced me to reinstall my NVIDIA drivers on two of my computers, but none of them are running your cards or OS.

However, if I recall correctly, these were updates that included your OS.

I have never had exactly the problem you are having.

I have no idea how any of this works running BOINC as a service.





juan BFBProject donor
Volunteer tester
Avatar
Send message
Joined: 16 Mar 07
Posts: 5315
Credit: 294,842,760
RAC: 471,760
Brazil
Message 1308294 - Posted: 21 Nov 2012, 10:14:01 UTC
Last modified: 21 Nov 2012, 10:57:19 UTC

I have some similar problem in the past with a host with XP and running 590+2x580 and Boinc running as a service.

At least in my case the problem is EVGA Precision does not run until i log to the host. Direct log or remote log. Of course remote log only with VNC or you will loose the GPU driver.

If no log is done, that makes the GPUs reach very fast high temperatures and an internal sensor automatic slow down the clock (i belive is a feature builded in the GPU).

The only solution i find was make an autolog-on on the host with a secondary user and start the EVGA precision on that user, so the EVGA automaticaly rise the fan speed and cool the GPU to a confortable level.

The only thing is, as the secondary user is loged, teoricaly your computer could be exposed if anyone access him localy, i solve the problem by making the secondary user a very limiter user, practicaly i only could fire the EVGA program, but anyway that is cleary a possible point of weekness, and if some one acess localy the host by using the secondary already loged user, it could stop by a mistake the EVGA program and the problem will return, i avoid it by removing the local keyboard/mouse access on this host, but of course thats is a very week security measure.

Other compleate different thing than eventualy make something like that (produce a random very slow to process WU) as find by Jason´s in some host the CR of the memory goes automaticaly to 1T, but the correct setting is 2T, at least on my MB, for some misterious reason in some WU that´s makes the GPU slow down the clock rate. If this happening and you do a reset everything back to normal, even the problematic WU when returns to work fast.

Solution? go to bios set-up and manualy fix the CR of the memory to 1T.

Please understand me, i just try to show two similar situation, i have not your technical skills and cleary don´t understand why some things happens, just know this 2 situations happens with me, and whow i fix them, few months ago.

Hope helps

BTW - If you find any program that speed up the fan without need to log or how to run EVGA precision as a service, please share with me your findings, i will be very grateful, still autolog my XT host and if the EVGA program not running, it will go to overheat very fast.
____________

jravin
Send message
Joined: 25 Mar 02
Posts: 930
Credit: 99,718,638
RAC: 85,483
United States
Message 1308349 - Posted: 21 Nov 2012, 13:52:31 UTC

FWIW: I noticed a slowdown on one of my GPUs a few days ago - WUs that take around 15-25 minutes were dragging on for upwards of 2 hours, for no apparent reason. EVGA Precision showed the card running at full frequency. I couldn't figure out the problem, so in best old preogrammer style, I restarted the machine - and that fixed my problem (which has not recurred so far).

So I assume something in the card got confused - memory leak, maybe?
____________

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 4187
Credit: 114,674,343
RAC: 142,567
United States
Message 1308350 - Posted: 21 Nov 2012, 14:02:16 UTC

Have you tried running RivaTuner as a service as well? I don't recall the full syntax at the moment, but using the command line "sc create" I think is where you would want to start.
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3420
Credit: 46,535,924
RAC: 9,270
Russia
Message 1308351 - Posted: 21 Nov 2012, 14:09:21 UTC

Thanks all a lot!
As I understood this problem worth some consideration indeed. Auto-logon would be not good cause it's not home PC indeed.
Is it possible to flash required fan settings directly in GPU own BIOS memory ?
____________

Horacio
Send message
Joined: 14 Jan 00
Posts: 536
Credit: 73,650,655
RAC: 80,981
Argentina
Message 1308362 - Posted: 21 Nov 2012, 14:47:35 UTC - in response to Message 1308351.

Thanks all a lot!
As I understood this problem worth some consideration indeed. Auto-logon would be not good cause it's not home PC indeed.
Is it possible to flash required fan settings directly in GPU own BIOS memory ?

It seems rather difficult... You will need a pre-made BIOS for the specific model/vendor with that FAN speed changed, or you will need to read the current BIOS and rebuild/hack it and then overrite it on the harware...

I guess it will be more easy to find how to control fan through software and then write your own little service... (Not an easy fix for everybody, but I guess this wont be too difficult for you Raistmer)

____________

Profile S@NL - eFMer - efmer.com/boincProject donor
Volunteer tester
Avatar
Send message
Joined: 7 Jun 99
Posts: 512
Credit: 129,003,670
RAC: 35,820
United States
Message 1308366 - Posted: 21 Nov 2012, 14:52:13 UTC - in response to Message 1308351.

Thanks all a lot!
As I understood this problem worth some consideration indeed. Auto-logon would be not good cause it's not home PC indeed.
Is it possible to flash required fan settings directly in GPU own BIOS memory ?

You should be able to use manual control with the slider. (Disable the auto mode)
This should stick, at least it did on my system.

The last driver seems to down clock some cards. Seems to happen close to 80C.
Probably the keep the temps in a safe zone.
____________
TThrottle Control your temperatures. BoincTasks The best way to view BOINC. Anza Borrego Desert hiking.

Profile Alex Storey
Volunteer tester
Avatar
Send message
Joined: 14 Jun 04
Posts: 537
Credit: 1,650,397
RAC: 335
Greece
Message 1308371 - Posted: 21 Nov 2012, 15:07:10 UTC - in response to Message 1308351.

Is it possible to flash required fan settings directly in GPU own BIOS memory?


Maybe you can try this instead:
http://www.nvidia.com/object/nvidia_system_tools_6.02.html

It's got a fan speed setting and the tools are tied into the NVIDIA Control Panel and can be made to work at startup, but I'm not sure about your Windows version.

Profile S@NL - eFMer - efmer.com/boincProject donor
Volunteer tester
Avatar
Send message
Joined: 7 Jun 99
Posts: 512
Credit: 129,003,670
RAC: 35,820
United States
Message 1308376 - Posted: 21 Nov 2012, 15:25:11 UTC - in response to Message 1308371.

Is it possible to flash required fan settings directly in GPU own BIOS memory?


Maybe you can try this instead:
http://www.nvidia.com/object/nvidia_system_tools_6.02.html

It's got a fan speed setting and the tools are tied into the NVIDIA Control Panel and can be made to work at startup, but I'm not sure about your Windows version.

Works nice on older cards.
It's a legacy program, they stopped developing a while.
____________
TThrottle Control your temperatures. BoincTasks The best way to view BOINC. Anza Borrego Desert hiking.

juan BFBProject donor
Volunteer tester
Avatar
Send message
Joined: 16 Mar 07
Posts: 5315
Credit: 294,842,760
RAC: 471,760
Brazil
Message 1308431 - Posted: 21 Nov 2012, 18:24:09 UTC - in response to Message 1308349.

FWIW: I noticed a slowdown on one of my GPUs a few days ago - WUs that take around 15-25 minutes were dragging on for upwards of 2 hours, for no apparent reason. EVGA Precision showed the card running at full frequency. I couldn't figure out the problem, so in best old preogrammer style, I restarted the machine - and that fixed my problem (which has not recurred so far).

So I assume something in the card got confused - memory leak, maybe?


Check you memory timings, look the post i made before, thats exactly what happens with me months ago due the memory CR problem.

____________

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3420
Credit: 46,535,924
RAC: 9,270
Russia
Message 1309182 - Posted: 23 Nov 2012, 9:18:12 UTC - in response to Message 1308431.

FWIW: I noticed a slowdown on one of my GPUs a few days ago - WUs that take around 15-25 minutes were dragging on for upwards of 2 hours, for no apparent reason. EVGA Precision showed the card running at full frequency. I couldn't figure out the problem, so in best old preogrammer style, I restarted the machine - and that fixed my problem (which has not recurred so far).

So I assume something in the card got confused - memory leak, maybe?


Check you memory timings, look the post i made before, thats exactly what happens with me months ago due the memory CR problem.

Thanks. GPU restored its performance w/o reboot, just after logon and fan speed increase. Currently it runs few days already w/o slowdown (with fan at increased rotation speed/noise level). So I think the problem in GPU overheating rather than BIOS settings (was no reboot) but will check BIOS settings too when reboot will required/scheduled.
____________

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3420
Credit: 46,535,924
RAC: 9,270
Russia
Message 1309183 - Posted: 23 Nov 2012, 9:19:43 UTC - in response to Message 1308376.
Last modified: 23 Nov 2012, 9:49:37 UTC

Is it possible to flash required fan settings directly in GPU own BIOS memory?


Maybe you can try this instead:
http://www.nvidia.com/object/nvidia_system_tools_6.02.html

It's got a fan speed setting and the tools are tied into the NVIDIA Control Panel and can be made to work at startup, but I'm not sure about your Windows version.

Works nice on older cards.
It's a legacy program, they stopped developing a while.


EDIT: here the survived link on binary from some mirror:
Unfortunately, binary not available for download. only description is available.
http://nvworld.ru/utilities/nvsystools/
____________

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3420
Credit: 46,535,924
RAC: 9,270
Russia
Message 1309184 - Posted: 23 Nov 2012, 9:21:12 UTC - in response to Message 1308366.


You should be able to use manual control with the slider. (Disable the auto mode)
This should stick, at least it did on my system.

I did not find any fan/OC related options in NV control panel of 263.06 driver. Did you mean NV own control utility or slider in RivaTuner app ?

____________

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3420
Credit: 46,535,924
RAC: 9,270
Russia
Message 1309191 - Posted: 23 Nov 2012, 9:41:11 UTC - in response to Message 1308362.

Thanks all a lot!
As I understood this problem worth some consideration indeed. Auto-logon would be not good cause it's not home PC indeed.
Is it possible to flash required fan settings directly in GPU own BIOS memory ?

It seems rather difficult... You will need a pre-made BIOS for the specific model/vendor with that FAN speed changed, or you will need to read the current BIOS and rebuild/hack it and then overrite it on the harware...



Just recalled some discussion about leaving OCed values in software and apply them on each boot or flashing them onto GPU. Maybe it was about ATi GPUs, not NV ones...
____________

Profile MikeProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 01
Posts: 24087
Credit: 33,100,173
RAC: 23,675
Germany
Message 1309194 - Posted: 23 Nov 2012, 9:46:35 UTC

I`m using MSI Afterburner.
Works on nvidia also.

____________

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3420
Credit: 46,535,924
RAC: 9,270
Russia
Message 1309196 - Posted: 23 Nov 2012, 9:50:28 UTC - in response to Message 1309194.
Last modified: 23 Nov 2012, 9:52:48 UTC

I`m using MSI Afterburner.
Works on nvidia also.

Thanks, Mike. Downloading...
EDIT: looks like this utility is successor of RivaTuner too.
____________

juan BFBProject donor
Volunteer tester
Avatar
Send message
Joined: 16 Mar 07
Posts: 5315
Credit: 294,842,760
RAC: 471,760
Brazil
Message 1309198 - Posted: 23 Nov 2012, 10:00:18 UTC - in response to Message 1309196.
Last modified: 23 Nov 2012, 10:00:37 UTC

I`m using MSI Afterburner.
Works on nvidia also.

Thanks, Mike. Downloading...
EDIT: looks like this utility is successor of RivaTuner too.

MSI Afterburner works as a service? With out need to log on any user?
____________

Profile S@NL - eFMer - efmer.com/boincProject donor
Volunteer tester
Avatar
Send message
Joined: 7 Jun 99
Posts: 512
Credit: 129,003,670
RAC: 35,820
United States
Message 1309209 - Posted: 23 Nov 2012, 10:48:57 UTC - in response to Message 1309198.

I`m using MSI Afterburner.
Works on nvidia also.

Thanks, Mike. Downloading...
EDIT: looks like this utility is successor of RivaTuner too.

MSI Afterburner works as a service? With out need to log on any user?

You need to log in.
GPU access is only allowed for a logged in user. So no service.
____________
TThrottle Control your temperatures. BoincTasks The best way to view BOINC. Anza Borrego Desert hiking.

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8502
Credit: 49,962,635
RAC: 50,948
United Kingdom
Message 1309217 - Posted: 23 Nov 2012, 11:25:33 UTC - in response to Message 1309209.

I`m using MSI Afterburner.
Works on nvidia also.

Thanks, Mike. Downloading...
EDIT: looks like this utility is successor of RivaTuner too.

MSI Afterburner works as a service? With out need to log on any user?

You need to log in.
GPU access is only allowed for a logged in user. So no service.

Windows Server 2003 (see opening post) is comparable to Windows XP - and Windows XP allows services to access the GPU, even when the service logs on using a different account than the foreground user.

1 · 2 · Next

Message boards : Number crunching : "Sleepy" NV GPU - how to "wake up" ?

Copyright © 2014 University of California