Titan V core clocks in compute mode

Message boards : Number crunching : Titan V core clocks in compute mode
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 1718
Credit: 700,768,335
RAC: 2,412,086
United States
Message 1986932 - Posted: 24 Mar 2019, 18:14:23 UTC

so this is in reference to this older thread, looks like it got locked due to inactivity.

https://setiathome.berkeley.edu/forum_thread.php?id=83613&postid=1968047#1968047
For the Titan, it has 3 power states, P0,P1,P2. I set the dropdown to Max Performance. Watching the course of a work unit, it mostly hovers on P1, with an occasional switch to P2.

     GPU                             Memory
P0   135 Min - 135 Max          1700min-1700Max
P1   135 Min - 1335 Max         1700min-1700Max
P2   135 Min - 1912 Max         1700min-1700Max


I was thinking about this randomly, and came across this post: Titan V slower than 1080ti tensorflow:18.08-py3 and 396.54 drivers

according to TomK@Nvidia, you CAN unlock the TitanV core clock speed as of Linux driver 415.25 (417.35 on windows). but you need to run an additional command in the terminal to get it.

nvidia-smi --cuda-clocks=OVERRIDE


one user in the thread says it works. maybe Petri, or Tod, or anyone else with a Titan V can try it out.

Might be worth trying on Pascal or RTX cards also, but i can't say for sure if it will work, it might have only been added for Titans.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1986932 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9648
Credit: 887,713,878
RAC: 1,715,418
United States
Message 1986934 - Posted: 24 Mar 2019, 18:46:10 UTC

I just tried it on my RTX 2080.

nvidia-smi -i 3 --cuda-clocks=OVERRIDE
Overriding CUDA clock limit is not supported for this device.
Treating as warning and moving on.
All done.
Must be only for Titan V
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1986934 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 1718
Credit: 700,768,335
RAC: 2,412,086
United States
Message 1986939 - Posted: 24 Mar 2019, 19:48:53 UTC - in response to Message 1986934.  

I assume you tried one of your systems with the 415 driver and not the ones with 410, correct?

Just crossing the i’s and dotting the t’s ;)
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1986939 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9648
Credit: 887,713,878
RAC: 1,715,418
United States
Message 1986945 - Posted: 24 Mar 2019, 20:20:49 UTC - in response to Message 1986939.  

Yes, I tried it first on the daily driver with 410.104. It just threw up an error and said command not recognized and to refer to help. Then I went to the custom loop rig with 415.27 drivers and tried it and got the printout in my previous post. Recognized the command but said the card GPU3 (RTX 2080) was not able to use it.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1986945 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 1718
Credit: 700,768,335
RAC: 2,412,086
United States
Message 1986961 - Posted: 24 Mar 2019, 23:37:05 UTC - in response to Message 1986945.  

Ah. Too bad. It was worth a shot lol.

Well I hope the info helps someone with a Titan V :)
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1986961 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9648
Credit: 887,713,878
RAC: 1,715,418
United States
Message 1986962 - Posted: 24 Mar 2019, 23:44:41 UTC

Yes, there are a few of them floating about. Would like to hear from a Titan V owner what the clock override achieves.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1986962 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 3452
Credit: 196,085,972
RAC: 521,755
United States
Message 1986986 - Posted: 25 Mar 2019, 3:02:20 UTC

I found at least 5 Titan systems in the top 100 on the Leaderboard. Shot them all PM with this thread.
Since I am not sure if it only applies to Titan V or not, if it was a Titan I sent them a message.

Tom
I will stop procrastinating tomorrow.
\\// Live Long & Prosper (starting tomorrow ;)
ID: 1986986 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9648
Credit: 887,713,878
RAC: 1,715,418
United States
Message 1986991 - Posted: 25 Mar 2019, 3:24:56 UTC

You'd have to look at each of their stderr.txt outputs to determine. It does printout the Titan full name. For example Petri's output.

setiathome_CUDA: CUDA Device 1 specified, checking...
Device 1: TITAN V is okay
SETI@home using CUDA accelerated device TITAN V
Unroll autotune 80. Overriding Pulse find periods per launch. Parameter -pfp set to 80
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1986991 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1654
Credit: 534,888,342
RAC: 472,291
Finland
Message 1987086 - Posted: 25 Mar 2019, 18:15:45 UTC

root@Linux1:~/KWSN-Bench-Linux-MBv7_v2.01.08# nvidia-smi -i 1 --cuda-clocks=OVERRIDE
All done.


No error!!!!
I'll have to test more to find out if I can get some more speed out of the card.

Petri
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1987086 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9648
Credit: 887,713,878
RAC: 1,715,418
United States
Message 1987088 - Posted: 25 Mar 2019, 18:21:36 UTC - in response to Message 1987086.  

Thanks for the notice, Petri. Let us know what you can do with the overclock.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1987088 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 1718
Credit: 700,768,335
RAC: 2,412,086
United States
Message 1987091 - Posted: 25 Mar 2019, 18:34:16 UTC - in response to Message 1987086.  

root@Linux1:~/KWSN-Bench-Linux-MBv7_v2.01.08# nvidia-smi -i 1 --cuda-clocks=OVERRIDE
All done.


No error!!!!
I'll have to test more to find out if I can get some more speed out of the card.

Petri


awesome. you should see a good speed boost with the core clock raising from 1335 to 1900+MHz

it will undoubtedly use more power (closer to the 300W limit) and generate more heat though. be prepared for that.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1987091 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1654
Credit: 534,888,342
RAC: 472,291
Finland
Message 1987096 - Posted: 25 Mar 2019, 18:44:37 UTC

Yeah,

it runs now on pstate P0 instead of the lesser P2 that it used to run.

Performance level 2 in nvidia-settings is the pstate P0 that nvidia-smi reports . And yes, the power intake jumped from 140-160W to 324W and the the machine did an immediate shutdown to protect the power supply.

Hopefully some WUs got reported. Unfortunately I run at the same time a new development version and that seems to produce more inconclusives at the moment. And the the database seems to lag behind.

The nvidia-smi reported GPU temperature to be 72C at P0 while it used to be under 60C in P2.

I'm going back to normal mode now.

I'll have to first underclock or something and then go up step by step. I'll run those tests off-line next weekend or something.
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1987096 · Report as offensive
Tod Project Donor

Send message
Joined: 17 Apr 99
Posts: 27
Credit: 143,582,371
RAC: 26,882
United States
Message 1987421 - Posted: 27 Mar 2019, 13:07:52 UTC - in response to Message 1987096.  

I've had to put my Titan V to work in another machine. GPU Pass-through is exceedingly difficult if you have identical GPUs. My Seti machine now is currently running 4 2080TIs. I hope to put the Titan V back in the Seti rig in the next month. I'd love to try this override.

On a similar note, has anyone tried and Titan RTX? I was thinking about picking up a couple of these. I know the Tensor cores aren't being used, but the specs to look like a sizeable upgrade from the 2080TI
ID: 1987421 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 1718
Credit: 700,768,335
RAC: 2,412,086
United States
Message 1987433 - Posted: 27 Mar 2019, 14:37:13 UTC - in response to Message 1987421.  

I've had to put my Titan V to work in another machine. GPU Pass-through is exceedingly difficult if you have identical GPUs. My Seti machine now is currently running 4 2080TIs. I hope to put the Titan V back in the Seti rig in the next month. I'd love to try this override.

On a similar note, has anyone tried and Titan RTX? I was thinking about picking up a couple of these. I know the Tensor cores aren't being used, but the specs to look like a sizeable upgrade from the 2080TI


are you running your system as a VM? what hypervisor are you using, and what are the specs of the whole host system?

I'm not sure if anyone has tried the Titan RTX. i think at the price of it, just pick up a titan V instead as i think it'll be faster (actually, at that price, just buy 2x 2080tis lol). i didn't get the impression that the Titan RTX would be that much faster than a 2080ti. it only has 4 more SMs, and the more than double the memory I don't think will change performance too much.

from what i've seen in the past and so far, the Titan V even gimped to 1335MHz was still just as fast or faster than a full speed RTX 2080ti. so I'd expect the unleashed Titan V to do even better still.

quick specs for reference:
Titan V - 80 SMs, 5120 CUDA cores, 12GB HBM2 (652 GB/s)
Titan RTX - 72 SMs, 4608 CUDA cores, 24GB GDDR6 (672 GB/s)
RTX 2080ti - 68 SMs, 4352 CUDA cores, 11GB GDDR6 (616 GB/s)

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1987433 · Report as offensive
Tod Project Donor

Send message
Joined: 17 Apr 99
Posts: 27
Credit: 143,582,371
RAC: 26,882
United States
Message 1987463 - Posted: 27 Mar 2019, 18:00:12 UTC - in response to Message 1987433.  

No, my Seti rig is stand alone and not in a VM. The rig I use for development is isolated from seti numbercrunching. This rig does use GPU passthrough for various VMs.

I might have to pull out that Titan and put it in my seti rig just for some testing. I am kinda curious now too. I know I have plenty of power (1600 watt PSU), but the card is air cooled. Likely will have to start out underclocking it a bit, then slowly ramp up.

I will be sure to keep you informed!
ID: 1987463 · Report as offensive
Oddbjornik Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 151
Credit: 255,250,430
RAC: 299,190
Norway
Message 1987466 - Posted: 27 Mar 2019, 18:16:24 UTC - in response to Message 1987463.  

Likely will have to start out underclocking it a bit, then slowly ramp up.
There's always the nvidia-smi --power-limit command.
ID: 1987466 · Report as offensive
Tod Project Donor

Send message
Joined: 17 Apr 99
Posts: 27
Credit: 143,582,371
RAC: 26,882
United States
Message 1987470 - Posted: 27 Mar 2019, 19:21:24 UTC

I put the Titan V in the seti rig and used the cuda-clocks override switch

I can confirm, as Petri33 indicated, the card indicates its running in P0 now. (or level 2 on the x-server nvidia gui). the clock fluctuates between (1600-1850). The power usage has jumped a lot, but the card has a TDP of 250watts. Which is where it stays mostly now. Pretty much the same power usage that the 2080Tis have. The Titan used to run with much less (around 150-170 watts)

I honestly don't know if there has been a marked decrease in processing time yet. Eyeballing the seti interface, it appears that most WUs on the titan are around 30 seconds, and the 2080Ti are around 39-40 seconds.

I haven't done any turning of the seti app itself, other than run the Cuda-10 verison of the app.

If there is something you'd like me to try / tune /test, let me know :-)
ID: 1987470 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9648
Credit: 887,713,878
RAC: 1,715,418
United States
Message 1987471 - Posted: 27 Mar 2019, 19:29:31 UTC - in response to Message 1987470.  
Last modified: 27 Mar 2019, 19:31:48 UTC

Not much tuning available with the CUDA10 app. Looks like you are running the -nobs parameter and that is all that is available.

[Edit]Just thought of something. Can you use power level to restrict the power and still keep the card in P0? IOW, restrict the power usage and just clip the top end of the clocks. I think theoretically that is possible.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1987471 · Report as offensive
Tod Project Donor

Send message
Joined: 17 Apr 99
Posts: 27
Credit: 143,582,371
RAC: 26,882
United States
Message 1987474 - Posted: 27 Mar 2019, 20:00:08 UTC

Yea, I can. Seems to stay at p0 regardless of where the power is. I will leave the power cap at 250 for a day or so, then down to 200 and see what kind of decrease I get.
ID: 1987474 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 1718
Credit: 700,768,335
RAC: 2,412,086
United States
Message 1987481 - Posted: 27 Mar 2019, 20:26:33 UTC - in response to Message 1987474.  

Yea, I can. Seems to stay at p0 regardless of where the power is. I will leave the power cap at 250 for a day or so, then down to 200 and see what kind of decrease I get.


remember, you can still overclock the card even with the power capped. this way you can make up most of the lost ground from the power cap.

i do this on my 1080tis. stock is 250w, but power capped to 200w. i've then applied overclocks to get as close to the clocks i could previously get at 250w. the card will just start pulling back on the core voltages a bit to keep the power use in check for what you set it at. i get nearly the same performance with less power use (and temps) this way.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1987481 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : Titan V core clocks in compute mode


 
©2019 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.