Can someone tell me why CUDA tasks take about 10 hours on this computer?

Message boards : Number crunching : Can someone tell me why CUDA tasks take about 10 hours on this computer?
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Deadmann

Send message
Joined: 30 Jun 99
Posts: 13
Credit: 19,410,104
RAC: 7
United States
Message 1438682 - Posted: 6 Nov 2013, 7:15:57 UTC

Can someone tell me why CUDA tasks take about 10 hours on this computer?

Computer ID: 4923505

I just set it to be visible a few min ago so hopefully people can see it.

It's an EVGA GT 620 which should take about 1 hour to do the CUDA tasks. (I had it in a different computer for a while and it was about that speed in it.) I have the 0.41 optimized binaries installed.

Thank you
ID: 1438682 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1438700 - Posted: 6 Nov 2013, 8:27:58 UTC - in response to Message 1438682.  


A quick look at a couple of the results that have validated show lots of restarts.
Others may be able to help sort out WU restart issues.
Grant
Darwin NT
ID: 1438700 · Report as offensive
BetelgeuseFive Project Donor
Volunteer tester

Send message
Joined: 6 Jul 99
Posts: 158
Credit: 17,117,787
RAC: 19
Netherlands
Message 1438753 - Posted: 6 Nov 2013, 13:34:30 UTC


You should check if the GPU is downclocking (use GPU-Z or some other tool to check the actual clock rate). I don't know if your GT-620 uses passive cooling, but this may not be sufficient when doing GPU crunching. Even when using active cooling you should check if it is not downclocking.

Tom

ID: 1438753 · Report as offensive
Profile Tim
Volunteer tester
Avatar

Send message
Joined: 19 May 99
Posts: 211
Credit: 278,575,259
RAC: 0
Greece
Message 1438759 - Posted: 6 Nov 2013, 13:51:22 UTC

Free one core of your CPU, if you use your CPU at 100%, and see if it is working.

Tim
ID: 1438759 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1438766 - Posted: 6 Nov 2013, 15:00:17 UTC - in response to Message 1438759.  

Free one core of your CPU, if you use your CPU at 100%, and see if it is working.

Tim

Shouldn't make any difference for a CUDA application - it's only the current OpenCL apps which need that.
ID: 1438766 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1438811 - Posted: 6 Nov 2013, 17:12:20 UTC

Try increasing thread priority - anything else running that might use GPU?

What about the setting for checkpointing?

clockrate is printed at restats and looks good, but 'memory clock' isn't and downclocks there affect speed to, so do check that one.
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1438811 · Report as offensive
Deadmann

Send message
Joined: 30 Jun 99
Posts: 13
Credit: 19,410,104
RAC: 7
United States
Message 1439593 - Posted: 7 Nov 2013, 9:09:11 UTC

The computer is my (84 year old) mothers. It's on one of those older cheap wooden computer desks with the slide out keyboard shelf. The shelf gets bumped just about every time someone walks in or out of the room... so the mouse moves... and the CUDA stops. Mom just uses it for emails and playing few different versions of solitaire. Maybe once a week looks as some web page.

I set up GPU-Z to log over night. I'll see if it down clocks anything.

What is the recommended version of the video drivers for CUDA to work best?

ID: 1439593 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1439594 - Posted: 7 Nov 2013, 9:20:21 UTC - in response to Message 1439593.  
Last modified: 7 Nov 2013, 9:29:00 UTC

What is the recommended version of the video drivers for CUDA to work best?


For most situations CUDA related, I generally recommend the latest WHQL. This is because Windows updates can and do update the driver subsystem (WDDM [, and later 'hybridised' XPDM]) on a 15 year plan, so corresponding driver updates *usually* ensure greater reliability & security.

There are special exceptions, most notably for new model GPUs, and the occasional genuine driver issue that slips through especially aroud major MS model changes & GPU generation, but those have been getting rarer.

With respect to possible performance differences using older drivers, it happens, though in most cases uptime/reliability trump the differences involved, again with special exceptions that don't apply for most ordinary general use PCs. Also as driver 'latency issues' are better understood, the Cuda applications do and will continue getting improved to mitigate those (having performance cake & eating it with reliability & security)
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1439594 · Report as offensive
Profile shizaru
Volunteer tester
Avatar

Send message
Joined: 14 Jun 04
Posts: 1130
Credit: 1,967,904
RAC: 0
Greece
Message 1439627 - Posted: 7 Nov 2013, 12:27:45 UTC - in response to Message 1439593.  

The shelf gets bumped just about every time someone walks in or out of the room... so the mouse moves... and the CUDA stops.


Then the good news is it's likely a settings issue rather than something more serious.

Personally, I would go to Boinc Manager and select 'Use GPU always' (may be somewhere else on different versions of Boinc, I dunno). That will force your GPU to (obviously) always run, while overriding any web-based prefs. Just make sure to check Solitaire before & after so there's no lag introduced:) Seriously.

If this suggestion is not an option then go to web-based preferences (account -> computing prefs -> 'In use' means mouse/keyboard activity in last...) and set 'in use' to a minute or two. You've likely got it set to an hour or something...

Hope this helps.
ID: 1439627 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1439636 - Posted: 7 Nov 2013, 12:56:06 UTC

If there is no GPU intensive stuff (games(*), videos, graphical applications) then you can set 'suspend GPU when in use' to no.
You can also set up exclusions for programs.

(*)ego shooters and other graphics heavy games. solitaire doesn't count ;)
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1439636 · Report as offensive
Profile shizaru
Volunteer tester
Avatar

Send message
Joined: 14 Jun 04
Posts: 1130
Credit: 1,967,904
RAC: 0
Greece
Message 1439651 - Posted: 7 Nov 2013, 13:46:08 UTC - in response to Message 1439627.  

Too late to edit...

Personally, I would go to Boinc Manager and select 'Use GPU always'


That should be "...Personally, I would go to Boinc Manager Activity Menu and select 'Use GPU always'..."
ID: 1439651 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1439656 - Posted: 7 Nov 2013, 14:04:59 UTC - in response to Message 1439636.  

If there is no GPU intensive stuff (games(*), videos, graphical applications) then you can set 'suspend GPU when in use' to no.
You can also set up exclusions for programs.

(*)ego shooters and other graphics heavy games. solitaire doesn't count ;)

Solitaire is a good enough test for 61-year old me. I doubt Deadmann's mother needs a sterner test than that.
ID: 1439656 · Report as offensive
Profile skildude
Avatar

Send message
Joined: 4 Oct 00
Posts: 9541
Credit: 50,759,529
RAC: 60
Yemen
Message 1439732 - Posted: 7 Nov 2013, 17:13:34 UTC

lets not forget the 620 is an entry level GPU and isn't all that powerful


In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope
ID: 1439732 · Report as offensive
Ulrich Metzner
Volunteer tester
Avatar

Send message
Joined: 3 Jul 02
Posts: 1256
Credit: 13,565,513
RAC: 13
Germany
Message 1439750 - Posted: 7 Nov 2013, 17:31:13 UTC - in response to Message 1439732.  

lets not forget the 620 is an entry level GPU and isn't all that powerful

Well, i have GT 640 PCIe and GT 430 PCI(!!) crunching along - both passively cooled, and considered entry level.

The GT 640 takes ~20-40 minutes for MB and ~2-3 hours for AP, depending on % blanked.
The GT 430 takes ~45-90 minutes for MB and ~5-6.5 hours for AP.

So 10 hours for a GT 620 seems a little too long - at least for me.
Aloha, Uli

ID: 1439750 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1439761 - Posted: 7 Nov 2013, 17:40:51 UTC

That host shows an APR of 41 for v6 - something is wrong if it is that slow with 'the next generation' even with extra autocorrelation, no way it should be taking 10h.

however each time it gets interrupted it has to restart from checkpoint (would LAIM help there? or dioes GPU always get unloaded?) if checkpoints are too far aprt (other settings) you could keep losing a lot of crunching time.

does the host run 24/7? at least at night there shouldn't be interuptions from mouse movement?
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1439761 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1439773 - Posted: 7 Nov 2013, 17:45:00 UTC - in response to Message 1439761.  

That host shows an APR of 41 for v6 - something is wrong if it is that slow with 'the next generation' even with extra autocorrelation, no way it should be taking 10h.

however each time it gets interrupted it has to restart from checkpoint (would LAIM help there? or dioes GPU always get unloaded?) if checkpoints are too far aprt (other settings) you could keep losing a lot of crunching time.

does the host run 24/7? at least at night there shouldn't be interuptions from mouse movement?

Depends if his mother has a cat...
ID: 1439773 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1439792 - Posted: 7 Nov 2013, 18:03:34 UTC

OK, that was just me being silly. Let's do this properly.

First, a link to host 4923505 so we can find it quickly.

BOINC v7.0.64, and - we know - running according to preferences (else the mouse move wouldn't interfere).

That was the version of BOINC which applied CPU throttling to GPU apps (silly David). It would be wise to check

Use at most
Can be used to reduce CPU heat 100% of CPU time

in the Computing preferences section of your account.
ID: 1439792 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1439795 - Posted: 7 Nov 2013, 18:07:07 UTC - in response to Message 1439761.  

(would LAIM help there? or dioes GPU always get unloaded?)

LAIM doesn't help, GPU tasks always get unloaded.
ID: 1439795 · Report as offensive
Juha
Volunteer tester

Send message
Joined: 7 Mar 04
Posts: 388
Credit: 1,857,738
RAC: 0
Finland
Message 1439852 - Posted: 7 Nov 2013, 19:08:53 UTC - in response to Message 1439761.  

does the host run 24/7? at least at night there shouldn't be interuptions from mouse movement?

My mouse moves on its own...
ID: 1439852 · Report as offensive
Deadmann

Send message
Joined: 30 Jun 99
Posts: 13
Credit: 19,410,104
RAC: 7
United States
Message 1440189 - Posted: 8 Nov 2013, 11:28:52 UTC

I've just been looking at a GPU-Z log. (I tried importing it into Excel and making a graph but ended up with junk... but that's not important.) It looks like both the Core and Mem clocks never went up when when Seti started. The amount of memory in use increased 244MB so I know when Seti was running. GPU load also went to 99%. The clocks did go up at other times during the day so I know that they do go up sometimes.

Core stayed at 50MHz
Mem stayed at 67MHz

In the daytime tomorrow I do a windows update on it... then I'll update to the 331.65 driver that I downloaded about a week ago. See if that fixes it.

I've had this video board GT 620 in my other computer http://setiathome.berkeley.edu/show_host_detail.php?hostid=2249613 peach-pie last January when the fan on my GT 240 croaked. I think the GT620 took about 3 times longer than the 240 for most work units. So the shorties were about 45min and the others correspondingly longer.

The computer does run 24/7. Mom almost always sleeps in a recliner in that room because of arthritis. She almost always gets up after 5-6 hours to use the bathroom and probably bumps the keyboard try then. But they should have 5-6 hours every day.

I did try running the GPU always and it does bog the version of solitaire she plays pretty badly. I think it's some version that has about 100 different games that are suppose to help keep an old person's mind sharp. (looks like the website for the company no longer exists.) Seems to be working.

I post the results of the updates here.
ID: 1440189 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : Can someone tell me why CUDA tasks take about 10 hours on this computer?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.