Uneven usage of GPUs

Hi all, apologies if this question has been raised previously but a quick search did not yield any results.

I seem to have some issues with one of my rigs that is running two GTX 480s. Prior to adding the second card a couple of weeks back, the existing 480 used to show the progress of around 0.1 - 0.5% every second (average max of 10 min per WU). However, after I added the second card, the progress indicator for this card (device 0) seems to move on at a snails pace taking up to 30 min per WU, while the newer card (device 1) is crunching much faster. First card is on an x16 PIC-E slot while the second is on an x8. I have observed this on many WUs and hence don't think it is an isolated case.

GPU-Z indicates both to be loaded at around 98%. I have also checked the task list for this rig and it shows 153 tasks under validation inconclusive, most of which I have discovered are due to wing mates using 560 ti with stock application. Do I need to tweak any settings to get both the cards crunch evenly? I do have another rig with two GTX 260 that is performing well without any fiddling, so I am confused as to what went wrong with this one.

As per GPU-Z, both the cards are running the same shader clock speed of 1401 MHz, and both are 98-99% loaded. Driver version is 285.58. I will try swapping the cards when I get back from work.

Update: I have now swapped the GPUs and the issue persists. I have now also observed that both the cards exhibit this behavior, which is erratic in nature. For some time, the crunching seems fine and then for no apparent reason, one of them goes in to slow mode. Still cant figure out what is causing this issue. Will keep investigating...

how big of a power supply are you using?
____________
Proud member of TSWB.

End terrorism by building a school

The reason I ask about the PSU is that it seems there may not be enough power being given out so that both GPU's can work at their optimum
____________
Proud member of TSWB.

End terrorism by building a school

and what about some "normal" long running tasks like http://setiathome.berkeley.edu/result.php?resultid=2660698769

this is one on my machine, runtime also half an hour, Angle Rate 0.274226 from a tape beginning with 22no10ab. But with this AR, the runtime is pretty normal.
____________

The rig is powered by a 1200W Gigabyte Odin, which I am guessing should be more than enough for the two 480s. The motherboard is Gigabyte GA-MA790X-UD4P.

I have also noticed that the system as a whole is sluggish in responding to any commands - be it a right click menu or opening and closing folders. I have killed all unnecessary background programs and even changed the antivirus from Avast to AVG (both free versions) as I have noticed the aggressive behavior of Avast. And the windows itself was reinstalled last month. Things go back to normal if I run only one card in either slots.

Few of the tasks are now taking up to 90 minutes on that card.
I am not very sure about allocating the CPU cores to the GPU with the Swan_sync command. Would anyone be able to guide me through?

I will have to check out the BIOS setting later after getting back from work.

GPU-Z indicates both to be loaded at around 98%. I have also checked the task list for this rig and it shows 153 tasks under validation inconclusive, most of which I have discovered are due to wing mates using 560 ti with stock application. Do I need to tweak any settings to get both the cards crunch evenly? I do have another rig with two GTX 260 that is performing well without any fiddling, so I am confused as to what went wrong with this one.

Thanks for that indepth analysis tbret. What I am observing is that while one card is churning out the WUs at an average of 10-15 min, the other card is usually taking about 30 and sometimes even 90 min to finish one. Both the cards perform well when alone no matter which slot is used, but the moment they are put in together, things slow down.

Here is a workunit that took 4,274.27 seconds to finish, and here is another one that took 3,203.11 seconds.

Both the cards perform well when alone no matter which slot is used, but the moment they are put in together, things slow down.

Thanks for that indepth analysis tbret. What I am observing is that while one card is churning out the WUs at an average of 10-15 min, the other card is usually taking about 30 and sometimes even 90 min to finish one. Both the cards perform well when alone no matter which slot is used, but the moment they are put in together, things slow down.

Here is a workunit that took 4,274.27 seconds to finish, and here is another one that took 3,203.11 seconds.

1) What sized PSU are you using? (either one is fast, but one is slow with two? may be power, but unless you have a multi-rail PSU and you're starving a card, I can't guess why it would be both, but not either, that gives you trouble)

2) Did you do a Custom/CLEAN driver reinstall with both cards in the computer? (go to 301 or 306)

3) What's the temperature of the cards? (may be the heat of both)

4) Either runs fast no matter which slot, just so long as it is only one?

This is a weird one.

OH, has that computer ever had an ATI driver on it? If so, you may need to use DriverSweeper to get rid of any remaining "pieces".

Several of us have sort-of had what you are talking about happen to us. I've had to reinstall drivers on occasion (clean). I had to get rid of MSI Afterburner (uninstall) one time and that cleared it up (don't ask me why, could have been coincidence).

What else, if anything, is running?

By the way, GPU-Z does not always show a downclock after a driver crash, even though the card is downclocked.

3) What's the temperature of the cards? (may be the heat of both)

EVGA Precision reports the temperatures at 47 deg C for card 1 and 52 deg C for card 2. Card 1, I am guessing is the inner one, which is slow.

What is the ambient temperature?
My GTX 560Ti & GTX 460 both run at over 70° with the fans running at almost full speed, but the ambient temperature is mid 30°c.
When the temperature drops below 30°c, they run at about 70°, but with the fans only running at about 70% of maximum possible speed.

I used to get driver crashes on this rig earlier which was fixed by a reinstall. And even if it is happening now, wouldnt it affect the performance of both the cards? Or would it make just one of the cards slow down?

Thanks for the tips tbret, and you never know what the culprit is. I shall try all that once I get back and see if there is any improvement.

And I am using Precision X 3.0.3 and both the cards are synced.

I just pulled up a screen cap from this rig, and the difference in crunch time is clearly evident. New observation is that the progress indicator for the GPU in question just stops, as if paused, for a while before picking up again.

You still didn't follow the simplest advice by Mike:
http://setiathome.berkeley.edu/forum_thread.php?id=69788&postid=1298241#1298241

You don't know how to "free at least one CPU core"?

The setting is:
- if you use web preferences:
http://setiathome.berkeley.edu/prefs.php?subset=global
"On multiprocessors, use at most 100% of the processors"

- if you use local preferences (do the change in BOINC Manager if you already use local preferences):
http://boinc.berkeley.edu/wiki/Local_preferences
"On multiprocessor systems, use at most [ 100.00 ] % of the processors"


To see will this have any effect - change to 50%
(this will free 3 cores on "AMD Phenom(tm) II X6 1055T Processor" (meaning that only 3 (instead of 6) CPU tasks will be started/run by BOINC))

If you see 'effect' - next try 99% (this will free 1 core on any CPU with up to 100 cores)
If you see the same 'effect' as with 3 cores free - leave it at 99%

If the 'effect' is less - next try 2 cores free:
% = 100 * (AllCores - FreeCores) / AllCores

% = 100 * (6 - 1) / 6 = 84% (always round UP)
% = 100 * (6 - 2) / 6 = 67%

So:
- anything 67...83% will free 2 cores on a six-core Processor (six-thread Processor in case of Intel)
- anything 50...66% will free 3 cores on a six-core Processor


____________



- ALF - "Find out what you don't do well ..... then don't do it!" :)