Only wrong -9 overflow results with GTX560/Ti?

Message boards : Number crunching : Only wrong -9 overflow results with GTX560/Ti?
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 1162111 - Posted: 14 Oct 2011, 6:15:43 UTC

I looked through the results of my machine and found a few 'Completed, confirmation is not clear' (or something like this) results, because of wingmen..

This three machines make only CUDA errors:

Two machines running stock apps:
http://setiathome.berkeley.edu/show_host_detail.php?hostid=6137511
http://setiathome.berkeley.edu/show_host_detail.php?hostid=6178234

One with opti apps:
http://setiathome.berkeley.edu/show_host_detail.php?hostid=6144916

They have all GTX560(Ti) GPU and an i7-2600(K) CPU.

Normally I would say it's because of the GTX560s because it's well known that some make errors.. - but also with stock CUDA app?

Or it's because of the (new) i7-2600(K) CPU, because the CUDA WU can't be prepared well on CPU before it will send/calculated to/on GPU?


- Best regards! - Sutaru Tsureku, team seti.international founder. - Optimize your PC for higher RAC. - SETI@home needs your help. -
ID: 1162111 · Report as offensive
LadyL
Volunteer tester
Avatar

Send message
Joined: 14 Sep 11
Posts: 1679
Credit: 5,230,097
RAC: 0
Message 1162138 - Posted: 14 Oct 2011, 9:25:13 UTC

the english term is 'validation inconclusive'

it would help to link the tasks.

host 6144916 is having general GPU trouble. might need a reboot, be too hot or too much (factory) overclocking. It's also running x32f.

both stock hosts also look like they are only spitting out -9 on GPU - usually a sign something has gone haywire.
Rebooting tends to clear those errors, by emptying GPU RAM.
Temperatures and in the case of 560s (factory) overclocking can also be an issue.

It's unlikely to be linked to the CPU type - rather because people go for 'newest, shinyesy, best' on both CPU and GPU.





ID: 1162138 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34258
Credit: 79,922,639
RAC: 80
Germany
Message 1162140 - Posted: 14 Oct 2011, 9:29:28 UTC

Dont forget the risk of dust bunnies.



With each crime and every kindness we birth our future.
ID: 1162140 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1162141 - Posted: 14 Oct 2011, 9:30:42 UTC

Don't forget x38g.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1162141 · Report as offensive
Profile Fred J. Verster
Volunteer tester
Avatar

Send message
Joined: 21 Apr 04
Posts: 3252
Credit: 31,903,643
RAC: 0
Netherlands
Message 1162221 - Posted: 14 Oct 2011, 15:43:15 UTC - in response to Message 1162141.  
Last modified: 14 Oct 2011, 16:29:30 UTC

---[Snipped]---



Or it's because of the (new) i7-2600(K) CPU, because the CUDA WU can't be prepared well on CPU before it will send/calculated to/on GPU?


Don't forget x38g.


Good point and easily overlooked.
A lot of -9 (Informational Overflow), -12 and -177 type errors.

setiathome_CUDA: CUDA Device 1 specified, checking...
Device 1: GeForce GTX 560 Ti is okay
SETI@home using CUDA accelerated device GeForce GTX 560 Ti
setiathome_enhanced 6.09 Visual Studio/Microsoft C++
libboinc: 6.3.22

Work Unit Info:
...............
WU true angle range is : 6.254450
Optimal function choices:
-----------------------------------------------------
name
-----------------------------------------------------
v_BaseLineSmooth (no other)
v_GetPowerSpectrum 0.00017 0.00000
v_ChirpData 0.00985 0.00000
v_Transpose4 0.00252 0.00000
FPU opt folding 0.00128 0.00000
SETI@home error -12 Unknown error
cudaAcc_find_triplets erroneously found a triplet twice in find_triplets_kernel
File: d:/Projects/SETI/seti_boinc/client/cuda/cudaAcc_pulsefind.cu
Line: 236

</stderr_txt>
]]>


Inconclusives are almost all -9 errors, 30 spikes, or 30 signals in total.
Started 27th september, then 7, 8 , 10, 11, 12 and 13 october, all -9 errors.
(Those -12 errors were all too familiar and the reason I swapped out an 9800GTX,
for an GTS250).
Most errors I've seen, came from the 'stock - CUDA -app' !?
And why should the I7-2600 not be able to do it's job as CPU?
(K stands for a 3000 instead of an 2000 Graphics Engine, also Unlocked Multiplier
I7-2600 v.s. I7-2600K
I7-2600K.
SANDY BRIDGE and more.
Another......
ID: 1162221 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 1162317 - Posted: 14 Oct 2011, 21:11:01 UTC
Last modified: 14 Oct 2011, 21:19:07 UTC

It's well known that the S@h CUDA app don't run well on some GTX560/Ti's (wrong -9 overflow results).
I thought only with the opti CUDA app.
But it looks also with stock 6.10_cuda_fermi app.


I looked to some results, and sometimes it's correct, wingman confirmed with CPU app the -9 overflow.


But - what happen if two GTX560/Ti are wingmen of one WU?
Both end with wrong -9 overflow result and the WU will be deleted?
Very bad for the science.


-9 overflow can be a well result, if correct calculated.
But can also be a sign for bad hardware.
Maybe the project could/should find a way to detect/differ both?


Someone could find a GTX560 or GTX560Ti on which the stock 6.10_cuda_fermi app run fine?
If not, maybe it would be better not to send out WUs to this GPU series.
Wasted internet bandwidth of S@h and unnecessary electricity bill for owner.


Maybe the new S@h v7 CUDA app (which will come maybe in January 2012) run well on GTX560/Ti's?


- Best regards! - Sutaru Tsureku, team seti.international founder. - Optimize your PC for higher RAC. - SETI@home needs your help. -
ID: 1162317 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 1162437 - Posted: 15 Oct 2011, 2:27:00 UTC
Last modified: 15 Oct 2011, 2:34:37 UTC

I found very quickly two more GTX560Ti machines, only CUDA errors (wrong -9 overflows):

6.10_cuda_fermi:
http://setiathome.berkeley.edu/show_host_detail.php?hostid=5949570

x32f:
http://setiathome.berkeley.edu/show_host_detail.php?hostid=4335051


To now I found no GTX560/Ti which run well with (stock or opti) CUDA app..


The machines download WUs for nothing..
And we are confused about the continuously maxed out internet connection of S@h? ;-)
But - why get this machines (so much) new WUs, I thought the max WUs/day should/would be 1 after only errors..
Ahh no.. a wrong -9 overflow result don't count immediately as error, only after comparison with wingman..

Hmm..


[EDIT of the title of this thread from:
Errors because of GTX560(Ti) or i7-2600(K)?
to:
Only wrong -9 overflow results with GTX560/Ti?]


- Best regards! - Sutaru Tsureku, team seti.international founder. - Optimize your PC for higher RAC. - SETI@home needs your help. -
ID: 1162437 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1162464 - Posted: 15 Oct 2011, 4:08:19 UTC - in response to Message 1162437.  

...
To now I found no GTX560/Ti which run well with (stock or opti) CUDA app..
...

See the Valid tasks for computers 5728027, 5940661, and 5853873.
                                                                   Joe
ID: 1162464 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1162466 - Posted: 15 Oct 2011, 4:11:52 UTC - in response to Message 1162464.  
Last modified: 15 Oct 2011, 4:14:24 UTC

and 5853873.


Russ Sheldon has 2, one was jittery the other not. Simply reseating the cards apparently brought them both good.

[Edit:] make that 3 in the same machine...

Jason
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1162466 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1162486 - Posted: 15 Oct 2011, 4:33:19 UTC - in response to Message 1162437.  

I found very quickly two more GTX560Ti machines, only CUDA errors (wrong -9 overflows):

6.10_cuda_fermi:
http://setiathome.berkeley.edu/show_host_detail.php?hostid=5949570

x32f:
http://setiathome.berkeley.edu/show_host_detail.php?hostid=4335051


To now I found no GTX560/Ti which run well with (stock or opti) CUDA app..


The machines download WUs for nothing..
And we are confused about the continuously maxed out internet connection of S@h? ;-)
But - why get this machines (so much) new WUs, I thought the max WUs/day should/would be 1 after only errors..
Ahh no.. a wrong -9 overflow result don't count immediately as error, only after comparison with wingman..

Hmm..


[EDIT of the title of this thread from:
Errors because of GTX560(Ti) or i7-2600(K)?
to:
Only wrong -9 overflow results with GTX560/Ti?]


- Best regards! - Sutaru Tsureku, team seti.international founder. - Optimize your PC for higher RAC. - SETI@home needs your help. -



There have been a lot of comments about 560Ti problems.

I can't speak for everyone, but I had a long string of bad work units on mine early-on. A reboot completely cured the problem and it hasn't happened since.

Every once in a while I do get a work unit that "overflows out," but that isn't exclusive to the 560Ti computer or GPU applications.

Four things:

1) You have to have a big enough PSU with a 560Ti.
2) You have to keep them cool (mine's never been a problem).
3) They don't seem to like aggressive overclocking.
4) Sometimes you get a string of noisy work units.

For the thousands and thousands of WUs which represent 1.3 million Credits on my 560Ti computer, other than the one early string of bad results which I noted above - fixed with a reboot (driver hang) - I haven't had any problems through two motherboards and two processors with multiple RAM configurations with two or three different driver versions.

But I have seen others complain that they were getting a lot of errors. Most of those were people overclocking the card.

Mine's a GV-N560SO-1GI-950

It's quiet and doesn't get hot.

The last video card I had that returned a high quantity of trash was a PNY 9800 GT that started overheating.
ID: 1162486 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1162501 - Posted: 15 Oct 2011, 4:47:25 UTC - in response to Message 1162437.  

To now I found no GTX560/Ti which run well with (stock or opti) CUDA app..

My 560Ti (which is factory overclocked) runs with no problems at all (other than trying to get enough work to keep it busy).


Grant
Darwin NT
ID: 1162501 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1162510 - Posted: 15 Oct 2011, 5:01:25 UTC - in response to Message 1162501.  

To now I found no GTX560/Ti which run well with (stock or opti) CUDA app..

My 560Ti (which is factory overclocked) runs with no problems at all (other than trying to get enough work to keep it busy).



I remember Slavac had problems early-on and just went to check his machines. Yep, sure enough...hundreds of errors.

Dan's is fine. Mine's fine.

I PMed Slavac. Hope he checks his account before he turns-in.
ID: 1162510 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1162538 - Posted: 15 Oct 2011, 5:34:17 UTC - in response to Message 1162510.  
Last modified: 15 Oct 2011, 5:35:20 UTC

I remember Slavac had problems early-on and just went to check his machines. Yep, sure enough...hundreds of errors.


Slavac's has been under my microscope for a while. Best we've been able to trace so far is some fundamental system problem with BIOS/chipset or similar, that needs attention he'll have to make time for. So far this example has been an exception, rather than the rule for 560ti's, but nonetheless a good example of *something* being not quite right.

Even my own 560ti, with factory overclock, has been chugging along like a trooper with no noticeable issues. At 2/3 of the throughput of my 480, with far lower power & cooling issues, I've got no complaints whatsoever.

Jason
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1162538 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1162554 - Posted: 15 Oct 2011, 6:16:37 UTC - in response to Message 1162538.  

Slavac's has been under my microscope for a while. Best we've been able to trace so far is some fundamental system problem with BIOS/chipset or similar, that needs attention he'll have to make time for. So far this example has been an exception, rather than the rule for 560ti's, but nonetheless a good example of *something* being not quite right.

Does he have power to the Auxillary power sockets? Trying to run full load with just PCIe power or PCIe & one Aux power lead connected will result in tears. Or a PSU with less than honest ratings.
Grant
Darwin NT
ID: 1162554 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1162556 - Posted: 15 Oct 2011, 6:30:00 UTC - in response to Message 1162554.  
Last modified: 15 Oct 2011, 6:35:07 UTC

Does he have power to the Auxillary power sockets? Trying to run full load with just PCIe power or PCIe & one Aux power lead connected will result in tears. Or a PSU with less than honest ratings.


AFAIK some of those factors were in question early on, he replaced the PSU with one that looked like plenty with proper rails totalling 1000W (though I don't know the brand, Rosewill (?) ). The extra PCIe power connections & correct configuration was discussed, so I'd be surprised if they weren't connected.

From an outside standpoint, it's more like he was a new user that brought in the wrong help rather than what we know, which is if you want it done right you have to do it yourself :) That'll no doubt, IMO, get sorted well & truly when Slavac can strap on the foam-dome & give it a good old fashioned strip down & rebuild with us heckling on from the sidelines.

Jason
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1162556 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1162649 - Posted: 15 Oct 2011, 15:42:57 UTC

Seeems like those 560s are pushed pretty hard out of the box and don't have much headroom for overclocking on your own.
Unlike some of the older cards that you could get lucky with sometimes and find a real screamer.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1162649 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1162670 - Posted: 15 Oct 2011, 17:04:09 UTC - in response to Message 1162649.  

Seeems like those 560s are pushed pretty hard out of the box and don't have much headroom for overclocking on your own.
Unlike some of the older cards that you could get lucky with sometimes and find a real screamer.

Don't I remember some early reports, particularly with the factory overclocked versions, that they needed some extra overvolting to get the higher speeds to take?
ID: 1162670 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1162681 - Posted: 15 Oct 2011, 17:29:12 UTC - in response to Message 1162670.  

Seeems like those 560s are pushed pretty hard out of the box and don't have much headroom for overclocking on your own.
Unlike some of the older cards that you could get lucky with sometimes and find a real screamer.

Don't I remember some early reports, particularly with the factory overclocked versions, that they needed some extra overvolting to get the higher speeds to take?

Could be..
I have never played around with voltages on any of my cards. Most seem to run hot enough as it is. So I just take what I can get with stock voltages.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1162681 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1162752 - Posted: 15 Oct 2011, 21:52:45 UTC - in response to Message 1162670.  
Last modified: 15 Oct 2011, 21:53:54 UTC

Seeems like those 560s are pushed pretty hard out of the box and don't have much headroom for overclocking on your own.
Unlike some of the older cards that you could get lucky with sometimes and find a real screamer.

Don't I remember some early reports, particularly with the factory overclocked versions, that they needed some extra overvolting to get the higher speeds to take?


Correct, and certainly forms an important part the emerging complex bigger picture with these particular cards.

What nutting out Slavac's particular situation will lead to is a fairly comprehensive guide to stabilising these (& other cards) cards. The reasons extra care is needed for these is largely to do with the price/performance bracket these fit into, & the tendency for them to be purchased on a constrictive budget, with little consideration for assembly/design methods for the rest of the supporting system.

As experienced builders we take some of these steps we do intuitively for granted, intentionally overspeccing components & using sophisticated implementation & troubleshooting techniques intuitively as we go, applying & finding settings etc.

These are high performance cards price targeted at the mid market, which opens some cans of worms.

Jason
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1162752 · Report as offensive

Message boards : Number crunching : Only wrong -9 overflow results with GTX560/Ti?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.