Monitoring inconclusive GBT validations and harvesting data for testing

Message boards : Number crunching : Monitoring inconclusive GBT validations and harvesting data for testing
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · 14 . . . 36 · Next

AuthorMessage
Profile -= Vyper =-
Volunteer tester
Avatar

Send message
Joined: 5 Sep 99
Posts: 1652
Credit: 1,065,191,981
RAC: 2,537
Sweden
Message 1814577 - Posted: 2 Sep 2016, 7:55:25 UTC
Last modified: 2 Sep 2016, 8:05:04 UTC

Petri mentioned SMX units and the unroll value! Should it be the same value as what the manufacturer says it should be?
It seems like that a 750Ti should have unroll 5 set! Is that correct?

"GeForce GTX 750

The 1Gb GeForce GTX 750 ships with 4 activated SMX units containing 512 Shader Cores and 32 texture units. The core clock frequency will be 1020 MHz while it can boost to 1085 MHz. The memory speed is locked at a 5010 MHz effective data rate based on a 1252 MHz quad data rate for GDDR5 over 128-bit memory bus.

GeForce GTX 750 Ti

The more interesting product will be the GTX 750 Ti which has 5 activated SMX units containing 640 Shader Cores and 40 texture units / 16 ROPs. The core clock frequency will be 1020 MHz while it can boost to 1085 MHz. The memory speed is locked at an 5400 MHz effective data rate based on a 1350 MHz quad data rate for GDDR5 over 128-bit memory bus.
"

_________________________________________________________________________
Addicted to SETI crunching!
Founder of GPU Users Group
ID: 1814577 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1814587 - Posted: 2 Sep 2016, 8:45:27 UTC - in response to Message 1814576.  
Last modified: 2 Sep 2016, 9:28:54 UTC

That line is present when the 750Ti Hangs or Stalls. It hasn't 'Stalled' yet, but it's probably just a matter of time.


My 4 750Ti's Gigabyte Black edition (Made for 24/7 in a serverpark environment) just chugs along just fine with Petris app.
Works like a charm for days and days.

http://setiathome.berkeley.edu/show_host_detail.php?hostid=8053171

Could you please try to increase the voltage to the GPU in your case. I've experienced several times that manufacturers bios is tweaked so it doesn't give the GPU "juice" enough to maintain itself.
Please oc the gpu voltage and try again.

I have experienced this on my 1080 and i did just that and several others GPUs in the past too.

Looking at your one AP here, Driver version: 367.35
That is a CUDA 8 Driver. In fact, that is the Same series driver I Updated to, that Stopped the Hangs that were happening with driver 352.79. Try it with this driver, http://www.nvidia.com/Download/driverResults.aspx/97645/en-us My 2 EVGA and 1 Zotac 750Ti cards chug along with the zi3 Apps also, as long as I use a CUDA 8 driver. I didn't have that problem with any of the Baseline Apps or the zi Special Apps, and I have compiled and tested Dozens of each App version. It Only happens with the zi3 Special versions. You can't change any voltages on a Mac, and since it works in Linux with driver 367.79 there is No reason to change it in Linux. The fact that it ONLY happens with the zi3 version Apps, and happens on both Platforms, is indication the problem is with the App. When the problem happens on Multiple cards on Multiple Platforms with just one particular App, it's a pretty good indication of where the problem exists.

BTW, here are the OSX CUDA Drivers, http://www.nvidia.com/object/mac-driver-archive.html
Note there aren't any CUDA 8 drivers. The only place you can find a OSX CUDA 8 driver is in the CUDA 8 Toolkit. I don't think that will go over very well, having to register as a Developer to download the Toolkit so you can install a working Driver. Also, the driver in the Toolkit doesn't work with the current OSX, Darwin 15.6. My guess is you'll have to wait until Darwin 16.0 before you will see a Public CUDA 8 driver.
ID: 1814587 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1814618 - Posted: 2 Sep 2016, 11:04:47 UTC - in response to Message 1814577.  

Also, I see the Autocorrelation Error on your machine as well,
Best autocorr: peak=45123.75, time=5.727
Strange it's always around that same exact time.
That's a lot of Inconclusives to dig through, you should update to x41zi3f as soon as possible.
ID: 1814618 · Report as offensive
Kiska
Volunteer tester

Send message
Joined: 31 Mar 12
Posts: 302
Credit: 3,067,762
RAC: 0
Australia
Message 1814619 - Posted: 2 Sep 2016, 11:06:43 UTC - in response to Message 1814618.  

Datafile has been archived here
ID: 1814619 · Report as offensive
Profile -= Vyper =-
Volunteer tester
Avatar

Send message
Joined: 5 Sep 99
Posts: 1652
Credit: 1,065,191,981
RAC: 2,537
Sweden
Message 1814626 - Posted: 2 Sep 2016, 12:09:55 UTC - in response to Message 1814618.  

Also, I see the Autocorrelation Error on your machine as well,
Best autocorr: peak=45123.75, time=5.727
Strange it's always around that same exact time.
That's a lot of Inconclusives to dig through, you should update to x41zi3f as soon as possible.


Where can i find it? I would do it as soon as i can get my hands on it.

_________________________________________________________________________
Addicted to SETI crunching!
Founder of GPU Users Group
ID: 1814626 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1668
Credit: 623,086,772
RAC: 156
Finland
Message 1814627 - Posted: 2 Sep 2016, 12:33:53 UTC - in response to Message 1814626.  

Also, I see the Autocorrelation Error on your machine as well,
Best autocorr: peak=45123.75, time=5.727
Strange it's always around that same exact time.
That's a lot of Inconclusives to dig through, you should update to x41zi3f as soon as possible.


Where can i find it? I would do it as soon as i can get my hands on it.


I could e-mail that to you..
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1814627 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1668
Credit: 623,086,772
RAC: 156
Finland
Message 1814628 - Posted: 2 Sep 2016, 12:42:36 UTC - in response to Message 1814618.  
Last modified: 2 Sep 2016, 12:44:21 UTC

Also, I see the Autocorrelation Error on your machine as well,
Best autocorr: peak=45123.75, time=5.727
Strange it's always around that same exact time.
That's a lot of Inconclusives to dig through, you should update to x41zi3f as soon as possible.



The exact same time would indicate that something gets overwritten (buffer underflow or overflow) or that the chirping produces an artifact in the data and the autocorrelation check finds a false positive.

I'll check if I can reproduce the same autocorr error. If yes then I'll start finding a fix.

Howabout the OpenCL version - does it have a cut off peak power so that it does not report overly high values?
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1814628 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1668
Credit: 623,086,772
RAC: 156
Finland
Message 1814630 - Posted: 2 Sep 2016, 12:52:58 UTC - in response to Message 1814577.  
Last modified: 2 Sep 2016, 12:53:10 UTC

Petri mentioned SMX units and the unroll value! Should it be the same value as what the manufacturer says it should be?
It seems like that a 750Ti should have unroll 5 set! Is that correct?

"GeForce GTX 750

The 1Gb GeForce GTX 750 ships with 4 activated SMX units containing 512 Shader Cores and 32 texture units. The core clock frequency will be 1020 MHz while it can boost to 1085 MHz. The memory speed is locked at a 5010 MHz effective data rate based on a 1252 MHz quad data rate for GDDR5 over 128-bit memory bus.

GeForce GTX 750 Ti

The more interesting product will be the GTX 750 Ti which has 5 activated SMX units containing 640 Shader Cores and 40 texture units / 16 ROPs. The core clock frequency will be 1020 MHz while it can boost to 1085 MHz. The memory speed is locked at an 5400 MHz effective data rate based on a 1350 MHz quad data rate for GDDR5 over 128-bit memory bus.
"


I have tried with different unroll values and found that the unroll should be the number of SMX units or at least 75% of them.
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1814630 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1814631 - Posted: 2 Sep 2016, 12:53:13 UTC - in response to Message 1814628.  
Last modified: 2 Sep 2016, 12:55:37 UTC


Howabout the OpenCL version - does it have a cut off peak power so that it does not report overly high values?


In prev build such big numbers would trigger sanity check.
Currently sanity check for autocorr disabled so any value will pass through to validator and validator will decide if such power valid or not (there is agreement between wingmans or not).

EDIT: and no, even with sanity check enabled there is no any "cut-off" - task just would end with computation error.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1814631 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1668
Credit: 623,086,772
RAC: 156
Finland
Message 1814632 - Posted: 2 Sep 2016, 12:54:11 UTC - in response to Message 1814631.  


Howabout the OpenCL version - does it have a cut off peak power so that it does not report overly high values?


In prev build such big numbers would trigger sanity check.
Currently sanity check for autocorr disabled so any value will pass through to validator and validator will decide if such power valid or not (there is agreement between wingmans or not).


Thanks.
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1814632 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1814891 - Posted: 3 Sep 2016, 22:39:07 UTC

I ran across a couple machines that may be of interest;

Coprocessors: [3] NVIDIA GeForce GTX 1080 (8192MB) OpenCL: 1.2 Operating System: Darwin 15.6.0
In progress (98) · Validation pending (480) · Validation inconclusive (402) · Valid (193) · Invalid (10) · Error (4)
http://setiathome.berkeley.edu/results.php?hostid=8018045

Coprocessors: NVIDIA GeForce GTX TITAN X (12288MB) OpenCL: 1.2 Operating System: Darwin 15.6.0
In progress (74) · Validation pending (224) · Validation inconclusive (223) · Valid (114) · Invalid (5) · Error (0)
http://setiathome.berkeley.edu/results.php?hostid=7297852

Shame with all that power the machines are forced to use an App that rates around 'Weakly similar' Q= 19.24%.
Much better to use the one at Beta that rates 'Strongly similar' Q= 99.82%. Even the Special App does better than Q= 19.

In other news I changed the two 750Ti to another machine and installed the latest version of Ubuntu 14.04, along with the updated driver. We'll see how it works now.
ID: 1814891 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1814935 - Posted: 4 Sep 2016, 2:48:49 UTC - in response to Message 1814891.  

In other news I changed the two 750Ti to another machine and installed the latest version of Ubuntu 14.04, along with the updated driver. We'll see how it works now.


That just reminded me that my Linux machine has a 680, which is one compute capability lower than the new code will do at the moment. Will have to think what to do about that, since the 3 platform simultaneous build system automation is coming along.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1814935 · Report as offensive
Kiska
Volunteer tester

Send message
Joined: 31 Mar 12
Posts: 302
Credit: 3,067,762
RAC: 0
Australia
Message 1815004 - Posted: 4 Sep 2016, 13:17:03 UTC

Just uploaded about ~250 results and datafiles.
ID: 1815004 · Report as offensive
Profile -= Vyper =-
Volunteer tester
Avatar

Send message
Joined: 5 Sep 99
Posts: 1652
Credit: 1,065,191,981
RAC: 2,537
Sweden
Message 1815315 - Posted: 6 Sep 2016, 5:57:34 UTC

Things seem to improve. Look further down and look at my consecutive valid tasks with Petris latest revision.

http://setiathome.berkeley.edu/host_app_versions.php?hostid=8053171

*Thumbs up*

_________________________________________________________________________
Addicted to SETI crunching!
Founder of GPU Users Group
ID: 1815315 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1815319 - Posted: 6 Sep 2016, 6:16:56 UTC - in response to Message 1813747.  
Last modified: 6 Sep 2016, 7:15:47 UTC

The OpenCL Apps scored much worse in Darwin 15.4 even though these Apps worked very well in Darwin 14.5.
KWSN-Darwin-MBbench v2.1.07
Running on TomsMacPro.local at Tue Aug 30 07:31:46 2016
---------------------------------------------------
Starting benchmark run...
---------------------------------------------------
Listing wu-file(s) in /testWUs :
reference_work_unit_r3215.wu

Listing executable(s) in /APPS :
MBv8_8.05r3346_nvidia_ssse3_x86_64-apple-darwin

Listing executable in /REF_APPs :
MBv8_8.05r3344_sse41_x86_64-apple-darwin
---------------------------------------------------
Current WU: reference_work_unit_r3215.wu
---------------------------------------------------
Skipping default app MBv8_8.05r3344_sse41_x86_64-apple-darwin, displaying saved result(s)
Elapsed Time: ………………………………… 2104 seconds
---------------------------------------------------
Running app with command : MBv8_8.05r3346_nvidia_ssse3_x86_64-apple-darwin
Elapsed Time : ……………………………… 480 seconds
Speed compared to default : 438 %
-----------------
Comparing results
                ------------- R1:R2 ------------     ------------- R2:R1 ------------
                Exact  Super  Tight  Good    Bad     Exact  Super  Tight  Good    Bad
        Spike      0      9     11     13      0        0      9     11     13      0
     Autocorr      0      1      1      1      0        0      1      1      1      0
     Gaussian      0      0      0      1      5        0      0      0      1      5
        Pulse      0      0      0      0      0        0      0      0      0      2
      Triplet      0      1      1      2      0        0      1      1      2      1
   Best Spike      0      1      1      1      0        0      1      1      1      0
Best Autocorr      0      1      1      1      0        0      1      1      1      0
Best Gaussian      0      0      0      0      1        0      0      0      0      1
   Best Pulse      0      0      0      0      1        0      0      0      0      1
 Best Triplet      0      1      1      1      0        0      1      1      1      0
                ----   ----   ----   ----   ----     ----   ----   ----   ----   ----
                   0     14     16     20      7        0     14     16     20     10

Unmatched signal(s) in R1 at line(s) 499 526 580 607 634 694 720
Unmatched signal(s) in R2 at line(s) 482 509 526 569 595 649 676 703 763 789
For R1:R2 matched signals only, Q= 7.881%
Result      : Weakly similar.
-------------------------------------------------------------------------------------


I booted into Darwin 14.5 to compile x41p_zi3g and decided to try a few of the Apps there. The Above is the App r3346 in El Capitan. Below is the exact same App in Yosemite;
KWSN-Darwin-MBbench v2.1.07
Running on TomsMacPro.local at Tue Sep 6 04:13:22 2016
---------------------------------------------------
Starting benchmark run...
---------------------------------------------------
Listing wu-file(s) in /testWUs :
blc2_2bit_guppi_57403_HIP11048_0006.17091.831.22.45.71.wu reference_work_unit_r3215.wu

Listing executable(s) in /APPS :
MBv8_8.05r3346_nvidia_ssse3_x86_64-apple-darwin MBv8_8.17r3516_NV_ssse3_x86_64-apple-darwin setiathome_8.10_x86_64-apple-darwin__opencl_nvidia_mac

Listing executable in /REF_APPs :
MBv8_8.05r3344_sse41_x86_64-apple-darwin
---------------------------------------------------
Current WU: blc2_2bit_guppi_57403_HIP11048_0006.17091.831.22.45.71.wu
---------------------------------------------------
Running default app with command : MBv8_8.05r3344_sse41_x86_64-apple-darwin
     4955.87 real      4942.34 user        11.31 sys
Elapsed Time: ………………………………… 4955 seconds
---------------------------------------------------
Running app with command : MBv8_8.05r3346_nvidia_ssse3_x86_64-apple-darwin
     1131.83 real       126.52 user       261.62 sys
Elapsed Time : ……………………………… 1131 seconds
Speed compared to default : 438 %
-----------------
Comparing results
Result      : Strongly similar,  Q= 99.89%
---------------------------------------------------


El Capitan: Result : Weakly similar, Q= 7.881%
Yosemite: Result : Strongly similar, Q= 99.89%

Quite a difference, wouldn't you say?

Here is the App running reference_work_unit_r3215.wu;
---------------------------------------------------
Done with blc2_2bit_guppi_57403_HIP11048_0006.17091.831.22.45.71.wu.
Current WU: reference_work_unit_r3215.wu
---------------------------------------------------
Skipping default app MBv8_8.05r3344_sse41_x86_64-apple-darwin, displaying saved result(s)
Elapsed Time: ………………………………… 2198 seconds
---------------------------------------------------
Running app with command : MBv8_8.05r3346_nvidia_ssse3_x86_64-apple-darwin -sbs 192 -oclfft_tune_gr 256 -oclfft_tune_wg 128
      326.80 real        97.70 user        87.68 sys
Elapsed Time : ……………………………… 327 seconds
Speed compared to default : 672 %
-----------------
Comparing results
Result      : Strongly similar,  Q= 99.47%
---------------------------------------------------


Here's the App at Beta in Yosemite;
---------------------------------------------------
Current WU: blc2_2bit_guppi_57403_HIP11048_0006.17091.831.22.45.71.wu
---------------------------------------------------
Skipping default app MBv8_8.05r3344_sse41_x86_64-apple-darwin, displaying saved result(s)
Elapsed Time: ………………………………… 4955 seconds
---------------------------------------------------
Running app with command : setiathome_8.10_x86_64-apple-darwin__opencl_nvidia_mac
     1086.06 real       128.77 user       241.15 sys
Elapsed Time : ……………………………… 1086 seconds
Speed compared to default : 456 %
-----------------
Comparing results
Result      : Strongly similar,  Q= 99.98%
---------------------------------------------------

Much different in EL Capitan;
---------------------------------------------------
Current WU: blc2_2bit_guppi_57403_HIP11048_0006.17091.831.22.45.71.wu
---------------------------------------------------
Skipping default app MBv8_8.05r3344_sse41_x86_64-apple-darwin, displaying saved result(s)
Elapsed Time: ………………………………… 4797 seconds
---------------------------------------------------
Running app with command : setiathome_8.10_x86_64-apple-darwin__opencl_nvidia_mac
     1444.76 real       117.24 user       257.14 sys
Elapsed Time : ……………………………… 1444 seconds
Speed compared to default : 332 %
-----------------
Comparing results
                ------------- R1:R2 ------------     ------------- R2:R1 ------------
                Exact  Super  Tight  Good    Bad     Exact  Super  Tight  Good    Bad
        Spike      0      0      2      6      0        0      0      2      6      0
     Autocorr      0      0      0      0      0        0      0      0      0      0
     Gaussian      0      0      0      0      0        0      0      0      0      0
        Pulse      0      0      0      5      0        0      0      0      5      1
      Triplet      0      0      0      1      0        0      0      0      1      0
   Best Spike      0      0      1      1      0        0      0      1      1      0
Best Autocorr      0      0      0      0      1        0      0      0      0      1
Best Gaussian      1      1      1      1      0        1      1      1      1      0
   Best Pulse      0      0      0      1      0        0      0      0      1      0
 Best Triplet      0      0      0      1      0        0      0      0      1      0
                ----   ----   ----   ----   ----     ----   ----   ----   ----   ----
                   1      1      4     16      1        1      1      4     16      2

Unmatched signal(s) in R1 at line(s) 608
Unmatched signal(s) in R2 at line(s) 592 636
For R1:R2 matched signals only, Q= 19.24%
Result      : Weakly similar.
---------------------------------------------------
ID: 1815319 · Report as offensive
Kiska
Volunteer tester

Send message
Joined: 31 Mar 12
Posts: 302
Credit: 3,067,762
RAC: 0
Australia
Message 1815328 - Posted: 6 Sep 2016, 7:24:20 UTC - in response to Message 1815319.  

A significant difference, perhaps we should stop distribution of the affected versions of apples broken 'driver'
ID: 1815328 · Report as offensive
Profile -= Vyper =-
Volunteer tester
Avatar

Send message
Joined: 5 Sep 99
Posts: 1652
Credit: 1,065,191,981
RAC: 2,537
Sweden
Message 1815331 - Posted: 6 Sep 2016, 7:39:34 UTC

Thats insane, and that is without changing a single line of code, only exchanging O/S version?
If so i wonder what is going on really. Seems like s@h need to ban El Capitan or what is the suggestion now?

_________________________________________________________________________
Addicted to SETI crunching!
Founder of GPU Users Group
ID: 1815331 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1815332 - Posted: 6 Sep 2016, 7:40:26 UTC - in response to Message 1815328.  
Last modified: 6 Sep 2016, 7:43:05 UTC

That's been talked about for a while, https://setiathome.berkeley.edu/forum_thread.php?id=78569&postid=1811003#1811003
Nothing has happened...yet.

It all started back here, https://setiathome.berkeley.edu/forum_thread.php?id=78569&postid=1813801#1813801

Hmmmm, my latest prediction is, there will be more people testing the Mac CUDA Apps at Beta soon.
ID: 1815332 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1815345 - Posted: 6 Sep 2016, 9:12:56 UTC - in response to Message 1815331.  
Last modified: 6 Sep 2016, 9:15:06 UTC

Thats insane, and that is without changing a single line of code, only exchanging O/S version?
If so i wonder what is going on really. Seems like s@h need to ban El Capitan or what is the suggestion now?

Well, don't we seeing just the same with OpenCL NV build over than year (as TBar stated) already ?...
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1815345 · Report as offensive
Kiska
Volunteer tester

Send message
Joined: 31 Mar 12
Posts: 302
Credit: 3,067,762
RAC: 0
Australia
Message 1815346 - Posted: 6 Sep 2016, 9:18:34 UTC

Fact is, we are going to see alot more inconclusive from macs due to the fact the checker in the benchmarker is based upon the validator
ID: 1815346 · Report as offensive
Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · 14 . . . 36 · Next

Message boards : Number crunching : Monitoring inconclusive GBT validations and harvesting data for testing


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.