Some Help with SoG buils

Message boards : Number crunching : Some Help with SoG buils
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1899396 - Posted: 5 Nov 2017, 20:43:19 UTC

Since I have trouble with the video lag when running SoG bulds on my host. Explained on this thread https://setiathome.berkeley.edu/forum_thread.php?id=82094
And no problem with that lag when run CUDA builds but crunch blc builds with CUDA is to slow, i think on a detour.
Besides the obvious path of rescheduling the WU from GPU to CPU etc. was thinking if is possible to configurate the Seti (or Boinc) to crunch Normal GPU WU with CUDA and if the rescheduled leave some blc to the GPU crunch them with the SoG builds?
If that detour could be programed on the GPU0 only, the one who generate the video output? Leaving the other s running normaly?
That could not fix the problem but certainly will give a help.
ID: 1899396 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1899398 - Posted: 5 Nov 2017, 20:55:05 UTC - in response to Message 1899396.  

You might try a ignore command in the cc_config.xml so that no Seti runs on that GPU. Just a thought
ID: 1899398 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1899419 - Posted: 5 Nov 2017, 22:43:10 UTC

But i want to run Seti on that GPU, but CUDA if there are arecibo WU and SoG when blc.

Just not know how to configurate to do that.
ID: 1899419 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1899421 - Posted: 5 Nov 2017, 23:05:28 UTC - in response to Message 1899419.  

But i want to run Seti on that GPU, but CUDA if there are arecibo WU and SoG when blc.

Just not know how to configurate to do that.
I think that it could probably be done by assigning different plan classes to each type of task, but you'd need to do a bit of custom rescheduling to accomplish that. The tasks for the GPUs would all download under one plan class (whichever of the two you place first in your app_info), but then you'd need to reassign the tasks for the other plan class after the download.
ID: 1899421 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1899424 - Posted: 5 Nov 2017, 23:20:19 UTC - in response to Message 1899396.  

Since I have trouble with the video lag when running SoG bulds on my host. Explained on this thread https://setiathome.berkeley.edu/forum_thread.php?id=82094

My i7 2600 with 2 GTX 1070s is able to play HD YouTube videos with only slight occasional stutters with very aggressive command line settings

Have you tried running that system with SBS set to 1024 (all other settings as they are)?
Have you tried running it with out any command line settings?


Another thought- have to tried running a Latency Monitor while crunching WUs with & without playing back video to see what the effect is?
Grant
Darwin NT
ID: 1899424 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1899426 - Posted: 5 Nov 2017, 23:33:12 UTC - in response to Message 1899424.  
Last modified: 5 Nov 2017, 23:33:48 UTC

My i7 2600 with 2 GTX 1070s is able to play HD YouTube videos with only slight occasional stutters with very aggressive command line settings

With the 1070 the lag is very occasional as you said, but still exist. But what i try to configurate first is the 1060 host.

Have you tried running that system with SBS set to 1024 (all other settings as they are)?

Yes. The command lines i used to test are in the other thread.

Have you tried running it with out any command line settings?

Yes. The lag still there

Another thought- have to tried running a Latency Monitor while crunching WUs with & without playing back video to see what the effect is?

LM show everithing OK. I run only 1 or 2 CPU WU on a 6 cores CPU´s. So in theory i have a lot of spare processing power to feed the GPU.

I try even running only the GPU WU. All OK with CUDA build, the problem appears only when the SoG build is running.

This lag is something like when we try to run a Vlar on the CUDA builds.....
ID: 1899426 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1899428 - Posted: 5 Nov 2017, 23:50:59 UTC - in response to Message 1899421.  

But i want to run Seti on that GPU, but CUDA if there are arecibo WU and SoG when blc.

Just not know how to configurate to do that.
I think that it could probably be done by assigning different plan classes to each type of task, but you'd need to do a bit of custom rescheduling to accomplish that. The tasks for the GPUs would all download under one plan class (whichever of the two you place first in your app_info), but then you'd need to reassign the tasks for the other plan class after the download.

Thanks but that´s is well beyound my pay grade... LOL
ID: 1899428 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1899429 - Posted: 5 Nov 2017, 23:54:18 UTC - in response to Message 1899426.  

My i7 2600 with 2 GTX 1070s is able to play HD YouTube videos with only slight occasional stutters with very aggressive command line settings

With the 1070 the lag is very occasional as you said, but still exist. But what i try to configurate first is the 1060 host.

Yeah, I just find it odd that your GTX 1060 system is getting the stutter/lag even with such high period_iterations_num and low TT values. And removing the High_perf option doesn't help?

And the version of SoG you are running was one of those Raistmer optimized to reduce lag on much lower powered GPU hardware than yours, so even running without any command line values, i'd have expected it to be OK.
Grant
Darwin NT
ID: 1899429 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1899436 - Posted: 6 Nov 2017, 0:23:39 UTC - in response to Message 1899421.  

I think that it could probably be done by assigning different plan classes to each type of task, but you'd need to do a bit of custom rescheduling to accomplish that. The tasks for the GPUs would all download under one plan class (whichever of the two you place first in your app_info), but then you'd need to reassign the tasks for the other plan class after the download.
I was thinking much the same. And I think W3Perl would be the one to talk to about customizing a cpu2gpu script for that. Basically take all the GPU tasks and split them off to 2 different plan_classes by file name. With a 1060 running it every 18 hours should work OK.
ID: 1899436 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1899500 - Posted: 6 Nov 2017, 10:43:55 UTC - in response to Message 1899426.  
Last modified: 6 Nov 2017, 10:46:40 UTC


This lag is something like when we try to run a Vlar on the CUDA builds.....
(that's quite BIG lag )

So, to make it clear, you see lag with SoG app on non-VLAR tasks? on all of them? Or on VHAR ones only?
Did you try to correlate lag with startup time (before real GPU prcessing start).
Can you post any picture from GPU-Z showing GPU load correlated with moments of LAG occurence?
Is lag the rare/occasional or on some tasks you see lags when, for example, key pressed and sequence of same letter printed in CMD window?

EDIT: and did you try just to change driver version on another (bigger/smaller - no matter, jsut another)
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1899500 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1899518 - Posted: 6 Nov 2017, 13:33:52 UTC - in response to Message 1899500.  

Will do the test ASAP when i arrive at home in few hrs.

Any sugestion on what command line i use for the test?
ID: 1899518 · Report as offensive
anon
Volunteer tester

Send message
Joined: 14 Jun 04
Posts: 12
Credit: 3,621,913
RAC: 0
Australia
Message 1899521 - Posted: 6 Nov 2017, 13:45:06 UTC

Run the MB8 win x86 SSE3 OpenCL NV SoG r3584 app at vanilla settings to do the testing & use this command line parameter in the app_info.xml file, <cmdline>-hp</cmdline>. The -hp command line parameter elevates the app CPU priority to high to keep the GPU processing to the maximum level.
ID: 1899521 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1899619 - Posted: 6 Nov 2017, 22:30:24 UTC
Last modified: 6 Nov 2017, 22:30:38 UTC

@Raistmer

Sorry i dont know how to put an img here so i DL to my one drive, hope that works

https://1drv.ms/f/s!Asjkc9Jyluh3zU1oCAEbvIPzuSir

For all the test i run an allready DL episode of The Walking Dead in low resolution (about 480 MB the entire 50 min file) stored on the SSD , so i imagine there is no internet or HDD lag.

I Made 4 test, two with CUDA builds and Arecibo NonVlar WU and no Lag appears as i explain before. Not try blc with CUDA since we all know it´s a waste of time... LoL

The others 2 are with the SoG builds with this command line: -use_sleep -sbs 128 -period_iterations_num 250 -tt 150 -no_defaults_scaling

To try to isolate the problem i run the test crunching GPU WU only the entire CPU was free .

The first one is with Arecibo Non Vlar and the second with blw WU, in both i see the lag on the screen, both are in the middle of the processing of the WU

Something interesting, each time the lag apears, in the memory controler graphics it show´s a dive, can´t say if thas happening before of after the efect the lag (too fast for my human eyes).

During the test i run on the host, Boinc, The crunching builds, MPC (for the movie) and GPU-Z only. The Boinc data & program directories are in a exception of the AV and nothing else was running on the host (at least not shows any other process active in the task manager screen). NO OC on CPU (6800)/GPU and just Plain Win 10

Hope this could give you a path to follow.

And thanks again for your help and please forgive my bad english.
ID: 1899619 · Report as offensive
bluestar

Send message
Joined: 5 Sep 12
Posts: 6995
Credit: 2,084,789
RAC: 3
Message 1899637 - Posted: 7 Nov 2017, 1:04:38 UTC - in response to Message 1899619.  
Last modified: 7 Nov 2017, 1:21:41 UTC



Why not have a couple of beers instead, juan?

Code:

img https://setiathome.berkeley.edu/user_profile/images/8606388_avatar.jpg /img

Replace img with [ ] and [/ ] with the text img inside, without any spaces or clicking the URL button, but next you again have the preview being visible from the code itself.

Trying to be helpful, only thing I can say is that making it a link here only strains things a little and it better should be embedded in the page itself, except for making it too large in size,
which once happened to me in the past.
ID: 1899637 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1899668 - Posted: 7 Nov 2017, 5:59:23 UTC - in response to Message 1899619.  
Last modified: 7 Nov 2017, 6:49:39 UTC

@Raistmer

The others 2 are with the SoG builds with this command line: -use_sleep -sbs 128 -period_iterations_num 250 -tt 150 -no_defaults_scaling

While I looking more precisely, this CMD line definitely not for lag-free cards.
-tt 150 means to allow ~150ms length of kernel. That is, 0.15s . Such intervals can be visible by human senses indeed.
I would attempt to test with such line: -sbs 256 -period_iterations_num 500 -tt 10
(and allow all scaling app needs for first test then to add -no_defaults_scaling on second iteration if lags remain)

EDIT: There is little GPU load increase also, each time GPU memory controller shows decrease.
So, GPU function ~100% (instead of mean 93%) those times.
So, it's some too long GPU kernel most probably. Need to find which one.
Try my recommendation first, then we'll continue debugging.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1899668 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1899671 - Posted: 7 Nov 2017, 6:34:19 UTC

Tested Arecibo WU with:

<cmdline>-sbs 256 -period_iterations_num 500 -tt 10</cmdline>

The lag still there with the same drop in the memory controler graph.
ID: 1899671 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1899677 - Posted: 7 Nov 2017, 6:50:56 UTC - in response to Message 1899671.  
Last modified: 7 Nov 2017, 6:53:34 UTC

Tested Arecibo WU with:

<cmdline>-sbs 256 -period_iterations_num 500 -tt 10</cmdline>

The lag still there with the same drop in the memory controler graph.


That's sad. Do you able to run with profiler to see exact lengths of kernels involved?

EDIT Is it GTX1060 GPU ?
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1899677 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1899680 - Posted: 7 Nov 2017, 7:13:32 UTC - in response to Message 1899677.  

Added: -no_defaults_scaling

The lag still there with the same drop in the memory controler graph

That's sad. Do you able to run with profiler to see exact lengths of kernels involved?


If anyone tell me how to do, sure

EDIT Is it GTX1060 GPU ?

Yes is a GTX 1060 3 GB

But i tested with a 980Ti, a 1070 and a 1080Ti on other hosts and the same lag exists, apparently less in the top cards but still there.

The only point all the hosts have in common is all are X99 MB, with 6800 or 6850 CPU´s
ID: 1899680 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1899683 - Posted: 7 Nov 2017, 7:38:06 UTC

Now something just came to mind that I did many years ago which may be of help.

Modern browsers and many video playback apps these days have a setting called "use hardware acceleration" (or similar) which I turned off many years ago here due to lag I was suffering back then. This setting is usually on by default and uses the GPU for that "hardware acceleration" so I wonder if this could be the root of your problem.

Cheers.
ID: 1899683 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1899731 - Posted: 7 Nov 2017, 12:58:21 UTC - in response to Message 1899680.  
Last modified: 7 Nov 2017, 12:59:56 UTC

The only point all the hosts have in common is all are X99 MB, with 6800 or 6850 CPU´s

And driver versions?

Well, you need to download and install nSight profiler or smth alike and most probably to run modified build to get any results (modification required due to unfriendly way BOINC framework dictates scientific app exit sequence: there is no call for exit() that does all required maintenance including profiler data flushing to file, there is just hard termination on OS level that leaves profiler data files empty (true for all profilers I used so far: CPU(PGO)/ATi/NV). And this requires working build environment and I still lack of such currently.
So, more advanced debugging should be postponed.

Currently try to run with -high_prec_timer -cpu_lock_fixed_cpu 2 (I assume you have at least 3 CPUs in your host).

If lag still there try to run with -v 6 option and upload stderr.txt for examination.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1899731 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : Some Help with SoG buils


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.