GUPPI Rescheduler for Linux and Windows - Move GUPPI work to CPU and non-GUPPI to GPU

Message boards : Number crunching : GUPPI Rescheduler for Linux and Windows - Move GUPPI work to CPU and non-GUPPI to GPU
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 37 · Next

AuthorMessage
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1809771 - Posted: 17 Aug 2016, 0:55:13 UTC - in response to Message 1809770.  
Last modified: 17 Aug 2016, 0:56:30 UTC

I have my preferences set to only use 49% of my CPU cores to preserve enough cores to feed each GPU task with one CPU core.


I'm going to suggest you try changing that value to 100% and see what happens.

I'm willing to bet your CPU work units will pick up again.

Let me know

Zalster

Edit, if you didn't remove the project max concurrent, might be a good idea to change the value to 8. Just a thought
ID: 1809771 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1809775 - Posted: 17 Aug 2016, 1:04:22 UTC - in response to Message 1809771.  

Changed it back to 100% and it got the CPU tasks working again. Unfortunately, it got all 4 spare CPU cores running CPU task and processor usage at 100%. I want to reserve one CPU core for general desktop maintenance. I don't want the CPU running at 100%, I'm more comfortable with it running at 90%.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1809775 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1809776 - Posted: 17 Aug 2016, 1:06:34 UTC - in response to Message 1809775.  
Last modified: 17 Aug 2016, 1:10:29 UTC

Can you put the project max concurrent back into the app_config?

If you can

<project_max_concurrent>7</project_max_concurrent>

that will remove 1 CPU core from running and free up a core for the OS and support

so 4 GPU work units and 3 CPU work units

Edit..

Be sure to reread the config files when you do this
ID: 1809776 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1809778 - Posted: 17 Aug 2016, 1:10:12 UTC - in response to Message 1809776.  

Can you put the project max concurrent back into the app_config?

If you can

<project_max_concurrent>7</project_max_concurrent>

that will remove 1 CPU core from running and free up a core for the OS and support

so 4 GPU work units and 3 CPU work units

BINGO! Put <project_max_concurrent>7</project_max_concurrent> back in and now back to running 7 total tasks with the one extra CPU task started pushed into waiting to run.

Thanks so much for your help!
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1809778 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1809779 - Posted: 17 Aug 2016, 1:10:55 UTC - in response to Message 1809778.  

No problem, Glad it worked out for you

Zalster
ID: 1809779 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1809783 - Posted: 17 Aug 2016, 1:35:00 UTC - in response to Message 1809779.  

The change did impact the responsiveness of the desktop. I guess with one full core dedicated to each GPU task, I might have to back off some of my parameter settings so they aren't as aggressive.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1809783 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1809786 - Posted: 17 Aug 2016, 1:40:50 UTC - in response to Message 1809783.  

You can change the 7 to a 6 and see if the responsiveness gets better.

If it doesn't, then I would consider putting -use_sleep into you commandline for MB.

You could then put the value back to 7 and see how it does
ID: 1809786 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1809789 - Posted: 17 Aug 2016, 1:48:23 UTC - in response to Message 1809786.  

I guess I could go back to use_sleep. I know there have been changes to its implementation and it supposedly works better with less impact. I just liked not using it since my times improved so dramatically by not using it. I wasn't taxing the CPU very hard and had a very normal responsive desktop even when not using sleep since I wasn't dedicating that much cpu_usage to each GPU task in my previous app_config. Guess it will mean lots of playing around to find the optimum running parameters. Probably shouldn't have started this experiment in the middle of the WOW contest.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1809789 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1809820 - Posted: 17 Aug 2016, 5:17:40 UTC - in response to Message 1809789.  

I guess I could go back to use_sleep. I know there have been changes to its implementation and it supposedly works better with less impact. I just liked not using it since my times improved so dramatically by not using it. I wasn't taxing the CPU very hard and had a very normal responsive desktop even when not using sleep since I wasn't dedicating that much cpu_usage to each GPU task in my previous app_config. Guess it will mean lots of playing around to find the optimum running parameters. Probably shouldn't have started this experiment in the middle of the WOW contest.



. . And so endeth the lesson. Since going to r3500 this morning my rig has locked up 4 times. It's such a drag. Hence the Murphy's Law statute, if it's working, don't fix it :)
ID: 1809820 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1809822 - Posted: 17 Aug 2016, 5:33:21 UTC - in response to Message 1809820.  

Did you add -use_sleep to your commandline

How many GPUs, how many work units per card, how many CPU cores and do you have any CPU work units being crunched?
ID: 1809822 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1809826 - Posted: 17 Aug 2016, 6:13:28 UTC - in response to Message 1809820.  
Last modified: 17 Aug 2016, 6:23:00 UTC

Sorry to hear that Stephen. Yes, Murphy's Law usually gets around to bite you in the butt at least once a year just to remind you it's alive and kicking.

So far, no issues with R3500 for me. I moved from R3486 so I don't think there was that much change for me. Keeping my fingers crossed.

[Edit] Just saw your response in the main troubleshooting thread:

. . Hi Raistmer,

. . I have added the command line for r3500 as follows:

-use_sleep -high_prec_timer -sbs 512 -tt 1500

I think the parameters are too aggressive and are more likely to be the cause of your instability.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1809826 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1809835 - Posted: 17 Aug 2016, 7:25:13 UTC - in response to Message 1809752.  

Raistmer,
I hope some of the ideas they have been floating around help you in your work.
(It is a shame Windows doesn't have the same high frequency timers that some other o/s have)

Unfortunately, supposed to be 100ns-granularity method (https://gist.github.com/Youka/4153f12cf2e17a77314c) gives same 1ms only resolution on my netbook.
So, still nothing better than Sleep(1) with high-prec timer enabled.


Late to the party but have you tried SwitchToThread()? We use a combination of _mm_pause and STT for some of the more violent synchronization stuff.

1) currently app has only single worker thread. Others either from runtime or BOINC's API helper one. So switching to other thread inside process will just return to worker thread.
2) To yield to another process I could use Sleep(0) instead. It's possible currently by providing -use_sleep_ex 0 option to the app.
3) Unfortunately, this will little to no help regarding CPU consumption on typical host where CPU BOUNC processes at idle priority and GPU BOINC processes and below-normal priority. In such situation Sleep(0), again, will return to the same GPU app and continue to suck CPU cycles.

The goal is to suspend worker thread (or process as whole) for amount of time less than 1ms to save CPU cycles w/o GPU starvation. I don't see how switching to another thread will fullfill this goal in current conditions.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1809835 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1809840 - Posted: 17 Aug 2016, 8:25:59 UTC - in response to Message 1809822.  
Last modified: 17 Aug 2016, 8:27:29 UTC

Did you add -use_sleep to your commandline

How many GPUs, how many work units per card, how many CPU cores and do you have any CPU work units being crunched?



At first just ran the distribution settings and then when that seemed OK I added the command line:

-use_sleep -high_prec_timer -sbs 512 -tt 1500

. . 2 x GTX970s

. . 2 CPU cores

. . No CPU crunching and only one WU per GPU.

.
ID: 1809840 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1809841 - Posted: 17 Aug 2016, 8:27:58 UTC - in response to Message 1809840.  
Last modified: 17 Aug 2016, 8:28:21 UTC

sorry just saw your edit
ID: 1809841 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1809842 - Posted: 17 Aug 2016, 8:30:56 UTC - in response to Message 1809826.  
Last modified: 17 Aug 2016, 8:31:35 UTC

Sorry to hear that Stephen.

So far, no issues with R3500 for me. I moved from R3486 so I don't think there was that much change for me. Keeping my fingers crossed.

[Edit] Just saw your response in the main troubleshooting thread:

. . Hi Raistmer,

. . I have added the command line for r3500 as follows:

-use_sleep -high_prec_timer -sbs 512 -tt 1500

I think the parameters are too aggressive and are more likely to be the cause of your instability.



. . I jumped from r3430 and just tried that to see what the system would bear :)

. . I will probably drop them back a little to about -sbs 384 and -tt 500

. .
ID: 1809842 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1809843 - Posted: 17 Aug 2016, 8:31:03 UTC - in response to Message 1809840.  

Did you add -use_sleep to your commandline

How many GPUs, how many work units per card, how many CPU cores and do you have any CPU work units being crunched?



At first just ran the distribution settings and then when that seemed OK I added the command line:

-use_sleep -high_prec_timer -sbs 512 -tt 1500

. . 2 x GTX970s

. . 2 CPU cores

. . No CPU crunching and only one WU per GPU.

.


I would go with -use_sleep -sbs 512 -tt 500 to start with.

If it does ok, then consider -high_perc_timer

gives us a starting point
ID: 1809843 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1809845 - Posted: 17 Aug 2016, 8:37:06 UTC - in response to Message 1809843.  



At first just ran the distribution settings and then when that seemed OK I added the command line:

-use_sleep -high_prec_timer -sbs 512 -tt 1500

. . 2 x GTX970s

. . 2 CPU cores

. . No CPU crunching and only one WU per GPU.

.


I would go with -use_sleep -sbs 512 -tt 500 to start with.

If it does ok, then consider -high_perc_timer

gives us a starting point



. . It is worth a try.
ID: 1809845 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1809850 - Posted: 17 Aug 2016, 8:49:08 UTC - in response to Message 1809843.  

-high_prec_timer quite experimental feature for now.
Though some sources say it affects only particular program, MSDN quite clear in that multimedia timer precision change is system-wide. If so, it can affect other programs too.
So, in case of GUI freezes the first thing I would change is to disable this option.
Also, it has sense ONLY with -use_sleep. No sense to increase timer precision if timer not used...
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1809850 · Report as offensive
Profile Shaggie76
Avatar

Send message
Joined: 9 Oct 09
Posts: 282
Credit: 271,858,118
RAC: 196
Canada
Message 1809886 - Posted: 17 Aug 2016, 12:06:23 UTC - in response to Message 1809835.  


1) currently app has only single worker thread. Others either from runtime or BOINC's API helper one. So switching to other thread inside process will just return to worker thread.
2) To yield to another process I could use Sleep(0) instead. It's possible currently by providing -use_sleep_ex 0 option to the app.
3) Unfortunately, this will little to no help regarding CPU consumption on typical host where CPU BOUNC processes at idle priority and GPU BOINC processes and below-normal priority. In such situation Sleep(0), again, will return to the same GPU app and continue to suck CPU cycles.

The goal is to suspend worker thread (or process as whole) for amount of time less than 1ms to save CPU cycles w/o GPU starvation. I don't see how switching to another thread will fullfill this goal in current conditions.


Don't let the name fool you -- SwitchToThread is effectively Sleep(timeLeftInCurrentQuantum) and has two things going for it:

1) unlike Sleep(0) it actually switches even if the only other runnable thread is in a lower priority group whereas in this case Sleep(0) will just return without yielding.

2) you avoid a hard context switch if there's nowhere to go (and you can tell from the return value) and so in that case you can spin on _mm_pause for a while because on hyper-threaded system this will let a thread running on the other half of the core run at essentially full-speed.

Although it doesn't say so in MSDN I'm pretty sure that it also means 'any thread in the system' not just in your process.
ID: 1809886 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1809895 - Posted: 17 Aug 2016, 12:33:24 UTC - in response to Message 1809886.  
Last modified: 17 Aug 2016, 12:33:38 UTC


Don't let the name fool you -- SwitchToThread is effectively Sleep(timeLeftInCurrentQuantum) and has two things going for it:

Ok, I give it a try. Better to check experimentally, especially cause all real sleep quantum measuring logic already in place.


1) unlike Sleep(0) it actually switches even if the only other runnable thread is in a lower priority group whereas in this case Sleep(0) will just return without yielding.

That would be better indeed.


2) you avoid a hard context switch if there's nowhere to go (and you can tell from the return value) and so in that case you can spin on _mm_pause for a while because on hyper-threaded system this will let a thread running on the other half of the core run at essentially full-speed.

Although it doesn't say so in MSDN I'm pretty sure that it also means 'any thread in the system' not just in your process.

Thanks, I'll do some experimentation with it on real system.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1809895 · Report as offensive
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 37 · Next

Message boards : Number crunching : GUPPI Rescheduler for Linux and Windows - Move GUPPI work to CPU and non-GUPPI to GPU


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.