Issues with 1 GPU on Penta Nano System

Message boards : Number crunching : Issues with 1 GPU on Penta Nano System
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1826830 - Posted: 26 Oct 2016, 19:28:56 UTC - in response to Message 1826788.  


The process priority increase only applies to GPU tasks. It probably makes no difference with -hp option.

This is incorrect. That option applies only to CPU. You want the option:
<process_priority_special>N</process_priority_special>

from the BOINC Client Configuration Wiki.

Client configuration
<process_priority>N</process_priority>, <process_priority_special>N</process_priority_special>
The OS process priority at which tasks are run. Values are 0 (lowest priority, the default), 1 (below normal), 2 (normal), 3 (above normal), 4 (high) and 5 (real-time - not recommended). 'special' process priority is used for coprocessor (GPU) applications, wrapper applications, and non-compute-intensive applications, 'process priority' for all others. The two options can be used independently. List-add.pngNew in 7.6.14

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1826830 · Report as offensive
Profile BilBg
Volunteer tester
Avatar

Send message
Joined: 27 May 07
Posts: 3720
Credit: 9,385,827
RAC: 0
Bulgaria
Message 1826831 - Posted: 26 Oct 2016, 19:45:27 UTC - in response to Message 1826788.  
Last modified: 26 Oct 2016, 20:17:04 UTC

The process priority increase only applies to GPU tasks.

1) From cc_config.xml

"
<process_priority>N</process_priority>, <process_priority_special>N</process_priority_special>
   The OS process priority at which tasks are run. Values are 0 (lowest priority, the default), 1 (below normal), 2 (normal), 3 (above normal), 4 (high) and 5 (real-time - not recommended).
'special' process priority is used for coprocessor (GPU) applications, wrapper applications, and non-compute-intensive applications, 'process priority' for all others. The two options can be used independently. (New in 7.6.14)
"

2) Your picture shows CPU task at "Above Normal" Priority:


 
 


- ALF - "Find out what you don't do well ..... then don't do it!" :)
 
ID: 1826831 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1826841 - Posted: 26 Oct 2016, 21:43:03 UTC

Actually what matter is computing thread priority and thread priority handling in Windows more complex than just process priority.
For example, with "above normal" process priority but "idle" thread priority resulting priority still will be idle (zero). To rise "idle" priority thread one need to come into real time priority process area. So, for CPU apps BOINC should additionally change base priority of working thread to smth above idle. W/o that rising process priority is senseless.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1826841 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1826847 - Posted: 26 Oct 2016, 22:27:50 UTC - in response to Message 1826841.  

Actually what matter is computing thread priority and thread priority handling in Windows more complex than just process priority.
For example, with "above normal" process priority but "idle" thread priority resulting priority still will be idle (zero). To rise "idle" priority thread one need to come into real time priority process area. So, for CPU apps BOINC should additionally change base priority of working thread to smth above idle. W/o that rising process priority is senseless.

There's a fairly clear exposition of the relationship between "priority classes" (for processes) and "priority levels" (for threads) in the MSDN article on Scheduling Priorities.
ID: 1826847 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1826858 - Posted: 26 Oct 2016, 23:42:32 UTC - in response to Message 1826847.  
Last modified: 26 Oct 2016, 23:44:16 UTC

Actually what matter is computing thread priority and thread priority handling in Windows more complex than just process priority.
For example, with "above normal" process priority but "idle" thread priority resulting priority still will be idle (zero). To rise "idle" priority thread one need to come into real time priority process area. So, for CPU apps BOINC should additionally change base priority of working thread to smth above idle. W/o that rising process priority is senseless.

There's a fairly clear exposition of the relationship between "priority classes" (for processes) and "priority levels" (for threads) in the MSDN article on Scheduling Priorities.

Exactly. And as that table shows to change idle thread priority via process priority change one need to go to real time priority of process.
All else will result in the same idle thread priority (and such behavior peculiar to idle priority thread).
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1826858 · Report as offensive
Profile RueiKe Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 492
Credit: 378,512,430
RAC: 785
Taiwan
Message 1826866 - Posted: 27 Oct 2016, 0:53:09 UTC - in response to Message 1826830.  


The process priority increase only applies to GPU tasks. It probably makes no difference with -hp option.

This is incorrect. That option applies only to CPU. You want the option:
<process_priority_special>N</process_priority_special>

from the BOINC Client Configuration Wiki.

Client configuration
<process_priority>N</process_priority>, <process_priority_special>N</process_priority_special>
The OS process priority at which tasks are run. Values are 0 (lowest priority, the default), 1 (below normal), 2 (normal), 3 (above normal), 4 (high) and 5 (real-time - not recommended). 'special' process priority is used for coprocessor (GPU) applications, wrapper applications, and non-compute-intensive applications, 'process priority' for all others. The two options can be used independently. List-add.pngNew in 7.6.14


Thanks all for your input on this. I will try Keith's recommendation later today.
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1826866 · Report as offensive
Profile RueiKe Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 492
Credit: 378,512,430
RAC: 785
Taiwan
Message 1826902 - Posted: 27 Oct 2016, 5:19:41 UTC - in response to Message 1826866.  


The process priority increase only applies to GPU tasks. It probably makes no difference with -hp option.

This is incorrect. That option applies only to CPU. You want the option:
<process_priority_special>N</process_priority_special>

from the BOINC Client Configuration Wiki.

Client configuration
<process_priority>N</process_priority>, <process_priority_special>N</process_priority_special>
The OS process priority at which tasks are run. Values are 0 (lowest priority, the default), 1 (below normal), 2 (normal), 3 (above normal), 4 (high) and 5 (real-time - not recommended). 'special' process priority is used for coprocessor (GPU) applications, wrapper applications, and non-compute-intensive applications, 'process priority' for all others. The two options can be used independently. List-add.pngNew in 7.6.14


Thanks all for your input on this. I will try Keith's recommendation later today.


I found that when I remove <process_priority>3</process_priority>, the CPU app prioriy goes to Idle and GPU apps are unchanged. When I add <process_priority_special>3</process_priority_special>, the MB app priority is still high. I am going to set it to 4 and see if there is any difference, but I guess since -hp option is being used, it may not make a difference. My hope is that it will give priority to GPU apps in getting FPU resources. So far, this doesn't have an impact on my underloaded GPU issue.
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1826902 · Report as offensive
Profile RueiKe Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 492
Credit: 378,512,430
RAC: 785
Taiwan
Message 1826905 - Posted: 27 Oct 2016, 9:04:52 UTC - in response to Message 1826902.  

I found that when I remove <process_priority>3</process_priority>, the CPU app prioriy goes to Idle and GPU apps are unchanged. When I add <process_priority_special>3</process_priority_special>, the MB app priority is still high. I am going to set it to 4 and see if there is any difference, but I guess since -hp option is being used, it may not make a difference. My hope is that it will give priority to GPU apps in getting FPU resources. So far, this doesn't have an impact on my underloaded GPU issue.


Update...
I made this change on 3 systems and within 3 hours, all 3 were down. One system had an OC fail, the other 2 were on a circuit breaker that tripped. I think this change has actually resulted in heavier loading, though I still need to confirm that with data.
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1826905 · Report as offensive
Profile RueiKe Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 492
Credit: 378,512,430
RAC: 785
Taiwan
Message 1827838 - Posted: 1 Nov 2016, 7:06:00 UTC

Latest update:
I did a clean Win10Pro install and reinstalled all drivers. System came back up with the same issue. Some additional observations:

Problem is worse with GUPPI tasks, Arecibo tasks take only slightly longer on this card.

I am now seeing invalid AP tasks on this card.

Perhaps it is a HW issue. Is there a good test of GPU compute integrity?
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1827838 · Report as offensive
Profile BilBg
Volunteer tester
Avatar

Send message
Joined: 27 May 07
Posts: 3720
Credit: 9,385,827
RAC: 0
Bulgaria
Message 1828003 - Posted: 2 Nov 2016, 20:09:28 UTC - in response to Message 1827838.  
Last modified: 2 Nov 2016, 20:12:41 UTC

Perhaps it is a HW issue. Is there a good test of GPU compute integrity?

you already know how to run tasks only on one GPU to see how it performs when "alone":
http://setiathome.berkeley.edu/forum_thread.php?id=80426&postid=1826782#1826782

"LuxMark is a OpenCL cross-platform benchmark"
http://www.luxrender.net/wiki/LuxMark#Download

"FurMark - GPU Stress Test, OpenGL Benchmark"
http://www.ozone3d.net/benchmarks/fur/

"GPU Caps Viewer is an OpenGL and OpenCL graphics card utility" (have tests (Demos))
http://www.ozone3d.net/gpu_caps_viewer/


"OCCT Perestroïka is a stability checking tool"
http://www.ocbase.com/
 
 


- ALF - "Find out what you don't do well ..... then don't do it!" :)
 
ID: 1828003 · Report as offensive
Profile RueiKe Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 492
Credit: 378,512,430
RAC: 785
Taiwan
Message 1828070 - Posted: 3 Nov 2016, 1:48:48 UTC - in response to Message 1828003.  
Last modified: 3 Nov 2016, 1:49:22 UTC

Perhaps it is a HW issue. Is there a good test of GPU compute integrity?

you already know how to run tasks only on one GPU to see how it performs when "alone":
http://setiathome.berkeley.edu/forum_thread.php?id=80426&postid=1826782#1826782

"LuxMark is a OpenCL cross-platform benchmark"
http://www.luxrender.net/wiki/LuxMark#Download

"FurMark - GPU Stress Test, OpenGL Benchmark"
http://www.ozone3d.net/benchmarks/fur/

"GPU Caps Viewer is an OpenGL and OpenCL graphics card utility" (have tests (Demos))
http://www.ozone3d.net/gpu_caps_viewer/

"OCCT Perestroïka is a stability checking tool"
http://www.ocbase.com/
 


Thanks for your input! I hope to make more progress on it this weekend and will report back here.
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1828070 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1831874 - Posted: 22 Nov 2016, 8:42:27 UTC - in response to Message 1826467.  

And here is the bug:
BOINC assigns device 2
4 slot of 64 used for this instance; total_GPU_instances_num=5
Info: BOINC provided OpenCL device ID used
Info: CPU affinity mask used: 0; system mask is ff
With such mask no CPUs allowed at all.
Perhaps Windows API reacts on such mask as it would be 0xff instead allowing ALL CPUs.
Please check affinity of all active tasks on that host - each of them is pinned to only single CPU or some have few available CPUs?


Well, seems the reason of bug is clear - system mask ff corresponds not 8 CPU system but 32 CPU one. Hence wrong path selected.

So I need value that correctly reflects real number of CPUs in system...
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1831874 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1831885 - Posted: 22 Nov 2016, 11:14:49 UTC - in response to Message 1831874.  

And here is the bug:
BOINC assigns device 2
4 slot of 64 used for this instance; total_GPU_instances_num=5
Info: BOINC provided OpenCL device ID used
Info: CPU affinity mask used: 0; system mask is ff
With such mask no CPUs allowed at all.
Perhaps Windows API reacts on such mask as it would be 0xff instead allowing ALL CPUs.
Please check affinity of all active tasks on that host - each of them is pinned to only single CPU or some have few available CPUs?


Well, seems the reason of bug is clear - system mask ff corresponds not 8 CPU system but 32 CPU one. Hence wrong path selected.

So I need value that correctly reflects real number of CPUs in system...


If system mask is the same as the hex value used for the start /affinity command then it should be much longer hex for a 32 CPU system. 0x55555555 would be for all even number CPUs on a 32 CPU system
I've used this command line to have BOINC run only on physical cores and not use HT cores for some test on my 16c/32t system.
start /AFFINITY 55555555 boinc.exe --detach
Using FF give apps access only to CPUs 0-7. But the start /affinity command may not be operating the same as calling the values in code.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1831885 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1831995 - Posted: 23 Nov 2016, 8:22:42 UTC - in response to Message 1831885.  

You are right, but all this irrelevant to new build already:
https://cloud.mail.ru/public/7aDi/EYjULaHQW

I need the same picture of affinity as before (for example, from ProcessExplorer one).
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1831995 · Report as offensive
Profile RueiKe Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 492
Credit: 378,512,430
RAC: 785
Taiwan
Message 1832167 - Posted: 24 Nov 2016, 11:25:08 UTC - in response to Message 1831995.  

You are right, but all this irrelevant to new build already:
https://cloud.mail.ru/public/7aDi/EYjULaHQW

I need the same picture of affinity as before (for example, from ProcessExplorer one).


I just finished testing. Looks good. I posted video of the test here:
https://youtu.be/5nj2B79uTo4
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1832167 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1832220 - Posted: 24 Nov 2016, 18:13:21 UTC - in response to Message 1832167.  

Seems now v8.21 can be build.
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1832220 · Report as offensive
Profile RueiKe Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 492
Credit: 378,512,430
RAC: 785
Taiwan
Message 1835098 - Posted: 10 Dec 2016, 1:32:03 UTC
Last modified: 10 Dec 2016, 1:43:21 UTC

Looks like Crimson 16.12.1 fixed the issue, though I need to monitor for a while before I conclude it is closed. New Crimson version seems to have fixed all issues with running more than 4 AMD GPUs in a system!
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1835098 · Report as offensive
Profile RueiKe Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 492
Credit: 378,512,430
RAC: 785
Taiwan
Message 1835103 - Posted: 10 Dec 2016, 1:45:19 UTC - in response to Message 1835098.  

Looks like Crimson 16.12.1 fixed the issue, though I need to monitor for a while before I conclude it is closed. New Crimson version seems to have fixed all issues with running more than 4 AMD GPUs in a system!


I spoke too soon. It shifted to another Device, but still the same GPU. My Triple ProDuo system has all 6 GPU's fully loaded also with an FX8370, so I think that rules out FPU capacity.
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1835103 · Report as offensive
Previous · 1 · 2 · 3

Message boards : Number crunching : Issues with 1 GPU on Penta Nano System


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.