Developing a Multi-Threaded Benchmarking App for Linux

Message boards : Number crunching : Developing a Multi-Threaded Benchmarking App for Linux
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 6 · Next

AuthorMessage
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1965982 - Posted: 18 Nov 2018, 3:41:56 UTC

I have been working on an enhanced python version of the benchmark script in Linux. I have included the capability to read all apps with args from the BenchCFG file and run them on all cores or a maximum number of cores specified on the command line. Still under development. I was hoping for some help on the following items:

1) I remember using a GPU command line option a long time ago to specify which device to use. I think it was -Device x (where x is a number from 0 to Num of GPUs - 1). The experience was in Windows, but when I try this in Linux, I get a segmentation fault. Is this the correct way to specify a device with the Linux app?

2) I would like to include the current stock apps as reference apps. Is there a location to download these?

3) I would like to include a command line option to run test WUs that simulate specific signals. Is there a standard test set of WUs for this purpose?

4) I would like to indicate progress made while the benchmark is running, similar to the current boinc manager. Does anyone know the mechanism used?

Thanks!
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1965982 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 1729
Credit: 704,307,700
RAC: 2,400,708
United States
Message 1965983 - Posted: 18 Nov 2018, 4:19:56 UTC

I'm sorry that I can't be of any help to your questions. But I appreciate you making an attempt to make a useful too.

Just out of curiosity, once you finish the MB benchmark tool, do you think you'd be interested in creating a new AP benchmark tool? the current one is quite clunky and we could use a more polished tool.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1965983 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9661
Credit: 890,202,639
RAC: 1,704,473
United States
Message 1965984 - Posted: 18 Nov 2018, 4:35:35 UTC - in response to Message 1965982.  

I have been working on an enhanced python version of the benchmark script in Linux. I have included the capability to read all apps with args from the BenchCFG file and run them on all cores or a maximum number of cores specified on the command line. Still under development. I was hoping for some help on the following items:

1) I remember using a GPU command line option a long time ago to specify which device to use. I think it was -Device x (where x is a number from 0 to Num of GPUs - 1). The experience was in Windows, but when I try this in Linux, I get a segmentation fault. Is this the correct way to specify a device with the Linux app?

2) I would like to include the current stock apps as reference apps. Is there a location to download these?

3) I would like to include a command line option to run test WUs that simulate specific signals. Is there a standard test set of WUs for this purpose?

4) I would like to indicate progress made while the benchmark is running, similar to the current boinc manager. Does anyone know the mechanism used?

Thanks!

Unfortunately I only know of the /download directory for the BOINC applications. Nothing for the science apps. You could set up a dummy test machine for getting the stock science apps and then copy over the executables.

There are quite a few test WU's for MB V8 over at Lunatics in the Test and Benchmark Tools directory. You can always use the current tasks in your cache for test WU's also. If you use the work unit download generator tools you can retrieve the finished tasks that pique your interest before they are purged from the database.

Richard or Jord might know the mechanism the Client uses to calculate the progress indicator.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1965984 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1965988 - Posted: 18 Nov 2018, 5:58:25 UTC - in response to Message 1965983.  

I'm sorry that I can't be of any help to your questions. But I appreciate you making an attempt to make a useful too.

Just out of curiosity, once you finish the MB benchmark tool, do you think you'd be interested in creating a new AP benchmark tool? the current one is quite clunky and we could use a more polished tool.


I am running with AMD GPUs which give invalid Astropulse results when using drivers newer than 15.12, so I am unable to run Astropulse. So I am unlikely to take this on. Not sure if AP will have new life with any of the new data sources.
GitHub: Ricks-Lab
Instagram: ricks_labs
ID: 1965988 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1965991 - Posted: 18 Nov 2018, 6:05:52 UTC

I have encountered a new problem. When I try to pause BOINC with the following subprocess call:
cmd = subprocess.Popen("/home/boinc/BOINC/boinccmd --set_run_mode never 172800", shell=True, stdout=subprocess.PIPE)

I get the error:
Operation failed: authentication error

BOINC is running as me and I am executing the python script as me. Any ideas?
ID: 1965991 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9661
Credit: 890,202,639
RAC: 1,704,473
United States
Message 1966000 - Posted: 18 Nov 2018, 6:48:26 UTC

Maybe this page for python will help. https://stackoverflow.com/questions/4106565/newbie-python-subprocess-write-error-broken-pipe
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1966000 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1966001 - Posted: 18 Nov 2018, 6:59:46 UTC - in response to Message 1966000.  

Maybe this page for python will help. https://stackoverflow.com/questions/4106565/newbie-python-subprocess-write-error-broken-pipe


Since I am actually getting output from the command (authentication error), I don't think it is a broken pipe. Plus I have successfully used subprocess many other cases in the program. I am going to try setting the setuid bit on boinccmd to see if that makes a difference.
ID: 1966001 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1966002 - Posted: 18 Nov 2018, 7:28:30 UTC - in response to Message 1966001.  

Maybe this page for python will help. https://stackoverflow.com/questions/4106565/newbie-python-subprocess-write-error-broken-pipe


Since I am actually getting output from the command (authentication error), I don't think it is a broken pipe. Plus I have successfully used subprocess many other cases in the program. I am going to try setting the setuid bit on boinccmd to see if that makes a difference.


This one is now solved. I had to be in the boinc home directory when calling boinccmd.
ID: 1966002 · Report as offensive
W3Perl Project Donor
Volunteer tester

Send message
Joined: 29 Apr 99
Posts: 239
Credit: 2,826,981,053
RAC: 3,542,427
France
Message 1966003 - Posted: 18 Nov 2018, 7:48:21 UTC - in response to Message 1966002.  

Maybe this page for python will help. https://stackoverflow.com/questions/4106565/newbie-python-subprocess-write-error-broken-pipe


Since I am actually getting output from the command (authentication error), I don't think it is a broken pipe. Plus I have successfully used subprocess many other cases in the program. I am going to try setting the setuid bit on boinccmd to see if that makes a difference.


This one is now solved. I had to be in the boinc home directory when calling boinccmd.


or you need to specify the password file if you don't want to chdir to boinc home directory :
/<boinc_path>/boinccmd --host <hostname>:<boinc_port> --passwd <gui_rpc_auth.cfg value> --set_gpu_mode <mode_value>

:<boinc_port> is optional if you use the standard port (31416)
ID: 1966003 · Report as offensive
W3Perl Project Donor
Volunteer tester

Send message
Joined: 29 Apr 99
Posts: 239
Credit: 2,826,981,053
RAC: 3,542,427
France
Message 1966006 - Posted: 18 Nov 2018, 7:56:19 UTC - in response to Message 1965982.  

I have been working on an enhanced python version of the benchmark script in Linux. I have included the capability to read all apps with args from the BenchCFG file and run them on all cores or a maximum number of cores specified on the command line. Still under development. I was hoping for some help on the following items:

1) I remember using a GPU command line option a long time ago to specify which device to use. I think it was -Device x (where x is a number from 0 to Num of GPUs - 1). The experience was in Windows, but when I try this in Linux, I get a segmentation fault. Is this the correct way to specify a device with the Linux app?


To get the list of available card :
nvidia-smi --query-gpu=index,name --format=csv,noheader
To specify the first card :
nvidia-smi --query-gpu=name --format=csv,noheader -i 0
ID: 1966006 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1966007 - Posted: 18 Nov 2018, 8:17:28 UTC - in response to Message 1966006.  

I have been working on an enhanced python version of the benchmark script in Linux. I have included the capability to read all apps with args from the BenchCFG file and run them on all cores or a maximum number of cores specified on the command line. Still under development. I was hoping for some help on the following items:

1) I remember using a GPU command line option a long time ago to specify which device to use. I think it was -Device x (where x is a number from 0 to Num of GPUs - 1). The experience was in Windows, but when I try this in Linux, I get a segmentation fault. Is this the correct way to specify a device with the Linux app?


To get the list of available card :
nvidia-smi --query-gpu=index,name --format=csv,noheader
To specify the first card :
nvidia-smi --query-gpu=name --format=csv,noheader -i 0


I am actually looking for the seti MB app command line option, so when I execute the job, I can specify which gpu to use. Is the mechnism for this different than on windows where it seems to be a command line option?
ID: 1966007 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1966008 - Posted: 18 Nov 2018, 8:20:17 UTC - in response to Message 1966003.  

Maybe this page for python will help. https://stackoverflow.com/questions/4106565/newbie-python-subprocess-write-error-broken-pipe


Since I am actually getting output from the command (authentication error), I don't think it is a broken pipe. Plus I have successfully used subprocess many other cases in the program. I am going to try setting the setuid bit on boinccmd to see if that makes a difference.


This one is now solved. I had to be in the boinc home directory when calling boinccmd.


or you need to specify the password file if you don't want to chdir to boinc home directory :
/<boinc_path>/boinccmd --host <hostname>:<boinc_port> --passwd <gui_rpc_auth.cfg value> --set_gpu_mode <mode_value>

:<boinc_port> is optional if you use the standard port (31416)


This is good to know, but since I will eventually make the script available publicly, I think the directory change will be more appropriate.
ID: 1966008 · Report as offensive
W3Perl Project Donor
Volunteer tester

Send message
Joined: 29 Apr 99
Posts: 239
Credit: 2,826,981,053
RAC: 3,542,427
France
Message 1966009 - Posted: 18 Nov 2018, 8:47:06 UTC - in response to Message 1966007.  

I have been working on an enhanced python version of the benchmark script in Linux. I have included the capability to read all apps with args from the BenchCFG file and run them on all cores or a maximum number of cores specified on the command line. Still under development. I was hoping for some help on the following items:

1) I remember using a GPU command line option a long time ago to specify which device to use. I think it was -Device x (where x is a number from 0 to Num of GPUs - 1). The experience was in Windows, but when I try this in Linux, I get a segmentation fault. Is this the correct way to specify a device with the Linux app?


To get the list of available card :
nvidia-smi --query-gpu=index,name --format=csv,noheader
To specify the first card :
nvidia-smi --query-gpu=name --format=csv,noheader -i 0


I am actually looking for the seti MB app command line option, so when I execute the job, I can specify which gpu to use. Is the mechnism for this different than on windows where it seems to be a command line option?


this is what ps show on my computer for the GPU :
../../projects/setiathome.berkeley.edu/setiathome_x41zc_x86_64-pc-linux-gnu_cuda92 -nobs -pfb 16 --device 0

for the CPU, I don't know if you can select on which core you want to run (they are all the same inside a CPU).
ID: 1966009 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2766
Credit: 549,879,821
RAC: 839,348
Canada
Message 1966010 - Posted: 18 Nov 2018, 9:09:38 UTC - in response to Message 1966009.  

for the CPU, I don't know if you can select on which core you want to run (they are all the same inside a CPU).
If one wanted a specific thread, or mask even, one could use something like:
schedtool -v -a 1,3,5 `pidof setiathome_x41zi_x86_64-pc-linux-gnu_cuda80`

As long as the app name is unique enough, it can be done with GPU or CPU tasks.
ID: 1966010 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1966017 - Posted: 18 Nov 2018, 10:54:25 UTC - in response to Message 1966010.  

for the CPU, I don't know if you can select on which core you want to run (they are all the same inside a CPU).
If one wanted a specific thread, or mask even, one could use something like:
schedtool -v -a 1,3,5 `pidof setiathome_x41zi_x86_64-pc-linux-gnu_cuda80`

As long as the app name is unique enough, it can be done with GPU or CPU tasks.


Thanks Brent! This will be useful in the future. Still need to figure out why I can't specify the GPU to be used for a job...
ID: 1966017 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2766
Credit: 549,879,821
RAC: 839,348
Canada
Message 1966019 - Posted: 18 Nov 2018, 11:10:21 UTC - in response to Message 1966017.  

Since your taking notes for the future, it also does Priority (Nice Levels):
# Assign CPU Priority (19=Nice, -20=High)
schedtool -n -20 `pidof astropulse_7.08_x86_64-pc-linux-gnu__opencl_nvidia_100`
ID: 1966019 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1966023 - Posted: 18 Nov 2018, 13:08:06 UTC
Last modified: 18 Nov 2018, 13:08:37 UTC

Concerning the starting of GPU tasks on a specified device, I have checked the MB app source code and found that "--device n" is correct, so maybe there is something else wrong with my code. I am using this chunk of code to start both GPU and CPU tasks:
                    #Execute job
                    shutil.copy2(slots.list[slot_num].job.wu_path + slots.list[slot_num].job.wu_name,
                            slots.list[slot_num].slot_dir +"/" + mb_const.activeWU)
                    if plat == "GPU":
                        for file_str in glob.glob(env.gpu_app_path + "*.cl"):
                            shutil.copy2(file_str, slots.list[slot_num].slot_dir +"/")
                    if plat == "CPU":
                        cmd_str_list = [env.cpu_app_path + slots.list[slot_num].job.app_name , slots.list[slot_num].job.app_args]
                    else:
                        device_arg = "--device " + str(slots.list[slot_num].device)
                        cmd_str_list = [env.gpu_app_path + slots.list[slot_num].job.app_name,
                                device_arg + " " + slots.list[slot_num].job.app_args]
                    if mb_const.DEBUG == True: print(cmd_str_list)
                    os.chdir(slots.list[slot_num].slot_dir)
                    slots.list[slot_num].job.cmd = subprocess.Popen(cmd_str_list, shell=False, stdout=subprocess.PIPE)
                    slots.list[slot_num].job.start_time = datetime.utcnow()
                    os.chdir(env.current_dir)

It works fine for CPU tasks, but still not working for GPU. Here is what cmd_str_list looks like for a CPU task:
['/home/rick/PyDev/benchMB/APPS_CPU/MBv8_8.05r3345_avx_linux64', '--nographics'] 

Maybe there is still something I have not prepared properly in the working directory for a GPU job.
ID: 1966023 · Report as offensive
Gene Project Donor

Send message
Joined: 26 Apr 99
Posts: 129
Credit: 39,015,313
RAC: 47,108
United States
Message 1966120 - Posted: 19 Nov 2018, 5:57:56 UTC

@ReuiKe
Re: downloadable stock apps...
I don't remember how I discovered it, but useful for those of us "anonymous platform" hosts that don't get the automatic downloads of the science apps. Even if we only want to run them as benchmarks to compare to optimized apps.
In the following list the application name, as found in the ->Computing ->Applications web page; then the full application file name.
Linux/x86_64    8.00
setiathome_8.00_x86_64-pc-linux-gnu
---
Linux/x86_64    8.22(opencl_nvidia_sah)
setiathome_8.22_x86_64-pc-linux-gnu__opencl_nvidia_sah
---
Linux/x86_64   8.22(opencl_nvidia_SoG)
setiathome_8.22_x86_64-pc-linux-gnu__opencl_nvidia_SoG
---

Point a browser to:
http://boinc2.ssl.berkeley.edu/sah/download_fanout/<file name> #without the < > of course and it will download to you.
These are the only stock apps that I have in hand and could verify this method. But, no doubt, you will see a pattern in the morphing from Applications description to the downloadable file name, in case you want additional stock apps.
(I hope I'm not revealing any Berkeley secrets here; I have not found any on-line index, or "How-To" , for fetching the stock apps.)

I will follow this thread for progress reports as I am frequently motivated to do benchmark experiments of my own.

AZGene;
ID: 1966120 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1966140 - Posted: 19 Nov 2018, 11:52:12 UTC - in response to Message 1966120.  

@ReuiKe
Re: downloadable stock apps...
I don't remember how I discovered it, but useful for those of us "anonymous platform" hosts that don't get the automatic downloads of the science apps. Even if we only want to run them as benchmarks to compare to optimized apps.
In the following list the application name, as found in the ->Computing ->Applications web page; then the full application file name.
Linux/x86_64    8.00
setiathome_8.00_x86_64-pc-linux-gnu
---
Linux/x86_64    8.22(opencl_nvidia_sah)
setiathome_8.22_x86_64-pc-linux-gnu__opencl_nvidia_sah
---
Linux/x86_64   8.22(opencl_nvidia_SoG)
setiathome_8.22_x86_64-pc-linux-gnu__opencl_nvidia_SoG
---

Point a browser to:
http://boinc2.ssl.berkeley.edu/sah/download_fanout/<file name> #without the < > of course and it will download to you.
These are the only stock apps that I have in hand and could verify this method. But, no doubt, you will see a pattern in the morphing from Applications description to the downloadable file name, in case you want additional stock apps.
(I hope I'm not revealing any Berkeley secrets here; I have not found any on-line index, or "How-To" , for fetching the stock apps.)

I will follow this thread for progress reports as I am frequently motivated to do benchmark experiments of my own.

AZGene;


Thanks Gene, That worked!

I hope to have a functional beta by the end of next weekend. Things I need to complete:
1) Write meaningful data to testData directory. Will likely leverage the comparison routine used in the previous package
2) Run and store results for reference apps
3) Figure out how to make previous compiled kernels available for new runs. I am using a slots approach, similar to boinc. It looks like boinc writes compiled kernels to a common directory, but in my case, the job running in a slot is expecting the kernels in the slots dir. Still some digging to do on this one.
ID: 1966140 · Report as offensive
Profile RueiKe Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 14 Feb 16
Posts: 465
Credit: 306,803,989
RAC: 298,914
Taiwan
Message 1966150 - Posted: 19 Nov 2018, 13:46:58 UTC - in response to Message 1966023.  

I have figured out this issue. I had to add a space before the "-device n". Not sure why it is needed since I don't put a space in front of the cpu app argument string.

Concerning the starting of GPU tasks on a specified device, I have checked the MB app source code and found that "--device n" is correct, so maybe there is something else wrong with my code. I am using this chunk of code to start both GPU and CPU tasks:
                    #Execute job
                    shutil.copy2(slots.list[slot_num].job.wu_path + slots.list[slot_num].job.wu_name,
                            slots.list[slot_num].slot_dir +"/" + mb_const.activeWU)
                    if plat == "GPU":
                        for file_str in glob.glob(env.gpu_app_path + "*.cl"):
                            shutil.copy2(file_str, slots.list[slot_num].slot_dir +"/")
                    if plat == "CPU":
                        cmd_str_list = [env.cpu_app_path + slots.list[slot_num].job.app_name , slots.list[slot_num].job.app_args]
                    else:
                        device_arg = "--device " + str(slots.list[slot_num].device)
                        cmd_str_list = [env.gpu_app_path + slots.list[slot_num].job.app_name,
                                device_arg + " " + slots.list[slot_num].job.app_args]
                    if mb_const.DEBUG == True: print(cmd_str_list)
                    os.chdir(slots.list[slot_num].slot_dir)
                    slots.list[slot_num].job.cmd = subprocess.Popen(cmd_str_list, shell=False, stdout=subprocess.PIPE)
                    slots.list[slot_num].job.start_time = datetime.utcnow()
                    os.chdir(env.current_dir)

It works fine for CPU tasks, but still not working for GPU. Here is what cmd_str_list looks like for a CPU task:
['/home/rick/PyDev/benchMB/APPS_CPU/MBv8_8.05r3345_avx_linux64', '--nographics'] 

Maybe there is still something I have not prepared properly in the working directory for a GPU job.
ID: 1966150 · Report as offensive
1 · 2 · 3 · 4 . . . 6 · Next

Message boards : Number crunching : Developing a Multi-Threaded Benchmarking App for Linux


 
©2019 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.