V10 of modified SETI MB CUDA + opt AP package for full multi-GPU+CPU use

Message boards : Number crunching : V10 of modified SETI MB CUDA + opt AP package for full multi-GPU+CPU use
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 . . . 15 · Next

AuthorMessage
Rob.B

Send message
Joined: 23 Jul 99
Posts: 157
Credit: 1,439,682
RAC: 0
United Kingdom
Message 869377 - Posted: 25 Feb 2009, 17:18:25 UTC

I'm running a quad + 1 cuda. I had 5 ap's running. Suspended all ap's let CUDA pick up a job then resumed 4. Running V8 team package. I assume my app_info.xml is stuffed, see below:

app_info>
<app>
<name>astropulse</name>
</app>
<file_info>
<name>ap_5.00r103_SSE3.exe</name>
<executable/>
</file_info>
<app_version>
<app_name>astropulse</app_name>
<version_num>500</version_num>
<file_ref>
<file_name>ap_5.00r103_SSE3.exe</file_name>
<main_program/>
</file_ref>
</app_version>
<app>
<name>astropulse_v5</name>
</app>
<file_info>
<name>ap_5.03r112_SSE3.exe</name>
<executable/>
</file_info>
<app_version>
<app_name>astropulse_v5</app_name>
<version_num>503</version_num>
<file_ref>
<file_name>ap_5.03r112_SSE3.exe</file_name>
<main_program/>
</file_ref>
</app_version>
<app>
<name>setiathome_enhanced</name>
</app>
<file_info>
<name>AK_v8b_win_SSE3_AMD_GPU_CPU_team.exe</name>
<executable/>
</file_info>
<file_info>
<name>cudart.dll</name>
<executable/>
</file_info>
<file_info>
<name>cufft.dll</name>
<executable/>
</file_info>
<file_info>
<name>libfftw3f-3-1-1a_upx.dll</name>
<executable/>
</file_info>
<file_info>
<name>MB_6.08_mod_CPU_team_CUDA.exe</name>
<executable/>
</file_info>

<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>528</version_num>
<file_ref>
<file_name>AK_v8b_win_SSE3_AMD_GPU_CPU_team.exe</file_name>
<main_program/>
</file_ref>
<file_ref>
<file_name>cudart.dll</file_name>
</file_ref>
<file_ref>
<file_name>cufft.dll</file_name>
</file_ref>
<file_ref>
<file_name>libfftw3f-3-1-1a_upx.dll</file_name>
</file_ref>
<file_ref>
<file_name>MB_6.08_mod_CPU_team_CUDA.exe</file_name>
</file_ref>
</app_version>

<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>603</version_num>
<file_ref>
<file_name>AK_v8b_win_SSE3_AMD_GPU_CPU_team.exe</file_name>
<main_program/>
</file_ref>
<file_ref>
<file_name>cudart.dll</file_name>
</file_ref>
<file_ref>
<file_name>cufft.dll</file_name>
</file_ref>
<file_ref>
<file_name>libfftw3f-3-1-1a_upx.dll</file_name>
</file_ref>
<file_ref>
<file_name>MB_6.08_mod_CPU_team_CUDA.exe</file_name>
</file_ref>
</app_version>


<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>607</version_num>
<file_ref>
<file_name>AK_v8b_win_SSE3_AMD_GPU_CPU_team.exe</file_name>
<main_program/>
</file_ref>
<file_ref>
<file_name>cudart.dll</file_name>
</file_ref>
<file_ref>
<file_name>cufft.dll</file_name>
</file_ref>
<file_ref>
<file_name>libfftw3f-3-1-1a_upx.dll</file_name>
</file_ref>
<file_ref>
<file_name>MB_6.08_mod_CPU_team_CUDA.exe</file_name>
</file_ref>
</app_version>

<app_version>
<app_name>setiathome_enhanced</app_name>
<version_num>608</version_num>
<file_ref>
<file_name>AK_v8b_win_SSE3_AMD_GPU_CPU_team.exe</file_name>
<main_program/>
</file_ref>
<file_ref>
<file_name>cudart.dll</file_name>
</file_ref>
<file_ref>
<file_name>cufft.dll</file_name>
</file_ref>
<file_ref>
<file_name>libfftw3f-3-1-1a_upx.dll</file_name>
</file_ref>
<file_ref>
<file_name>MB_6.08_mod_CPU_team_CUDA.exe</file_name>
</file_ref>
</app_version>

</app_info>


ID: 869377 · Report as offensive
Rob.B

Send message
Joined: 23 Jul 99
Posts: 157
Credit: 1,439,682
RAC: 0
United Kingdom
Message 869379 - Posted: 25 Feb 2009, 17:22:57 UTC

Sorry missed possibly useful info:

AMD Phenom X4
9400GT+ CUDA graphics card
4Gb Ram
Windows Vista Ultimate SP1
Boinc 6.4.5

Rob

ID: 869379 · Report as offensive
Rob.B

Send message
Joined: 23 Jul 99
Posts: 157
Credit: 1,439,682
RAC: 0
United Kingdom
Message 869380 - Posted: 25 Feb 2009, 17:28:21 UTC

Not sure if this helps, but I was running a 2 to 3 mix od AP 5.00 and AP V5 5_03.

Cuda driver: 7.15.11.8122 which I beleive equates to 182.xxxxx
ID: 869380 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 869396 - Posted: 25 Feb 2009, 18:20:18 UTC - in response to Message 869374.  

ok, im stuck with 9 ap's running on my 8-core with v8-10 with mbs sitting there waiting for some cuda action, not very satisfactory

i also want to return to mb on cuda only and optimised ap's on cpu only

is there an edit to the app info that will allow this ? is so could someone kindly post the edit please

boinc appears incapable of sharing the work in a logical manner, not suprising seeing as though we are trying to fool it :-)

this is in no way detracting from your great work Raistmer :-) i just want a simpler setup

regards

adrian


You need to edit app_info.xml for that.
Take as base one from V9 multi-GPU pack.
You can use it w/o modification actually, MB will work only on GPU and AP will work only on CPU.
So just download multi-GPU pack (V9 one).
ID: 869396 · Report as offensive
Gnitter

Send message
Joined: 2 Jan 07
Posts: 26
Credit: 19,909,753
RAC: 0
Sweden
Message 869401 - Posted: 25 Feb 2009, 18:43:32 UTC - in response to Message 869396.  
Last modified: 25 Feb 2009, 18:51:56 UTC

Reporting in....

So far so god. Running 8Ap + 2MB.
Saw somone sugested 7+2 for better performance
Will run this settings for now and se how things develop.

Thanx for all the efforts thats been put in to this Opt GPU thing :)


XPpro32 SP3
I7920 OC´ed
2x GTX260 stock speed

Edit:
Forgot to mention that i at the moment got 16AP waiting to be prosessed
at an estimated time of 12 hour each. Thats probably why Boinc is running
8 AP at the moment....
182.06 driver

Best regards
ID: 869401 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 869404 - Posted: 25 Feb 2009, 19:06:55 UTC - in response to Message 869401.  

To all who can't get it work correctly with different projects.
Please, keep in mind that primary goal of all these "team" packs is to make use both GPU and CPU for SETI MultiBeam project.
If you try to run something else (even including SETI AP provided in these pack too) you can (and sometimes will) meet situation with idle GPU if you leave BOINC unattended for long time. If BOINC will start ncpus CPU-only tasks and all of them will be not SETI MB then this mod cant help to keep GPU busy. At least one SETI MB task should be active for mod features work.
You can manually suspend/resume other projects or tasks or you can try to fine tune project shares - it's all on your own discretion.
ID: 869404 · Report as offensive
THE PHANTOM
Volunteer tester
Avatar

Send message
Joined: 13 Apr 01
Posts: 46
Credit: 1,154,279
RAC: 1
United States
Message 869415 - Posted: 25 Feb 2009, 19:54:35 UTC - in response to Message 869404.  
Last modified: 25 Feb 2009, 19:56:30 UTC

Will there be a V10 for those of us with an AMD as it can only do SSE3 and not SSSE3

Thanks.

Running a Phenom 9950 BE
ID: 869415 · Report as offensive
Profile Adrian Taylor
Volunteer tester
Avatar

Send message
Joined: 22 Apr 01
Posts: 95
Credit: 10,933,449
RAC: 0
United Kingdom
Message 869421 - Posted: 25 Feb 2009, 20:08:34 UTC - in response to Message 869396.  

great, many thanks for the info Raistmer

im running v9 multi gpu now, and it seems to work as i want so far

regards

adrian
63. (1) (b) "music" includes sounds wholly or predominantly characterised by the emission of a succession of repetitive beats
ID: 869421 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 869520 - Posted: 26 Feb 2009, 1:15:46 UTC


@ Raistmer

O.K., you said that the CUDA performance is better > V7 than in V7.. ;-)

[only multi GPU crunching]

I could take this from your V10 mod:

MB_6.08_mod_CUDA_V10
number_of_GPUs
libfftw3f-3-1-1a_upx.dll
cudart.dll
cufft.dll

..change the app_info.xml from the V7 mod [the app name entry] and it would running well?

I don't want to crash the downloaded WUs.


Maybe I could take the 'old' .dll's from the V7 mod?
After reading other post that people had probs with the new .dll's.


Thanx!

ID: 869520 · Report as offensive
Profile Bob Mahoney Design
Avatar

Send message
Joined: 4 Apr 04
Posts: 178
Credit: 9,205,632
RAC: 0
United States
Message 869710 - Posted: 26 Feb 2009, 16:03:08 UTC
Last modified: 26 Feb 2009, 16:34:57 UTC

I'm running V10 and had the following situation:

Computer has 4-core CPU
Computer has 6 GPU
cc_config.xml in BOINC directory set to ncpus=10
number_of_GPUs in S@H project directory set to 6
S@H preferences set to MB and both AP

It ran perfectly when I was issued 4xAPv5.00 workunits. There were 6xCUDA MB running plus 4xAP on the CPU.

Then I was issued a few more APv5.00. Checked BOINC tasks screen and noticed 6 AP were running at once, and only 4 CUDA MB running, since this met the maximum ncpu setting of 10. Two of the AP were, obviously, running as virtual cores.

At this point there were 2 more AP tasks waiting to start. I aborted those two, then I aborted the two extra that were running in virtual cores. Then I turned off preferences so no more AP. Rebooted the computer and everything looked good again - 4xAP and 6xCUDA running together.

I have kept AP off since then, now it runs with 4xMB CPU and 6xMB CUDA.

I would like to run AP, though. This looks like a bug in core allocations.

Info you requested:

The host URL: http://setiathome.berkeley.edu/show_host_detail.php?hostid=4814209
Win XP64 service pack 2
NVIDIA (EVGA) production driver version 182.06
edit: Running BOINC v6.4.6

Thanks, Raistmer, you rule!

Bob Mahoney
ID: 869710 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 869742 - Posted: 26 Feb 2009, 17:48:00 UTC - in response to Message 869710.  

I'm running V10 and had the following situation:

Computer has 4-core CPU
Computer has 6 GPU
cc_config.xml in BOINC directory set to ncpus=10
number_of_GPUs in S@H project directory set to 6
S@H preferences set to MB and both AP

It ran perfectly when I was issued 4xAPv5.00 workunits. There were 6xCUDA MB running plus 4xAP on the CPU.

Then I was issued a few more APv5.00. Checked BOINC tasks screen and noticed 6 AP were running at once, and only 4 CUDA MB running, since this met the maximum ncpu setting of 10. Two of the AP were, obviously, running as virtual cores.

At this point there were 2 more AP tasks waiting to start. I aborted those two, then I aborted the two extra that were running in virtual cores. Then I turned off preferences so no more AP. Rebooted the computer and everything looked good again - 4xAP and 6xCUDA running together.

I have kept AP off since then, now it runs with 4xMB CPU and 6xMB CUDA.

I would like to run AP, though. This looks like a bug in core allocations.

Info you requested:

The host URL: http://setiathome.berkeley.edu/show_host_detail.php?hostid=4814209
Win XP64 service pack 2
NVIDIA (EVGA) production driver version 182.06
edit: Running BOINC v6.4.6

Thanks, Raistmer, you rule!

Bob Mahoney

http://setiathome.berkeley.edu/forum_thread.php?id=52212&nowrap=true#869404
ID: 869742 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 869744 - Posted: 26 Feb 2009, 17:49:46 UTC - in response to Message 869742.  
Last modified: 26 Feb 2009, 18:04:02 UTC

SSE3 x86 and SSE3 x86 AMD-specific packs added.

ADDON:

SSE3 x64 Intel-specific pack added
ID: 869744 · Report as offensive
Profile Bob Mahoney Design
Avatar

Send message
Joined: 4 Apr 04
Posts: 178
Credit: 9,205,632
RAC: 0
United States
Message 869751 - Posted: 26 Feb 2009, 18:17:52 UTC

Thanks for the response on core allocation.

Another issue I've observed:

While running with v7b, when a CUDA task completed and a new one was started the Task Manager would show 25% of CPU utilized for that task. During that "prep and load" CUDA period, all the other CUDA tasks would still show the usual 1-4% CPU load. Seemed perfect.

Now with V10 with multi-GPU, during the "prep and load" CUDA period for a new task, all the other CUDA task's (which are already running) CPU utilization goes to 0% for the duration of that load period. In my case that is 16 seconds of NO work from ALL GPU's during the load of one CUDA unit.

This is a performance degradation. Is it possible to modify the setup so one CUDA doesn't suspend CPU for all others during the load time?

Thanks,
Bob Mahoney
ID: 869751 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 869761 - Posted: 26 Feb 2009, 19:07:34 UTC - in response to Message 869751.  
Last modified: 26 Feb 2009, 19:08:21 UTC


This is a performance degradation. Is it possible to modify the setup so one CUDA doesn't suspend CPU for all others during the load time?

Thanks,
Bob Mahoney


Yes it's possible.
It's consequence of one of mods that set processor affinity.
That is, all CUDA processes locked on core 1.If one of them uses CPU more actively than usually then other processes may starve.

Thanks for noticing this.

Could you confirm that other GPUs become idle for this time duration indeed?
That is, is their temp decreases for this period of time?
Will elapsed time increase if some another GPU does task switch often?

I need this info cause disabling that mod will have back another performance issues it fights with. So I must be sure that it degrades performance in multi-GPU configs.
ID: 869761 · Report as offensive
Profile Bob Mahoney Design
Avatar

Send message
Joined: 4 Apr 04
Posts: 178
Credit: 9,205,632
RAC: 0
United States
Message 869787 - Posted: 26 Feb 2009, 21:19:04 UTC - in response to Message 869761.  

...
Could you confirm that other GPUs become idle for this time duration indeed?
That is, is their temp decreases for this period of time?
Will elapsed time increase if some another GPU does task switch often?

I need this info cause disabling that mod will have back another performance issues it fights with. So I must be sure that it degrades performance in multi-GPU configs.


Confirmed. During the load of any GPU task the following happens:

My trusty plug-in wattmeter drops by 300watts. (Normal drop for loading of one CUDA task was approximately 50watts with v7b.)

All GPU core temperatures decline.

"Progress" column in BOINC Manager slows to about 1/20th of normal progress on all GPU tasks.

"To Completion" column in BOINC Manager starts to increase instead of normal decrease on all GPU tasks, except for the one loading which is, of course, at 0% complete.

Windows Task Manager "Processes" screen, "CPU" column goes to zero for all GPU tasks except for the one loading, which is at 25%. CPU based processes (MB or AP) stay normal, getting their fair share of cycles, except for the expected loss of the 25% they are lending to the loading GPU.

I'm sorry about giving you more work. On the bright side, v10 has been perfectly stable on this system. It is an interesting approach.

Bob Mahoney
ID: 869787 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 870088 - Posted: 27 Feb 2009, 16:24:12 UTC - in response to Message 869787.  
Last modified: 27 Feb 2009, 16:31:53 UTC

...
Could you confirm that other GPUs become idle for this time duration indeed?
That is, is their temp decreases for this period of time?
Will elapsed time increase if some another GPU does task switch often?

I need this info cause disabling that mod will have back another performance issues it fights with. So I must be sure that it degrades performance in multi-GPU configs.


Confirmed. During the load of any GPU task the following happens:

My trusty plug-in wattmeter drops by 300watts. (Normal drop for loading of one CUDA task was approximately 50watts with v7b.)

All GPU core temperatures decline.

"Progress" column in BOINC Manager slows to about 1/20th of normal progress on all GPU tasks.

"To Completion" column in BOINC Manager starts to increase instead of normal decrease on all GPU tasks, except for the one loading which is, of course, at 0% complete.

Windows Task Manager "Processes" screen, "CPU" column goes to zero for all GPU tasks except for the one loading, which is at 25%. CPU based processes (MB or AP) stay normal, getting their fair share of cycles, except for the expected loss of the 25% they are lending to the loading GPU.

I'm sorry about giving you more work. On the bright side, v10 has been perfectly stable on this system. It is an interesting approach.

Bob Mahoney


Ok, thanks for such detailed inverstigation of problem, will undo that mod and post new CUDA executable soon.

ADDON: It will take some time cause now I need to recover dev host from unsuccessful x64 CUDA tools installation attempt. Their x64 toolset seems more like beta version unfortunately.
ID: 870088 · Report as offensive
Profile Bob Mahoney Design
Avatar

Send message
Joined: 4 Apr 04
Posts: 178
Credit: 9,205,632
RAC: 0
United States
Message 870579 - Posted: 28 Feb 2009, 20:36:18 UTC - in response to Message 870088.  

Ok, thanks for such detailed inverstigation of problem, will undo that mod and post new CUDA executable soon.

ADDON: It will take some time cause now I need to recover dev host from unsuccessful x64 CUDA tools installation attempt. Their x64 toolset seems more like beta version unfortunately.


I've studied the situation some more. If fixing this minor problem comes at a loss of stability, I would rather keep V10 working as it is. It is the most stable of the Team configurations, and the approach seems to give you excellent control of all processes. V10 provides the best CPU utilization for those tasks allocated to CPU processing, especially the new AP V5.

So this fix is not an emergency.

Bob Mahoney
ID: 870579 · Report as offensive
Profile RottenMutt
Avatar

Send message
Joined: 15 Mar 01
Posts: 1011
Credit: 230,314,058
RAC: 0
United States
Message 870743 - Posted: 1 Mar 2009, 3:32:27 UTC
Last modified: 1 Mar 2009, 3:33:49 UTC

aren't AP work units suppose to run on the GPU?

i'm running boinc 6.6.11 and V10a and can't get any of the AP units to run on the gpu. right now i have 9 AP work units running simultaneously.

all i see is nine occurrences of "ap_5.03r112_sse3.exe*32" in the task manager.

GPU temp is low.
ID: 870743 · Report as offensive
Profile Geek@Play
Volunteer tester
Avatar

Send message
Joined: 31 Jul 01
Posts: 2467
Credit: 86,146,931
RAC: 0
United States
Message 870747 - Posted: 1 Mar 2009, 3:39:33 UTC - in response to Message 870743.  

aren't AP work units suppose to run on the GPU?

i'm running boinc 6.6.11 and V10a and can't get any of the AP units to run on the gpu. right now i have 9 AP work units running simultaneously.

all i see is nine occurrences of "ap_5.03r112_sse3.exe*32" in the task manager.

GPU temp is low.


NO......Astropulse work always runs on the CPU's. ONLY Seti Enhanced work runs on the GPU.
Boinc....Boinc....Boinc....Boinc....
ID: 870747 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 870748 - Posted: 1 Mar 2009, 3:39:38 UTC - in response to Message 870743.  

aren't AP work units suppose to run on the GPU?
Nope. See my response at Lunatics for links to discussion of factors that would also be stopping MB work downloads, if you are using a development version of Boinc, as Multibeam Work fetch for app_info type installations seems not too happy at the moment either. (reverting to 6.5.0 fixed it for me)

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 870748 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 . . . 15 · Next

Message boards : Number crunching : V10 of modified SETI MB CUDA + opt AP package for full multi-GPU+CPU use


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.