Modified SETI MB CUDA + opt AP package for full GPU utilization

Message boards : Number crunching : Modified SETI MB CUDA + opt AP package for full GPU utilization
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 13 · 14 · 15 · 16 · 17 · 18 · 19 . . . 25 · Next

AuthorMessage
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 852315 - Posted: 11 Jan 2009, 18:07:06 UTC - in response to Message 852298.  

11.01.2009 18:36:03|SETI@home|Sending scheduler request: Requested by user. Requesting 0 seconds of work, reporting 0 completed tasks

You have enough work (from BOINC point of view) now. Don't forget, AstroPulse and MultiBeam - the same SETI project, so BOINC can't understand that AP uses CPU only and MB CUDA uses GPU only (unfortunately).
I process Einstein + SETI CUDA MB for that reason...
ID: 852315 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 852320 - Posted: 11 Jan 2009, 18:13:06 UTC - in response to Message 852298.  
Last modified: 11 Jan 2009, 18:18:12 UTC

*sigh*
Believe me, I tried that a dozen times. Here is the latest:
11.01.2009 18:36:03|SETI@home|Sending scheduler request: Requested by user. Requesting 0 seconds of work, reporting 0 completed tasks
11.01.2009 18:36:08|SETI@home|Scheduler request completed: got 0 new tasks


I don't get why Seti doesn't even request work when I update it manually. Other projects do that automatically when WUs are below the buffer. I set the buffer to 10 days to see if there is a change, but there isn't. :(


How much other unsuspended work do you have?, Could it be over 10 days worth?, Try suspending all projects bar Seti.
Do you have any Seti tasks suspended?, if you have even one suspended, Boinc won't ask for any Seti work.

Claggy

Edit: If you then just get AP work, try unselected AP in your preferences to force MB through.
ID: 852320 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 852351 - Posted: 11 Jan 2009, 18:48:45 UTC - in response to Message 852276.  
Last modified: 11 Jan 2009, 18:52:18 UTC

...
So, some deviation in run time is possible even w/o any changes in CUDA app (other loaded processes and DLLs can affect opt AP too, not only CUDA MB).

I noticed this when v5 opti came out. My first 4 v5 WUs had a large deviation in CPU time. Two were around 185,000 seconds, and the other two were around 250,000. All four were crunching at the same time.

There is preliminary exit for AP too (as result overflow -9 for SETI MB), maybe one of tasks had that overflow...

They went all the way through. Exit status was 0x0. My normal running time is around 170,000 seconds. I was just pointing out that I had two that took ~47% longer for no apparent reason.

Way too much time has passed to post info about the tasks or stderr's and such. I should have saved all of that though.

[edit]: Actually, one of them is still waiting. sorry, it was 225,000 not 250,000. still more time than it should be though. http://setiathome.berkeley.edu/workunit.php?wuid=376519487
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 852351 · Report as offensive
Profile Björn

Send message
Joined: 29 Jul 01
Posts: 22
Credit: 1,058,001
RAC: 0
Germany
Message 852391 - Posted: 11 Jan 2009, 19:49:06 UTC - in response to Message 852315.  

11.01.2009 18:36:03|SETI@home|Sending scheduler request: Requested by user. Requesting 0 seconds of work, reporting 0 completed tasks

You have enough work (from BOINC point of view) now. Don't forget, AstroPulse and MultiBeam - the same SETI project, so BOINC can't understand that AP uses CPU only and MB CUDA uses GPU only (unfortunately).
I process Einstein + SETI CUDA MB for that reason...


I even paused AstroPulse and still got nothing. I will let those AstroPulse units finish and see if the problems is still occuring.

*sigh*
Believe me, I tried that a dozen times. Here is the latest:
11.01.2009 18:36:03|SETI@home|Sending scheduler request: Requested by user. Requesting 0 seconds of work, reporting 0 completed tasks
11.01.2009 18:36:08|SETI@home|Scheduler request completed: got 0 new tasks


I don't get why Seti doesn't even request work when I update it manually. Other projects do that automatically when WUs are below the buffer. I set the buffer to 10 days to see if there is a change, but there isn't. :(


How much other unsuspended work do you have?, Could it be over 10 days worth?, Try suspending all projects bar Seti.
Do you have any Seti tasks suspended?, if you have even one suspended, Boinc won't ask for any Seti work.

Claggy

Edit: If you then just get AP work, try unselected AP in your preferences to force MB through.


The AstroPulse units should be worth 8 or 9 days, that's why I tried 10.

Right now I have one CPDN WU (suspended), 5 AstroPulse WU (running/waiting to run), 1 GPU GRID WU (running).
I had GPU GRID and the AP WU's suspended when requesting work though.

I'll keep you posted what happens when I run out of work.
ID: 852391 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 852397 - Posted: 11 Jan 2009, 20:02:42 UTC - in response to Message 852351.  


[edit]: Actually, one of them is still waiting. sorry, it was 225,000 not 250,000. still more time than it should be though. http://setiathome.berkeley.edu/workunit.php?wuid=376519487


Cache: L1=64K L2=1024K
Features: FPU TSC PAE CMPXCHG8B APIC SYSENTER MTRR CMOV/CCMP MMX FXSAVE/FXRSTOR SSE SSE2 HT SSE3
In ap_gfx_main.cpp: in ap_graphics_init(): Starting client.
### Restart at 27.04 percent.
In ap_gfx_main.cpp: in ap_graphics_init(): Starting client.
### Restart at 71.08 percent.

There were 2 restarts. Maybe they were in some hard place and more than usual computations were lost... don't know.. Try to keep suspended tasks in memory option enabled (if your host's memory amount allow it of course).
ID: 852397 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 852398 - Posted: 11 Jan 2009, 20:04:38 UTC - in response to Message 852391.  

I'll keep you posted what happens when I run out of work.

OK. BTW, when you suspend AP task you will not get any SETI work, not more AP nor MB tasks. BOINC doesn't download work for projects with suspended tasks (as was mentioned earlier).

ID: 852398 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 852402 - Posted: 11 Jan 2009, 20:08:39 UTC - in response to Message 852397.  


[edit]: Actually, one of them is still waiting. sorry, it was 225,000 not 250,000. still more time than it should be though. http://setiathome.berkeley.edu/workunit.php?wuid=376519487


Cache: L1=64K L2=1024K
Features: FPU TSC PAE CMPXCHG8B APIC SYSENTER MTRR CMOV/CCMP MMX FXSAVE/FXRSTOR SSE SSE2 HT SSE3
In ap_gfx_main.cpp: in ap_graphics_init(): Starting client.
### Restart at 27.04 percent.
In ap_gfx_main.cpp: in ap_graphics_init(): Starting client.
### Restart at 71.08 percent.

There were 2 restarts. Maybe they were in some hard place and more than usual computations were lost... don't know.. Try to keep suspended tasks in memory option enabled (if your host's memory amount allow it of course).

4gb of RAM, and I have the 'leave in memory' setting set. It always has been. I was thinking it might be along the lines of something OzzFan mentioned that Eric said, which was when BOINC decides to suspend a task for high-priority mode, it does the 'no mercy' rule and doesn't let a task make it to a checkpoint or finish the I/O it is doing. That scenario usually gives the statefile error though.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 852402 · Report as offensive
Profile Björn

Send message
Joined: 29 Jul 01
Posts: 22
Credit: 1,058,001
RAC: 0
Germany
Message 852416 - Posted: 11 Jan 2009, 20:27:02 UTC - in response to Message 852398.  

I'll keep you posted what happens when I run out of work.

OK. BTW, when you suspend AP task you will not get any SETI work, not more AP nor MB tasks. BOINC doesn't download work for projects with suspended tasks (as was mentioned earlier).



LOL. After I read your lines again just now, I looked over the AP WU's again and one of them was still suspended.. D'oh!
Sorry for making such a big deal out of it and thanks for all your help, guys!
ID: 852416 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 852418 - Posted: 11 Jan 2009, 20:27:16 UTC - in response to Message 852402.  
Last modified: 11 Jan 2009, 20:28:03 UTC

That wasn't me, that was either Joe or Richard. (Not upset, just want to give credit where credit is due.)
ID: 852418 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 852441 - Posted: 11 Jan 2009, 21:06:30 UTC - in response to Message 852418.  
Last modified: 11 Jan 2009, 21:11:31 UTC

That wasn't me, that was either Joe or Richard. (Not upset, just want to give credit where credit is due.)

Hm...I think you're right..I'm leaning toward Joe. I'll try to find it.

Edit: Found it. It was Richard. :D http://setiathome.berkeley.edu/forum_thread.php?id=51244 Credit is now granted to the proper recipient.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 852441 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14655
Credit: 200,643,578
RAC: 874
United Kingdom
Message 852443 - Posted: 11 Jan 2009, 21:14:48 UTC - in response to Message 852441.  
Last modified: 11 Jan 2009, 21:17:10 UTC

message 850858

Any 0'd statefile wouldn't, AFAIK, be displayed in the std_err txt of an optimised app: but it shouldn't be zeroed in the first place, because of protective code the optimisers put in to work round 'terminate with no mercy'.

Edit] Takes a small bow. You found it just as I was volunteering to put myself forward into the spotlight.
ID: 852443 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 852509 - Posted: 12 Jan 2009, 0:52:11 UTC - in response to Message 852443.  

message 850858

Any 0'd statefile wouldn't, AFAIK, be displayed in the std_err txt of an optimised app: but it shouldn't be zeroed in the first place, because of protective code the optimisers put in to work round 'terminate with no mercy'.
...

Well, there's task 1111524549, discussed 3 days ago in this thread, which is an exception. That's the first one I've seen since Jason added the extra file flush actions, though.
                                                              Joe
ID: 852509 · Report as offensive
Profile Woyteck - Boinc Busters Poland
Avatar

Send message
Joined: 3 Jun 99
Posts: 49
Credit: 3,203,845
RAC: 0
Poland
Message 852526 - Posted: 12 Jan 2009, 1:29:07 UTC

Noticed huge disproportion in 'to completion' time for Seti and Astropulse now when I crunch Seti with CUDA.
Initial to completion time for Astropulse jumped to 3300 hours! (1 WU is done within about 40h on my machine).
Also now Seti reports it to be about 7h, bit finished the unit in 5-7 minutes...

Any chances to fix the estimate?

--
Get up, stand up! Don\'t give up the fight!
Credits will make everybody feel high! ;-)
ID: 852526 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 852602 - Posted: 12 Jan 2009, 9:12:17 UTC - in response to Message 852526.  

Noticed huge disproportion in 'to completion' time for Seti and Astropulse now when I crunch Seti with CUDA.
Initial to completion time for Astropulse jumped to 3300 hours! (1 WU is done within about 40h on my machine).
Also now Seti reports it to be about 7h, bit finished the unit in 5-7 minutes...

Any chances to fix the estimate?

1) It's BOINC question so much better to post such questions in BOINC forums or BOINC-related threads - that's why different forums and threads exist.
2) You should manually estimate real time to completion and edit so called DCF (duration correction factor) value in BOINC options file. For more detailed info do search for DCF value - there is plenty of info on forums, just not in CUDA MB related thread.
ID: 852602 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 852847 - Posted: 13 Jan 2009, 1:58:47 UTC - in response to Message 852602.  
Last modified: 13 Jan 2009, 2:17:20 UTC

Ok, new bug fxes available - so new modded app available too :)
VLAR bug still with us although most advanced GPUs (fastest ones) could be free from it already - worth to test. Midrange like my own 9600GSO still suffer from it, especially under Vista.
Again, it's worth to try under XP - there are chances it will work under XP better.

Current bug fixes fight mostly with different overflows. Actually, they should eliminate overflows at all. So, please, report any overflow you will get if it not from VLAR and not from task was ran after driver crash w/o OS reboot.

Where to download Raistmer's_opt_package_V5.rar :
http://files.mail.ru/APJBOK
and Lunatics http://lunatics.kwsn.net/gpu-crunching/modified-seti-mb-cuda-opt-ap-package-for-full-gpu-utilize.msg12177.html#msg12177 as usual.
ID: 852847 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 852867 - Posted: 13 Jan 2009, 2:29:33 UTC - in response to Message 852847.  

Raistmer, I'm waiting for a couple of APs to finish then I'm going to detach and start all over. Is there anything else I will need to get this running right now?
I will grab your package and the one from Lunatics tomorrow and anything else you might suggest.


PROUD MEMBER OF Team Starfire World BOINC
ID: 852867 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 852868 - Posted: 13 Jan 2009, 2:36:27 UTC - in response to Message 852867.  
Last modified: 13 Jan 2009, 2:44:09 UTC

Raistmer, I'm waiting for a couple of APs to finish then I'm going to detach and start all over. Is there anything else I will need to get this running right now?
I will grab your package and the one from Lunatics tomorrow and anything else you might suggest.


You need only this package. I just post mirror for those who can't access to Lunatics site.
This build should be equivalent to 6.07 stock beta available now in all aspects besides performance. But as usually, if this true or not life will show...

ADDON: Wow, and performance of current 6.07 stock maybe even better.
Will do more benchmarking, but for now it works under fully loaded CPU at least not worse than my mod. If it will keep such timings for other ARs... very good :)
ID: 852868 · Report as offensive
Profile Adrian Taylor
Volunteer tester
Avatar

Send message
Joined: 22 Apr 01
Posts: 95
Credit: 10,933,449
RAC: 0
United Kingdom
Message 852948 - Posted: 13 Jan 2009, 6:29:17 UTC

been running v4 of this on my mac pro 8800 GT in windows 7 beta, and all was well

just updated to opt_package_V5.rar and allthe cuda units errored with the following:

<core_client_version>6.5.0</core_client_version>
<![CDATA[
<message>
Incorrect function. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
setiathome_CUDA: Found 1 CUDA device(s):
Device 1 : GeForce 8800 GT
totalGlobalMem = 536870912
sharedMemPerBlock = 16384
regsPerBlock = 8192
warpSize = 32
memPitch = 262144
maxThreadsPerBlock = 512
clockRate = 1500000
totalConstMem = 65536
major = 1
minor = 1
textureAlignment = 256
deviceOverlap = 0
multiProcessorCount = 14
setiathome_CUDA: CUDA Device 1 specified, checking...
Device 1: GeForce 8800 GT is okay
SETI@home using CUDA accelerated device GeForce 8800 GT
Rise priority modification by Raistmer based on rev400 of SETI@home sources
Priority of worker thread rised successfully
Total GPU memory 536870912 free GPU memory 492306432
setiathome_enhanced 6.02 Visual Studio/Microsoft C++
libboinc: 6.3.22

Work Unit Info:
...............
WU true angle range is : 2.430996
Optimal function choices:
-----------------------------------------------------
name
-----------------------------------------------------
v_BaseLineSmooth (no other)
v_GetPowerSpectrum 0.00018 0.00000
v_ChirpData 0.01625 0.00000
v_Transpose4 0.01086 0.00000
FPU opt folding 0.00221 0.00000
Cuda error 'cudaMemcpyToSymbol(cudaAcc_GaussFit_settings, (void*) &settings, sizeof(settings))' in file 'd:/BTR/seticuda/seti_boinc/client/cuda/cudaAcc_gaussfit.cu' in line 454 : invalid device symbol.

</stderr_txt>
]]>



went back to V4 and all is well again, hope this helps ;-)

thanks for the efforts Raistmer (and of course all the others in this continuing saga)

regards

adrian
63. (1) (b) "music" includes sounds wholly or predominantly characterised by the emission of a succession of repetitive beats
ID: 852948 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 852995 - Posted: 13 Jan 2009, 11:23:08 UTC - in response to Message 852948.  

Update driver, it's app-driver interaction problem.
ID: 852995 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 853072 - Posted: 13 Jan 2009, 16:33:47 UTC

Ok, just an update. I detached from project uninstalled my BOINC and started fresh. Downloaded new BOINC Manager, reattached to SETI,waited until all new stuff had finished downloading, and stopped BOINC manager, installed your newest app and restarted.
Wish me luck! :)


PROUD MEMBER OF Team Starfire World BOINC
ID: 853072 · Report as offensive
Previous · 1 . . . 13 · 14 · 15 · 16 · 17 · 18 · 19 . . . 25 · Next

Message boards : Number crunching : Modified SETI MB CUDA + opt AP package for full GPU utilization


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.