[LINUX] A Faster SETI app to go with my optimized boinc apps

Message boards : Number crunching : [LINUX] A Faster SETI app to go with my optimized boinc apps
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · Next

AuthorMessage
Metod, S56RKO
Volunteer tester

Send message
Joined: 27 Sep 02
Posts: 309
Credit: 113,221,277
RAC: 9
Slovenia
Message 104020 - Posted: 26 Apr 2005, 11:26:23 UTC - in response to Message 104017.  
Last modified: 26 Apr 2005, 11:27:30 UTC

> Thanks..but I still cannot get it to work...same problem...if you have spare
> time, will you test the executable seti_boinc (3947291 bytes)?

I get exactly the same error as you are.

Again: try to re-do the last step of compilation ... like this:

cd client
rm seti_boinc
make seti_boinc

This kind of hokus-pokus helped when I got erroneous executable.

BTW, you might find out that the executable is note entirely static out-of-the-box. Mine wasn't at least. You will have to adapt the linking command and run it manually in this case.

> Intel C++ compiler takes time, but it seems to optimize a LOT...(or it is just
> complaining many things?)

Yup, intel CC is a chatter box. Likes to inform you about every optimization it does. But then again, there are many as resulting binaries proove.

Metod ...
ID: 104020 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 104024 - Posted: 26 Apr 2005, 11:54:46 UTC - in response to Message 104020.  
Last modified: 26 Apr 2005, 12:14:23 UTC

> > Thanks..but I still cannot get it to work...same problem...if you have
> spare
> > time, will you test the executable seti_boinc (3947291 bytes)?
>
> I get exactly the same error as you are.
>
> Again: try to re-do the last step of compilation ... like this:
>
> cd client
> rm seti_boinc
> make seti_boinc
>
> This kind of hokus-pokus helped when I got erroneous executable.
>
> BTW, you might find out that the executable is note entirely static
> out-of-the-box. Mine wasn't at least. You will have to adapt the linking
> command and run it manually in this case.
>
Thanks for you suggestion, but it doesn't work for me...I wonder how you pass "-lfftw3f" option to icpc? I put it in CFLAGS environment variable at first.

I cannot believe these happen at the same time.
-----------
$ ./seti_boinc
bash: ./seti_boinc: No such file or directory
$ file ./seti_boinc
./seti_boinc: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.0, dynamically linked (uses shared libs), not stripped
------------

Is it a fault of Debian? (I'm using debian sarge)

btw I finished my gcc-based fftw seti@home v.4.07 benchmark now, and the result was 12728 secs (official 4.02 took 15915secs) so about 79%. Without fftw, my seti@home took 13415secs, so it doesn't make a big difference. However, Chris' executable takes only 8538 secs on my machine....is this the difference of gnu and Intel?...incredible. I suspect I should have made some mistakes somewhere...

I am tired and ready to abandon :)
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 104024 · Report as offensive
Metod, S56RKO
Volunteer tester

Send message
Joined: 27 Sep 02
Posts: 309
Credit: 113,221,277
RAC: 9
Slovenia
Message 104027 - Posted: 26 Apr 2005, 12:19:26 UTC - in response to Message 104024.  


> export CFLAGS='-fast -mcpu=pentium4 -march=pentium4 -ipo -ipo_obj -lgtk -lgdk
> -lgmodule -lgthread -lglib -lX11 -lXext -lXi -lfftw3f'

The libraries should not be in CFLAGS variable. configure script should add them automatically if it finds all it needs for certain feature.

As for FFTW: I had to use CPPFLAGS="-I/usr/local/fftw/include" and LDFLAGS="-L/usr/local/fftw/lib" as I installed fftw3 libraries to /usr/local/fftw tree. Other than that, I had to change the way configure script checks for decent (working) FFTW library. I guess the check is against an older version of FFTW and tries to use function not available in FFTW3 library.

> Is it a fault of Debian? (I'm using debian sarge)

Me too.

> is this the difference of gnu and Intel?

Yes and no. Ned's i686 binary (generic ia32 executable) runs only slightly slowe r than my generic ia32 binary (Intel binary si some 5.8% faster than GCC binary). The biggest difference is optimization for particular CPU type (eg. for P4). Thius is where Intel CC does good job comparing to GCC.
Metod ...
ID: 104027 · Report as offensive
Ned Slider

Send message
Joined: 12 Oct 01
Posts: 668
Credit: 4,375,315
RAC: 0
United Kingdom
Message 104035 - Posted: 26 Apr 2005, 13:16:06 UTC - in response to Message 104027.  
Last modified: 26 Apr 2005, 13:17:40 UTC

>
> > is this the difference of gnu and Intel?
>
> Yes and no. Ned's i686 binary (generic ia32 executable) runs only slightly
> slowe r than my generic ia32 binary (Intel binary si some 5.8% faster than GCC
> binary). The biggest difference is optimization for particular CPU type (eg.
> for P4). Thius is where Intel CC does good job comparing to GCC.
>

Agreed. I see relatively little additional performance increase for specifically compiling for a specific processor architecture over i686 using gcc, at least for AMD processors (I've not tried specifically for Intel processors as it probably wouldn't beat Metod's clients). The main benefits seem to come from -ffast-math and using the fftw3 libs for the gcc compiler - everything else is relatively minor in comparison.

Ned

*** My Guide to Compiling Optimised BOINC and SETI Clients ***
*** Download Optimised BOINC and SETI Clients for Linux Here ***
ID: 104035 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 104042 - Posted: 26 Apr 2005, 13:37:44 UTC

Thanks all....now I'm trying to build in Fedora Core3 environment running in vmware. I'm very sorry not to use debian for compilation, but unfortunately install of Intel Compiler is smoother on Fedora (maybe on any rpm based distros.) I'm a debian maniac though.
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 104042 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 104229 - Posted: 26 Apr 2005, 21:55:07 UTC

FINALLY I DID IT!!

MY BAD!!!!

I found why I couldn't make it. I needed to make static-linked binary!!! Where is it mentioned?? Actually Chris' page says executables are statically linked, but AFAIK nobody says it must be statically linked...and I don't like statically linked binary because in classic SETI@home, statically linked binary was very slow.

Debian is GREAT! (yes, I admit Fedora is also nice, but personally I prefer Debian. just a personal preference)

Now I'm benchmark-testing!!

Anyway thanks all!!!!!!!!!!!!!!!!

why doesn't intel compiler have enough capability of dynamically linked executables? Is it mentioned somewhere?

BTW the reason why I found my fault was the error message of gdb.
----------gdb session-----------
gdb> r
zsh: no such file or directory: /home/boinc/build/build/seti_boinc-2005-04-12-intel/client/seti_boinc

Program exited with code 0177.
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
You can't do that without a process to debug.
----------------------------------
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 104229 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 104445 - Posted: 27 Apr 2005, 11:21:43 UTC - in response to Message 104027.  


> > Is it a fault of Debian? (I'm using debian sarge)
>
> Me too.
>
> > is this the difference of gnu and Intel?
>
> Yes and no. Ned's i686 binary (generic ia32 executable) runs only slightly
> slowe r than my generic ia32 binary (Intel binary si some 5.8% faster than GCC
> binary). The biggest difference is optimization for particular CPU type (eg.
> for P4). Thius is where Intel CC does good job comparing to GCC.
>

hmmm I finished benchmark of my Intel based SETI@home, and found mine was clearly slower than your P4 only version (setiathome-4.07-fftw.i686-p4-linux-gnu). I wonder why mine is slower..same distro (debian) so the same or similar static libraries....and as a matter of course the same compile options.

For ref, it took 9216 secs for my SETI to crunch the reference workunit while yours took 8538 secs. I linked libc6-2.3.2.

Now I'm trying to compile on Fedora and compare :)
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 104445 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 104490 - Posted: 27 Apr 2005, 14:44:42 UTC
Last modified: 27 Apr 2005, 14:59:11 UTC

Benchmark of my 2nd Intel based-SETI@home completed on my Pentium4 (HT). So far results are as follows:

15915 secs official 4.02 (100%)
8538 secs Metod's setiathome-4.07-fftw.i686-p4-linux-gnu (54%)
9216 secs my first Intel 4.07 with the same options as Metod's
9794 secs my 2nd Intel 4.07 with the same options (linked with i686 optimized glibc static libc.a/libm.a)

First of all, I have made a big mistake...it's Metod's binary that I've been using and compared as "Ned's binary"...sorry..I thought your homepage (http://mkx.feridot.com/boinc/) was Ned's..please write your name on it :) I removed "/boinc/" from the url and found it's yours. But at first, I didn't notice it because it's not in English.. Or I may need to check Metod's binary again.

BTW the result.sah's look fine in all binaries.

And I don't know what makes the difference between Metod's binary and mine. It may be due to the build date (Metod used May 31, vs. mine is Apr 25...but does it make so much difference? Especially mine got worse with optimized libraries. It may be due to the way of my benchmark. I run SETI@home with other applications in various circumstances. Another candidate binary is my Intel based binary compiled on i686 optimized Fedore Core3.

Now I feel I'm SOOOOO maniac. lol
ID: 104490 · Report as offensive
Metod, S56RKO
Volunteer tester

Send message
Joined: 27 Sep 02
Posts: 309
Credit: 113,221,277
RAC: 9
Slovenia
Message 104508 - Posted: 27 Apr 2005, 15:13:36 UTC - in response to Message 104490.  

Intel CC can produce perfectly usable dynamic-linked binaries. What I found out is that seti-boinc configure script doesn't handle icc very well ... sometimes creates a bit funny Makefile.

> And I don't know what makes the difference between Metod's binary and mine.
> It may be due to the build date (Metod used May 31, vs. mine is Apr 25...but
> does it make so much difference?

I had to replace a couple of source files with older ones. As far as I remember binary from all-origina sources ran a bit slower than this one. But: the result would definitely fail validation ... see the beginning of this thread.

Another thing to consider during benchmark tests is any other activity on the machine. Especially if you're running on semi-idle HT machine.

Metod ...
ID: 104508 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 104551 - Posted: 27 Apr 2005, 17:10:17 UTC - in response to Message 104508.  
Last modified: 27 Apr 2005, 17:27:32 UTC

> I had to replace a couple of source files with older ones. As far as I
> remember binary from all-origina sources ran a bit slower than this one. But:
> the result would definitely fail validation ... see the beginning of <a> href="http://setiweb.ssl.berkeley.edu/forum_thread.php?id=13017">this
> thread[/url].
>

okay I wanted to know you have modified the source files. Then it makes sense (why your binary is faster than mine although compiling options are the same), and so now I don't have to run your binary again for benchmark :) Yes, I know about validation because I'd read Ned's page. Funny thing is my GNU compiled binary w/o fftw made more errors (number of errors as well as the values of errors: especially around line 490 and 508) than Intel compiled binary without any modifications in source files except for fftw related modifications. I don't know it's because of the compiler or fftw, but I don't mind. (maybe it's thanks to fftw..because of lower precision) I'm satisfied with Intel's compiler with fftw...since the original signal is 8bit data, single float calculation should be enough.

I just didn't know -fast option didn't make statically linked seti_boinc on my machine. I have to put -static explicitly. At first in order to make sure icpc worked correctly I made a very tiny "Hello world" program as usual and confirmed it made a dynamically-linked working binary. So until yesterday I believed my seti_boinc had to work even if it's dynamically linked...otherwise -fast option had to imply -static also and work properly (as the manual says)...actually boinc core client and boincmgr were linked statically w/o -static option and now working correctly.

> Another thing to consider during benchmark tests is any other activity on the
> machine. Especially if you're running on semi-idle HT machine.
>
Yes, it will affect much in my machine. I'm afraid my testing environment is not so accurate. I suspect it should have made the reverse results in my Intel compiled binaries.

BTW I have sent you email, but everything has been solved now. Thanks!
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 104551 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 104783 - Posted: 28 Apr 2005, 6:35:40 UTC
Last modified: 28 Apr 2005, 6:46:34 UTC

Finally I tried a one-to-one benchmark (because my box is a HT box) in a "silent" condition to compare Metod's binary with my Mar-31 binary to see how slow my compilation was....the result was

mine 9859 secs vs. Metod's 9813 secs...46 seconds' difference is just an error!!

Lesson: I found the benchmark test must be done with extreme care.
Lesson 2: The crunching time depends much on other conditions.
Lesson 3: The crunching time looks shorter when other applications are busy at the same time. This is just my impression (not confident) and apply only to SETI@home client... core client's benchmark gives lower scores when other processes are going on.

Thanks all!!!!!!! especially Metod!

Mav.
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 104783 · Report as offensive
Profile spacemeat
Avatar

Send message
Joined: 4 Oct 99
Posts: 239
Credit: 8,425,288
RAC: 0
United States
Message 104886 - Posted: 28 Apr 2005, 15:49:14 UTC

Ned, you have patched seti source, what did you change? im trying to compile a fftw3 sparc64 client and it fails with what looks like syntax errors that this platform doesn't like. if i can get this working and its has similar improvements, i can work on a static linux build.
ID: 104886 · Report as offensive
Ned Slider

Send message
Joined: 12 Oct 01
Posts: 668
Credit: 4,375,315
RAC: 0
United Kingdom
Message 104966 - Posted: 29 Apr 2005, 0:03:26 UTC - in response to Message 104886.  

> Ned, you have patched seti source, what did you change? im trying to compile
> a fftw3 sparc64 client and it fails with what looks like syntax errors that
> this platform doesn't like. if i can get this working and its has similar
> improvements, i can work on a static linux build.
>

3 things:

First, get the fftw3 seti source patches from sorceforge and extract the 6 files in place of the regular seti source files.

Next, in your new seti_boinc/client/seti.cpp file from the above sources, near the start of the seti_do_work() function, comment out the line free_a(analysis_state.data).

Finally, in makefile.in (or the resulting makefile after you've run ./configure), you need to add the link for the fftw3 lib as so:

-L. \
-looura \
-lfftw3f

These changes were made as originally suggested by Paolo on SorceForge here:

https://sourceforge.net/forum/forum.php?thread_id=1256083&forum_id=422289

All I've done is make these changes to the seti source (1 Dec 2004) and repackage in the hope to make it easier for others to use :)

Once you get a working client, let me know if you need help making it static :)

Ned

*** My Guide to Compiling Optimised BOINC and SETI Clients ***
*** Download Optimised BOINC and SETI Clients for Linux Here ***
ID: 104966 · Report as offensive
Profile spacemeat
Avatar

Send message
Joined: 4 Oct 99
Posts: 239
Credit: 8,425,288
RAC: 0
United States
Message 105012 - Posted: 29 Apr 2005, 2:04:38 UTC - in response to Message 104966.  


> These changes were made as originally suggested by Paolo on SorceForge here:
>
> https://sourceforge.net/forum/forum.php?thread_id=1256083&forum_id=422289
>
> All I've done is make these changes to the seti source (1 Dec 2004) and
> repackage in the hope to make it easier for others to use :)
>
> Once you get a working client, let me know if you need help making it static
> :)
>
> Ned
>
>

thanks, im going through compile now. im not going to bother benchmarking before/after as it would take forever to do tweaks and testing on 333MHz. it seems well proven that the most significant gains are from FFTW and -ffast-math.
ID: 105012 · Report as offensive
Ned Slider

Send message
Joined: 12 Oct 01
Posts: 668
Credit: 4,375,315
RAC: 0
United Kingdom
Message 105017 - Posted: 29 Apr 2005, 2:28:59 UTC - in response to Message 105012.  

>
> > These changes were made as originally suggested by Paolo on SorceForge
> here:
> >
> >
> https://sourceforge.net/forum/forum.php?thread_id=1256083&forum_id=422289
> >
> > All I've done is make these changes to the seti source (1 Dec 2004) and
> > repackage in the hope to make it easier for others to use :)
> >
> > Once you get a working client, let me know if you need help making it
> static
> > :)
> >
> > Ned
> >
> >
>
> thanks, im going through compile now. im not going to bother benchmarking
> before/after as it would take forever to do tweaks and testing on 333MHz. it
> seems well proven that the most significant gains are from FFTW and
> -ffast-math.

Yes, and I'd just throw in your other standard optimizations for that system. We tested LOTS of different options and 80-90% of the improvements we saw came from the above.

Good luck - I'm out of town for a few days, but look forward to seeing how you've got on when I get back :)

Ned


>
*** My Guide to Compiling Optimised BOINC and SETI Clients ***
*** Download Optimised BOINC and SETI Clients for Linux Here ***
ID: 105017 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 105095 - Posted: 29 Apr 2005, 7:11:59 UTC
Last modified: 29 Apr 2005, 7:18:08 UTC

I found I was so maniac that I've added other options to make it faster....

I added "-Zp4 -pc32" options (as a whole CFLAGS='-fast -static -mcpu=pentium4 -march=pentium4 -ipo -ipo_obj -axP -pc32 -Zp4 ' including redundancy). As a matter of course, result.sah is within validation limits.

It got 3% faster.....and I found another thing; in a benchmark test, if other applications are running, seti@home client finished faster (exactly speaking, it indicates "shorter time.") This time, it's almost completely silent condition, and got 10625secs vs. 10954secs. in one-to-one benchmark test. I found no more useful options.

regards...
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 105095 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 105334 - Posted: 29 Apr 2005, 14:08:05 UTC
Last modified: 29 Apr 2005, 15:04:32 UTC

nothing :) I will be testing/using my client..
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 105334 · Report as offensive
Profile spacemeat
Avatar

Send message
Joined: 4 Oct 99
Posts: 239
Credit: 8,425,288
RAC: 0
United States
Message 105357 - Posted: 29 Apr 2005, 15:10:59 UTC - in response to Message 105095.  

>.....and I found another thing; in a benchmark test, if other
> applications are running, seti@home client finished faster (exactly speaking,
> it indicates "shorter time.")

that's interesting...
ID: 105357 · Report as offensive
Tetsuji Maverick Rai
Volunteer tester
Avatar

Send message
Joined: 25 Apr 99
Posts: 518
Credit: 90,863
RAC: 0
Japan
Message 105363 - Posted: 29 Apr 2005, 15:31:48 UTC - in response to Message 105357.  
Last modified: 29 Apr 2005, 15:42:37 UTC

> >.....and I found another thing; in a benchmark test, if other
> > applications are running, seti@home client finished faster (exactly
> speaking,
> > it indicates "shorter time.")
>
> that's interesting...
>

yes, it happened with classic SETI@home also. If you run extra SETI@home clients or any other heavy load application at the same time when you run a benchmark test, SETI@home returns smaller processing time at last...

As for Netod's client, it recorded the crunching time for the reference workunit as 8538 secs on my box with other seti@home clients running, but with only another SETI@home client (because my box is HT enabled, I wanted to compare it with my client) it's 9813 secs. It's much bigger than I had expected.

I put my client a bit faster than Metod's, which crunches the reference work unit about 53% of the time the official 4.02 takes...It's tar+gzipped, about 950kb.

setimav.tar.gz

Exclusively for Pentium4 on Linux. Statically linked. I am happy if anyone try benchmark test with this :)
Luckiest in the world. WMD = Weapon of Mass Distraction.
Click this table.
ID: 105363 · Report as offensive
Profile MrMaxx
Avatar

Send message
Joined: 22 Apr 99
Posts: 135
Credit: 1,645,913
RAC: 1
United States
Message 105370 - Posted: 29 Apr 2005, 15:37:44 UTC

Just wondering if I should be using the Athlon-XP clients or just the standard i686 clients? I've got an "AuthenticAMD Athlon", not an Athlon-XP. Will it make any difference if I use AthlonXP on a NON-XP CPU?
Thanks
ID: 105370 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · Next

Message boards : Number crunching : [LINUX] A Faster SETI app to go with my optimized boinc apps


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.