Are Benchmarks Accurate?

Message boards : Number crunching : Are Benchmarks Accurate?
Message board moderation

To post messages, you must log in.

AuthorMessage
Mark Stevens

Send message
Joined: 15 Feb 01
Posts: 28
Credit: 177,705
RAC: 0
United Kingdom
Message 3715 - Posted: 3 Jul 2004, 16:49:34 UTC

Having just started on boinc I was interested to see the benchmarks on my PCs. I was puzzled though by the integer MIPs though which didn't seem to bear much relationship to the cpu speed.

For example, I have 2 Dell Precisions 450. One with twin 3.06GHz Xeons and the other with twin 2.66GHz Xeons. Both are running Windows XP Pro SP1.

Their respective benchmarks were:
2 x 3.06Ghz fp MIPs: 1964
int MIPs: 2379
2 x 2.66GHz fp MIPs: 1719
int MIPs: 3517

As you can see the slower PC has integer MIPs which are 50% higher which doesn't make much sense?! Both were doing anything else at the time as far as I was aware. Anyhow, I ran them again to make sure. Nothing else was running for sure this time.

The floating point MIPs were about the same but the integer ones varied again. Not as much as before but still different. As follows:

2 x 3.06Ghz fp MIPs: 1983
int MIPs: 2632
2 x 2.66GHz fp MIPs: 1727
int MIPs: 2904 (expected 2292 based on above)

Does anyone know what's causing this difference?

Thanks
Mark
ID: 3715 · Report as offensive
Heffed
Volunteer tester

Send message
Joined: 19 Mar 02
Posts: 1856
Credit: 40,736
RAC: 0
United States
Message 3754 - Posted: 3 Jul 2004, 20:02:20 UTC
Last modified: 3 Jul 2004, 20:03:40 UTC

Benchmarks are still a bit iffy. But they are much better than they have been. ;-)
ID: 3754 · Report as offensive
Planet Cox

Send message
Joined: 20 Aug 02
Posts: 4
Credit: 31,712
RAC: 0
United Kingdom
Message 3755 - Posted: 3 Jul 2004, 20:02:52 UTC

Well I don't know but I have been looking up some other people's PC numbers and I accept that there may be differences in mother board specs and all but it seems odd to me that some running near enough the same processor have very different ratings. Prompted me to download some benchmark software. This PC i'm on has a gig of DDR 400 ram and all but it (boinc) still says its much slower then a PC with a processor speed of only 100Mhz more. The other PC running boinc here crawls along and that has ratings 1/4 of this and seems to take 4 times longer so maybe they are sort of correct. I might try running it in Linux on the same PC and see if it makes any difference.
ID: 3755 · Report as offensive
Guido_A_Waldenmeier_

Send message
Joined: 3 Apr 99
Posts: 482
Credit: 4,774
RAC: 0
Liechtenstein
Message 3761 - Posted: 3 Jul 2004, 20:10:33 UTC

AMD64 3000+ at 2100 mhz winxp 410 mhz ram
i tune up it a little bit ;-) its for me fast more speed not need
2004-07-03 22:06:53 - 2633 double precision MIPS (Whetstone) per CPU
2004-07-03 22:06:53 - 5168 integer MIPS (Dhrystone) per CPU
[/url] [/url]
Bei der Eroberung des Weltraums sind zwei Probleme zu lösen: die Schwerkraft und der Papierkrieg. Mit der Schwerkraft wären wir fertig geworden.Wernher von Braun
ID: 3761 · Report as offensive
Profile Thierry Van Driessche
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3083
Credit: 150,096
RAC: 0
Belgium
Message 3776 - Posted: 3 Jul 2004, 20:39:17 UTC

The results of benchmark are depending from different factors, a.o.
1. the OS,
2. the running processes,
3. the available RAM.

Here is only one example of result of a benchmark, using SiSoftware Sandra Professional version 2004.2.9.104 on a PC running at 2.88GHz:

With all processes running
Dhrystone ALU : 8587MIPS
Whetstone FPU : 3517MFLOPS
Whetstone iSSE2 : 6207MFLOPS

Unnecessary processes killed and after freeing RAM
Dhrystone ALU : 8818MIPS
Whetstone FPU : 3597MFLOPS
Whetstone iSSE2 : 6363MFLOPS
ID: 3776 · Report as offensive
Mark Stevens

Send message
Joined: 15 Feb 01
Posts: 28
Credit: 177,705
RAC: 0
United Kingdom
Message 4196 - Posted: 4 Jul 2004, 22:54:15 UTC

Yes, I forgot I had Sandra (v1). The results from that are more consistant and in the correct proportion.

Running the boinc benchmark again the fp MIPs are always about the same but the integer ones vary wildly (from 2379-3697). Seems the boinc benchmark is a little off for some reason?!

They mention on the description of scoring that bionc uses the benchmark results. Does this mean my scores (I've yet to get any) will be off?
ID: 4196 · Report as offensive
D. Gustibus

Send message
Joined: 3 Jan 04
Posts: 9
Credit: 242,311
RAC: 0
United States
Message 4210 - Posted: 5 Jul 2004, 0:00:36 UTC

Speaking of benchmarks, can someone shed some light on the benchmark situation with HT processors? I thought I read a post a few days ago (Heffed?) that mentioned that the benchmarks were faulty on the HT processors and recommended running a manual benchmark with HT disabled. But I also understand that BOINC runs new benchmarks from time to time, and these of course will be done while the system is in normal operation with HT enabled.

On my particular system, there is a difference in the integer BOINC benchmark of almost 100% with HT enabled/disabled. What gives here? :)
ID: 4210 · Report as offensive
Profile Keck_Komputers
Volunteer tester
Avatar

Send message
Joined: 4 Jul 99
Posts: 1575
Credit: 4,152,111
RAC: 1
United States
Message 4336 - Posted: 5 Jul 2004, 9:23:00 UTC

The benchmarking system does still leave a bit to be desired. It has been majorly improved however. At one time it was so bad that HT processors normally claimed double the credit they should and as much as 10 times what they should.

@D. Gustibus
Running benchmarks with HT off is not a good idea, unless that is how you normally run your computer. I think the post you saw was one where we were trying to evaluate how well the new benchmarks were working.

John Keck
BOINCing since 2002/12/08
ID: 4336 · Report as offensive
Profile Thierry Van Driessche
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3083
Credit: 150,096
RAC: 0
Belgium
Message 4346 - Posted: 5 Jul 2004, 9:50:07 UTC
Last modified: 20 Jul 2004, 12:51:31 UTC

These are some of my results of benchmarking.

HT enabled
--- - 2004-06-25 12:28:35 - Number of CPUs: 2
--- - 2004-06-25 12:28:35 - 1585 double precision MIPS (Whetstone) per CPU
--- - 2004-06-25 12:28:35 - 1876 integer MIPS (Dhrystone) per CPU

--- - 2004-06-26 10:20:39 - 1568 double precision MIPS (Whetstone) per CPU
--- - 2004-06-26 10:20:39 - 1786 integer MIPS (Dhrystone) per CPU

HT disabled, all processes running, no RAM optimization
--- - 2004-06-26 10:32:36 - Number of CPUs: 1
--- - 2004-06-26 10:32:36 - 1798 double precision MIPS (Whetstone) per CPU
--- - 2004-06-26 10:32:36 - 3868 integer MIPS (Dhrystone) per CPU
HT disabled, unnecessary processes killed, after RAM optimization
--- - 2004-06-26 10:37:59 - Number of CPUs: 1
--- - 2004-06-26 10:37:59 - 1851 double precision MIPS (Whetstone) per CPU
--- - 2004-06-26 10:37:59 - 4001 integer MIPS (Dhrystone) per CPU
All above benchmarks have been run manually.

Running CPU benchmark AUTOMATICALLY
--- - 2004-07-01 10:39:21 - Number of CPUs: 2
--- - 2004-07-01 10:39:21 - 1560 double precision MIPS (Whetstone) per CPU
--- - 2004-07-01 10:39:21 - 1823 integer MIPS (Dhrystone) per CPU

Conclusions:
1. between the two dates, 6/25 and 7/1, there are not that big differences,
2. between the 2 consecutive benchmarks of the 6/25, the differences are quite acceptable,
3. benchmark results are quite depending on processes that are running and on optimization of RAM,
4. Whetstone numbers does not change that much using HT or not but Dhrystone is dramatically affected.

Greetings from Belgium.
ID: 4346 · Report as offensive
Mark Stevens

Send message
Joined: 15 Feb 01
Posts: 28
Credit: 177,705
RAC: 0
United Kingdom
Message 4380 - Posted: 5 Jul 2004, 12:14:53 UTC

Assuming the benchmarks are off target. In relation to the scoring, what's worse: having a benchmark which is too low or too high?
ID: 4380 · Report as offensive
Profile Thierry Van Driessche
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3083
Credit: 150,096
RAC: 0
Belgium
Message 4387 - Posted: 5 Jul 2004, 12:26:52 UTC - in response to Message 4380.  

> Assuming the benchmarks are off target. In relation to the scoring, what's
> worse: having a benchmark which is too low or too high?

In relation to the claimed credit, the best benchmark is the one with the highest score.
ID: 4387 · Report as offensive
SURVEYOR
Volunteer tester

Send message
Joined: 19 Oct 02
Posts: 375
Credit: 608,422
RAC: 0
United States
Message 4468 - Posted: 5 Jul 2004, 16:03:36 UTC

Check this
Benchmark Results
==============================
Intel Pentium 4HT, 3400 MHz (17 x 200) 1024 MB (DDR SDRAM)
Microsoft Windows XP Home Edition Service Pack 1 (05.01.2600.00)

Boinc 3.19 Setiathome version 3.08
...Number of CPUs: 2
...1867 double precision MIPS (Whetstone) per CPU
...2250 integer MIPS (Dhrystone) per CPU
-----------------------------
FreshDiagnose Version 6.60 by fRESH dEVICES http://www.freshdevices.com/
..CPU Bench Result
..WhetStone FPU 4,859 MWIPS
..DhryStone ALU 7,394 MDIPS
..Speed 3400 MHz
-----------------------------
SiSoftware Sandra Version 2004.SP1
...Benchmark Results
...Dhrystone ALU : 10096MIPS
...Whetstone FPU : 4244MFLOPS
...Whetstone iSSE2 : 7494MFLOPS

============================
Intel Pentium 4A, 2533 MHz (19 x 133) 512 MB (DDR SDRAM)
Microsoft Windows XP Home Edition Service Pack 1 (05.01.2600.00)

Boinc 3.19 Setiathome version 3.08
...Number of CPUs: 1
...1625 double precision MIPS (Whetstone) per CPU
...3634 integer MIPS (Dhrystone) per CPU
------------------------------
FreshDiagnose Version 6.60 by fRESH dEVICES http://www.freshdevices.com/
..CPU Bench Result
..WhetStone FPU 2,699 MWIPS
..DhryStone ALU 5,778 MDIPS
..Speed 2524 MHz
------------------------------
SiSoftware Sandra Version 2004.SP1
...Benchmark Results
...Dhrystone ALU : 6478MIPS
...Whetstone FPU : 1860MFLOPS
...Whetstone iSSE2 : 3394MFLOPS
===========================

BOINC Alpha Tester
BOINC Beta Tester
ID: 4468 · Report as offensive

Message boards : Number crunching : Are Benchmarks Accurate?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.