How Are Benchmarks Calculated?

Message boards : Number crunching : How Are Benchmarks Calculated?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3

AuthorMessage
Dave Mickey

Send message
Joined: 19 Oct 99
Posts: 178
Credit: 11,122,965
RAC: 0
United States
Message 52555 - Posted: 10 Dec 2004, 4:32:42 UTC

Just another data point, probably of little interest to
users of modern CPUs -

I have a 333MHz Celeron, 192MB RAM. BOINC predicts
26 hours per WU, and they actually take 18-20.

However, since registering that host it has been off-BOINC
for some extended times (actually running Classic instead)
such that the stat on the web page for "time client is on",
or whatever those 3 params are, is quite a bit below 100%.
(Currently about 70%)

Does that param that the server collects about %uptime
figure into either the estimated completion time or the
amount of work to be sent, or both?

Dave
ID: 52555 · Report as offensive
Ingleside
Volunteer developer

Send message
Joined: 4 Feb 03
Posts: 1546
Credit: 15,832,022
RAC: 13
Norway
Message 52598 - Posted: 10 Dec 2004, 9:58:56 UTC - in response to Message 52555.  

>
> Does that param that the server collects about %uptime
> figure into either the estimated completion time or the
> amount of work to be sent, or both?
>

Active_frac, or "% of time user is active" as shown on web, doesn't change estimated crunch-time, but is used then calculating how much work to download and if fast enough computer to crunch wu before deadline.

The smallest used is 10%, while new installs always uses 100%.
ID: 52598 · Report as offensive
Profile Paul D. Buck
Volunteer tester

Send message
Joined: 19 Jul 00
Posts: 3898
Credit: 1,158,042
RAC: 0
United States
Message 52635 - Posted: 10 Dec 2004, 15:10:25 UTC - in response to Message 52492.  

Ben,

As Ned said:

> I've been thinking of adding a small faq to my site relating to common
> questions concerning some of the more technical aspects of the credit system
> as it seems to relate to my optimised boinc clients (benchmark scores).
> ... [edits] ...
> I was wondering if you'd mind if I lifted/borrowed your excellent explanations and used them on my
> site, acknowledging yourself, of course, as the source?

I too would like to do this ...
ID: 52635 · Report as offensive
Profile Benher
Volunteer developer
Volunteer tester

Send message
Joined: 25 Jul 99
Posts: 517
Credit: 465,152
RAC: 0
United States
Message 52651 - Posted: 10 Dec 2004, 16:52:02 UTC

Ned and Paul,

Sure go ahead. :)

David posted this recent one to the BOINC manual.

http://boinc.berkeley.edu/benchmark.php linked from "credit" page:
http://boinc.berkeley.edu/credit.php
ID: 52651 · Report as offensive
Profile Paul D. Buck
Volunteer tester

Send message
Joined: 19 Jul 00
Posts: 3898
Credit: 1,158,042
RAC: 0
United States
Message 52670 - Posted: 10 Dec 2004, 20:20:21 UTC - in response to Message 52651.  

> Ned and Paul,
>
> Sure go ahead. :)

Thanks!

And I am done! It is added to the term "Benchmarks" in the glossary.
ID: 52670 · Report as offensive
JAF
Avatar

Send message
Joined: 9 Aug 00
Posts: 289
Credit: 168,721
RAC: 0
United States
Message 52687 - Posted: 10 Dec 2004, 22:34:20 UTC

I'm still a bit confused as to why the project doesn't write benchmark code that's similar to the actual Seti crunching code (instead of the Whetstone/Dhrystone tests?) Call it "wuwuWobblestones" or something like that.

I realize that an occasional WU take much longer or much shorter time to crunch, but a majority (at least on my systems) are with 5% of each other.
<img src='http://www.boincsynergy.com/images/stats/comb-912.jpg'>
ID: 52687 · Report as offensive
Ned Slider

Send message
Joined: 12 Oct 01
Posts: 668
Credit: 4,375,315
RAC: 0
United Kingdom
Message 52690 - Posted: 10 Dec 2004, 22:56:30 UTC

Thanks Ben,

Will be a while before I have time to update my site as I've just started a new job and it's currently eating into my personal time, but I'll get it posted as soon as I can.

Regards,

Ned
*** My Guide to Compiling Optimised BOINC and SETI Clients ***
*** Download Optimised BOINC and SETI Clients for Linux Here ***
ID: 52690 · Report as offensive
Profile Keck_Komputers
Volunteer tester
Avatar

Send message
Joined: 4 Jul 99
Posts: 1575
Credit: 4,152,111
RAC: 1
United States
Message 52798 - Posted: 11 Dec 2004, 9:21:19 UTC - in response to Message 52687.  

> I'm still a bit confused as to why the project doesn't write benchmark code
> that's similar to the actual Seti crunching code (instead of the
> Whetstone/Dhrystone tests?) Call it "wuwuWobblestones" or something like
> that.
>
> I realize that an occasional WU take much longer or much shorter time to
> crunch, but a majority (at least on my systems) are with 5% of each other.
>
It has to be fairly general otherwise you get good estimates for seti but they may be way off for LHC, CPDN or some other project.

Two of the biggest problems with getting accurate estimates right now are:
The benchmarks do not consider cache size or memory bandwidth.
Getting an accurate estimate of the ratio of IOPs to FLOPs for the workunits (this may also vary from workunit to workunit).
BOINC WIKI

BOINCing since 2002/12/8
ID: 52798 · Report as offensive
JAF
Avatar

Send message
Joined: 9 Aug 00
Posts: 289
Credit: 168,721
RAC: 0
United States
Message 53286 - Posted: 12 Dec 2004, 23:34:34 UTC - in response to Message 52798.  

> It has to be fairly general otherwise you get good estimates for seti but they
> may be way off for LHC, CPDN or some other project.
>
> Two of the biggest problems with getting accurate estimates right now are:
> The benchmarks do not consider cache size or memory bandwidth.
> Getting an accurate estimate of the ratio of IOPs to FLOPs for the workunits
> (this may also vary from workunit to workunit).
>
Thanks for the response John, and that is what I was missing. Since I've been only doing Seti, I lost track of the point that Boinc is not just Seti.

Has there ever been a consideration of a "project benchmark"? In other words, Boinc would run a benchmark for each project one signs up for. Then we might be able to get more accurate estimations for the amount(s) of work downloaded according to our preferences. In my case, my slowest (estimated) computer crunches WU's the fastest, and my fastest (estimated) crunches the slowest.

<img src='http://www.boincsynergy.com/images/stats/comb-912.jpg'>
ID: 53286 · Report as offensive
Profile Benher
Volunteer developer
Volunteer tester

Send message
Joined: 25 Jul 99
Posts: 517
Credit: 465,152
RAC: 0
United States
Message 53418 - Posted: 13 Dec 2004, 6:16:07 UTC

JAF,

There is some code for this "project benchmark". It is being considered.

ID: 53418 · Report as offensive
Profile Celeron300A@450

Send message
Joined: 16 May 99
Posts: 2
Credit: 318,306
RAC: 0
Malaysia
Message 121781 - Posted: 10 Jun 2005, 17:26:45 UTC - in response to Message 52402.  

> about 3.8 hours, and the WU's recieved fit that box. However my Intel Celeron
> 2.8 box, says that I should be crunching WU's in 4.22 hours. Problem is, it
> takes exactly double that time. When I ran a new set of Benchmarks yesterday,
> the WU's I just got, are for the exact same amount of time 4.22 hours. I did


well, my celeron 2.4ghz is crunching wu's in under 4 hours, when boinc says it should take more than 8 hours.

im not sure if these are factors, but ill list them for analysis :

1) using single stick of cheap unbranded ram. 512mb shared with on board agp chip

2) mobo chipset is single channel via

3) swap file is set to 1024mb

4) usage load is basically for surfing, email, streaming radio.

i do have concerns that i may run out of work. not so much that i wont finish work in time.


With optimised client, benchmarks are over 5 hours while actual is under 2 hours.
ID: 121781 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20331
Credit: 7,508,002
RAC: 20
United Kingdom
Message 121786 - Posted: 10 Jun 2005, 17:35:48 UTC - in response to Message 53418.  
Last modified: 10 Jun 2005, 17:36:15 UTC

There is some code for this "project benchmark". It is being considered.

Benher,

Can you elaborate?

How do the ideas in "2nd: Credit for Clients (Credit errors)" compare?

Regards,
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 121786 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20331
Credit: 7,508,002
RAC: 20
United Kingdom
Message 121793 - Posted: 10 Jun 2005, 17:51:06 UTC - in response to Message 121786.  
Last modified: 10 Jun 2005, 17:53:30 UTC

There is some code for this "project benchmark". It is being considered.
Benher,

Can you elaborate?

How do the ideas in "2nd: Credit for Clients (Credit errors)" compare?

Regards,
Martin

Further thought, will the new benchmarking allow for:

"From another thread, there was very strong sentiment that SCIENCE done rather than CPU WORK done should be credited. Hence, optimised clients should get the same credit for a WU as do unoptimised clients that take twice as long to run the same WU."

Can the benchmarking allow for those users that are running optimised clients and/or optimised systems?

(OK, so I think this really needs to be done by characterising a system by running the project client with test data rather than attempting an artificial benchmark.)

Regards,
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 121793 · Report as offensive
Previous · 1 · 2 · 3

Message boards : Number crunching : How Are Benchmarks Calculated?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.