How Are Benchmarks Calculated?

Message boards : Number crunching : How Are Benchmarks Calculated?
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
Profile mlcudd
Volunteer tester
Avatar

Send message
Joined: 11 Apr 03
Posts: 782
Credit: 63,647
RAC: 0
United States
Message 24030 - Posted: 9 Sep 2004, 9:50:35 UTC

Hi All,
There has been some great talk about all the WU's available which is fantastic, and I wish not to shed a bad light on the progress, however I have a question. I have plenty of WU's, I mean PLENTY!. With what I have right now, it will take me 26 days to crunch. My one fast Athlon box crunches WU's in about 3.8 hours, and the WU's recieved fit that box. However my Intel Celeron 2.8 box, says that I should be crunching WU's in 4.22 hours. Problem is, it takes exactly double that time. When I ran a new set of Benchmarks yesterday, the WU's I just got, are for the exact same amount of time 4.22 hours. I did not request WU's, as soon as my WU's were uploading, 20 downloads were on the transfer tab, even before my upload was complete.. The server was contacted as the WU began uploading, so when the WU was finished uploading I still had to manually update to get the WU reported.
Maybe I have misunderstood, but I thought the Benchmarks told the SSL Server what you can handle. Or is this telling me that my computer should be crunching faster. As slow as the Celeron is, never make that mistake again.

Warm Regards,

Rocky

ID: 24030 · Report as offensive
Petit Soleil
Avatar

Send message
Joined: 17 Feb 03
Posts: 1497
Credit: 70,934
RAC: 0
Canada
Message 24072 - Posted: 9 Sep 2004, 12:50:57 UTC
Last modified: 9 Sep 2004, 12:53:40 UTC

Hi Rocky,

It looks strange that your celeron box takes twice the time calculated by the benchmarks to process one work unit. It is usualy the opposite. My machine here when doing benchmark is telling BOINC that it should take 9 hours to complete but in fact it takes 6 hours. That's what all have been reported so far.

It's a fact that celerons are not good crunchers but almost 9 hours on a 2.8 GHZ looks very slow to me. That said, I am not a CPU specialist. Have a look here http://setiweb.ssl.berkeley.edu/forum_thread.php?id=3757
You might find other celeron's results to compair with.

Could the server being mistaken your celeron box with another one? Have a look in your account / view computer and check if everything seem OK there.

That's all I can think of right now.

EDIT // Do you show graphics on that box?

Friendly
Marc

-.-. --.- -.. -..- . - --... ...-- .-.-. -.-
ID: 24072 · Report as offensive
Profile Paul D. Buck
Volunteer tester

Send message
Joined: 19 Jul 00
Posts: 3898
Credit: 1,158,042
RAC: 0
United States
Message 24098 - Posted: 9 Sep 2004, 14:15:33 UTC - in response to Message 24030.  

> Maybe I have misunderstood, but I thought the Benchmarks told the SSL Server
> what you can handle. Or is this telling me that my computer should be
> crunching faster. As slow as the Celeron is, never make that mistake again.

The time attached to the WU when it is queued is not based on the benchmark (to the best of my knowledge).

All my computers have that same time of 4:17:25 to process and they are done in about 3 hours. It is my understanding that this is an "average" processing time assigned so that work can be allocated.

The fixed time duration may be a artiffact of us being still in early days as they work on the BOINC System code. Later days may see this time be dynamically assigned so that the work queued is more accurately tuned to the system capabilities.


ID: 24098 · Report as offensive
Profile mlcudd
Volunteer tester
Avatar

Send message
Joined: 11 Apr 03
Posts: 782
Credit: 63,647
RAC: 0
United States
Message 24100 - Posted: 9 Sep 2004, 14:16:15 UTC


Marc,
Thank you for your response. First I am not using Graphics at all on any box. (Which I Am Glad). I have checked the computers and listed them below. I cannot figure out why there is such a difference. I can understand a WU going over time because of added variables, such as completeing other rpocesses that are processor intensive. However I try to use that box as little as possible for my "Additional work". I use my Athlon because no matter what program I am working on it still crunches away at the WU's. My 3rd box which is my ME box..is somewhat of a dinosaur, so I expect it to be slow. I will look for other posts to tell me how to "Tweak" my Celeron Box.


Computer ID
162941 SIGNALSEARCHER 130.07 3901.81 Intel(R) Celeron(R) CPU 2.80GHz Pentium Microsoft Windows XP Home Edition, Service Pack 1, (05.01.2600.00)
*** The Benchmarks for this box are 4.22 hours, which are one half of actual time per WU.


151121 SLAVE 29.50 7511.34 AMD Athlon(tm) XP 1800+ Pentium Microsoft Windows XP Home Edition, Service Pack 1, (05.01.2600.00)
*** The Benchmarks for this box are 4.26 hours which is usually within 15 minutes per WU.


153222 oemcomputer 6.97 2395.39 GenuineIntel Pentium Microsoft Windows Millennium , (04.90.3000.00)
The Benchmarks on this my slowest box, are 11.25 hours, which is also within 15 minutes per WU.


I do appreciate all the help you have given to me and others Marc.
Have A Great Day, and A Better Tomorrow. :-)

Warm Regards,

Rocky Cudd




ID: 24100 · Report as offensive
Profile mlcudd
Volunteer tester
Avatar

Send message
Joined: 11 Apr 03
Posts: 782
Credit: 63,647
RAC: 0
United States
Message 24103 - Posted: 9 Sep 2004, 14:24:11 UTC - in response to Message 24098.  


> The time attached to the WU when it is queued is not based on the benchmark
> (to the best of my knowledge).
>
> The fixed time duration may be a artiffact of us being still in early days as
> they work on the BOINC System code. Later days may see this time be
> dynamically assigned so that the work queued is more accurately tuned to the
> system capabilities.

Paul,
I appreciate your response. It popped up just as I was replying to Marc.
Question, and I hate to appear ignorant but, why perform the benchmarks at all right now? and how does the server generally know not to send more work units than you can complete? Or is this just an anomaly that is occuring right now do to the swapping of files back and forth at USC? Because before these errors, my computer followed the settings in my preferences and gave me pretty much exactly the amount of WU's needed. Now I have no Idea. I uploaded another WU 15 minutes ago and got 20 more WU's. There is no way under God's green earth that I will be able to crunch all these during this time period. I just wish we could pass them to other computers.
I know I read somewher before that the Bencmarks told the server what size files each box could handle, but thuis does not even appear to be the case.

Thank you very much for giving me and so many others help along the way. I hear from you and Marc quite often and I really appreciate it.

Warm Regards,

Rocky Cudd

ID: 24103 · Report as offensive
Petit Soleil
Avatar

Send message
Joined: 17 Feb 03
Posts: 1497
Credit: 70,934
RAC: 0
Canada
Message 24105 - Posted: 9 Sep 2004, 14:32:36 UTC - in response to Message 24098.  
Last modified: 9 Sep 2004, 14:42:43 UTC

> The time attached to the WU when it is queued is not based on the benchmark
> (to the best of my knowledge).

Hi Paul,

What is the benchmark for then? I have realised that the time attached to the WU is changing each time Boinc is doing benchmarks. The batch I have now are 08:54:11 but the ones before was 9 hours something. Boinc did a benchmark in between.

When a host asks for 4 days of work the server has to be able to figure out how many WU this host is capable of completing in that given time. It just can't be the same for a Pentium MMX 100 MHZ and a Athlon 64. It's got to rely on something and I think benchmarking is the only way to figure that out.

I am not contradicting you, just trying to understand.



ID: 24105 · Report as offensive
Petit Soleil
Avatar

Send message
Joined: 17 Feb 03
Posts: 1497
Credit: 70,934
RAC: 0
Canada
Message 24113 - Posted: 9 Sep 2004, 15:00:15 UTC
Last modified: 9 Sep 2004, 15:06:12 UTC

I have just made a search for CELERON in the message board and found other thread talking about how off are the benchmark results for the celeron. But as you mentionned you did not have that problem before. Could it be that something is wrong in the sever's benchmark data base?

You won't be able to crunch all those extra WU in time and I don't know if there is a way to "transfer" them to another machine. You will probably have to reset your celeron box when you've reached the dead line so you won't crunch for nothing.

Friendly
Marc
-.-. --.- -.-. --.- -.. -..- . - --... ...-- .-.-. -.-
ID: 24113 · Report as offensive
Profile mlcudd
Volunteer tester
Avatar

Send message
Joined: 11 Apr 03
Posts: 782
Credit: 63,647
RAC: 0
United States
Message 24126 - Posted: 9 Sep 2004, 15:42:26 UTC

Thanks Marc,
That is about what I figured. I was hoping Rom would pick up on this thread, but i know he is inundated with re-coding for new versions. They seem to keep him busy. And your right I have been thinking about it, and it never happened before, so I am sure it must be the way the new system is evaluating each PC. If the benchmarks really make no difference in the Wu's you recieve, then I thought I should take a chance and "Transfer some " to another PC, but I am afraid of major meltdown with a Full Cache. I am going to try to email Rom and see if he gets a chance to reply. I have not seen him on the boards in a couple of days.

Warm regards,

Rocky Cudd
ml.cudd@verizon.net
ID: 24126 · Report as offensive
canis lupus

Send message
Joined: 26 Oct 03
Posts: 154
Credit: 13,061
RAC: 0
Message 24129 - Posted: 9 Sep 2004, 15:49:10 UTC - in response to Message 24126.  
Last modified: 9 Sep 2004, 15:49:58 UTC

@Rocky, @Petit

> Thanks Marc,
> That is about what I figured. I was hoping Rom would pick up on this thread,
> but i know he is inundated with re-coding for new versions. They seem to keep
> him busy. And your right I have been thinking about it, and it never happened
> before, so I am sure it must be the way the new system is evaluating each PC.
> If the benchmarks really make no difference in the Wu's you recieve, then I
> thought I should take a chance and "Transfer some " to another PC, but I am
> afraid of major meltdown with a Full Cache. I am going to try to email Rom and
> see if he gets a chance to reply. I have not seen him on the boards in a
> couple of days.
>

The scheduling server keeps track of which host has received which WUs - It won't accept WUs returned from a different host to which it sent them as valid, so moving them to another box won't work.

--
<p>Regards, Paul - Just slowly BOINCing along...</p>

ID: 24129 · Report as offensive
Profile mlcudd
Volunteer tester
Avatar

Send message
Joined: 11 Apr 03
Posts: 782
Credit: 63,647
RAC: 0
United States
Message 24134 - Posted: 9 Sep 2004, 16:18:50 UTC

Thank You Paul,
It was worth a shot to ask...the worst that could be said is NO...or Catastrophic Disaster. :-)

Warm Regards,

Rocky Cudd
ID: 24134 · Report as offensive
Thunder

Send message
Joined: 3 May 03
Posts: 65
Credit: 993,581
RAC: 0
United States
Message 24136 - Posted: 9 Sep 2004, 16:27:06 UTC

I also have one Celeron box (a lowly 1.2Ghz) that also benchmarks much faster than an actual WU will run. BOINC anticipates a WU will take 5h18m on it, while it typically will finish a WU in around 8h50m.

I have tried to figure out what could be different about the Celeron that would cause the benchmark to run much faster than the actual WU, but I'm rather at a loss. The L1 and L2 cache are the same size on this one as an older P3 that I have, so that's not it.

I'd love to hear if anyone else figures out what it is, so I'll keep checking back here... my curiosity is killin' me. ;)
ID: 24136 · Report as offensive
Profile mlcudd
Volunteer tester
Avatar

Send message
Joined: 11 Apr 03
Posts: 782
Credit: 63,647
RAC: 0
United States
Message 24143 - Posted: 9 Sep 2004, 16:45:29 UTC

Hi Thunder,
Well I am totally at a loss now. I have a processor twice the size of yours, and getting nearly the same crunching times? Somethings Amiss. I just haven't figured it out. I am presently online with intel service, and they are telling me, " We will have an answer for you shortly'. They apparently had to transfer the question back to the States from wherever their Support Services are. ( I Think India). So I will wait patiently for the email to arrive, and I will post the results.

I hope everybody has "A Great Day Today, and A Better Tomorrow".

Regards,

Rocky Cudd
ID: 24143 · Report as offensive
Petit Soleil
Avatar

Send message
Joined: 17 Feb 03
Posts: 1497
Credit: 70,934
RAC: 0
Canada
Message 24147 - Posted: 9 Sep 2004, 16:53:57 UTC
Last modified: 9 Sep 2004, 16:55:40 UTC

Weird !!!

We have a 2.8 a 1.2 GHZ Celeron both crunching in about 9 hours !!! As I said earlier I am not a CPU specialist but... Could it be due to different cache size? I know some celeron has 128k others have 256k of L2 cache. I am lost too.

9 hours is about what my G4 800 MHZ need to crunch one WU.
ID: 24147 · Report as offensive
Profile mlcudd
Volunteer tester
Avatar

Send message
Joined: 11 Apr 03
Posts: 782
Credit: 63,647
RAC: 0
United States
Message 24151 - Posted: 9 Sep 2004, 17:17:17 UTC

Hi All,
I do not have a reply from intel yet, and I might just stop by one of their Corporate Support Offices. There is one right here in Fredericksburg, Virginia.
And I have emailed Rom to ask for a little insight, but we all know he is rather inundated at present, keeping things positive.

Regards,

Rocky
ID: 24151 · Report as offensive
Profile Crunch3r
Volunteer tester
Avatar

Send message
Joined: 15 Apr 99
Posts: 1546
Credit: 3,438,823
RAC: 0
Germany
Message 24154 - Posted: 9 Sep 2004, 17:23:52 UTC - in response to Message 24147.  
Last modified: 9 Sep 2004, 17:24:09 UTC

> Weird !!!
>
> We have a 2.8 a 1.2 GHZ Celeron both crunching in about 9 hours !!! As I said
> earlier I am not a CPU specialist but... Could it be due to different cache
> size? I know some celeron has 128k others have 256k of L2 cache. I am lost
> too.
>
> 9 hours is about what my G4 800 MHZ need to crunch one WU.
>

You´ve allready got the clue to your question. It is the different cache size and about the cpu architecture. Your 1.2 Ghz celeron is based on the pentium3 design wich is more efficient per cpu cycle. Your 2.8 Ghz Celeron is based on the old pentium4 wilamette (or something like that),wich has a lower fsb and cache.
ID: 24154 · Report as offensive
Thunder

Send message
Joined: 3 May 03
Posts: 65
Credit: 993,581
RAC: 0
United States
Message 24194 - Posted: 9 Sep 2004, 19:57:28 UTC
Last modified: 9 Sep 2004, 21:09:57 UTC

Yep, that's right (hadn't thought of it yet myself)

I have a "Tualatin" Celeron "A" with a 16K+16K L1 cache and a 256K L2 cache

MLCudd & Petit Soleil, you both have "Northwood" Celerons with a 12K+8K L1 cache and 128K L2 cache.

It doesn't explain why the benchmark is still substantially off for MY celeron as well, but it certainly explains why your cele's aren't fairing much better... obviously the S@H process requires more than 128k cached while it's working on substantial portions of a WU, so on mine, it's holding that data entirely in L2 cache, while yours are having to do reads to main memory (which is at a HUGE speed 'penalty' compared to the on-chip cache).

BTW, MLCudd.... Aim High! Go USAF! ;) (you know a zoomie's gotta poke a little fun at a squid) :)
ID: 24194 · Report as offensive
D. Gustibus

Send message
Joined: 3 Jan 04
Posts: 9
Credit: 242,311
RAC: 0
United States
Message 24233 - Posted: 9 Sep 2004, 22:08:36 UTC

I don't claim to understand the use of benchmarks in BOINC and have been mystified from the beginning by the estimated completion times BOINC shows. Just as a data point, I have two identical Intel boxes here, both with the same motherboard, and both equipped with 2.4Ghz processors. One is a 2.4 Celeron, the other a 2.4 P4.

The P4 shows estimated completion times of ~6:20, and actually completes a WU in about 4:15.

The Celeron shows esimated completion times ~4:30 and actually completes a WU in about 6:30.

Go Figure...


</img>


ID: 24233 · Report as offensive
Profile Rom Walton (BOINC)
Volunteer tester
Avatar

Send message
Joined: 28 Apr 00
Posts: 579
Credit: 130,733
RAC: 0
United States
Message 24237 - Posted: 9 Sep 2004, 22:18:19 UTC

There are three big factors that we have with the present day benchmarks.

Compilers, cache sizes, and machine load at the time the benchmarks are taken.

Ben Herndon has been making quite a few recommendations on how to achieve benchmark parity across compilers. You all will start to notice the benchmark numbers start to make more since across all the platforms with the next release.

We aren’t going to be able to do very much with the cache size problem.

With the machine load problem we have the client re-benchmark the computer every 5 days or on a version change.

The benchmark numbers are used in calculating how much work is to be sent to a client.

----- Rom
BOINC Development Team, U.C. Berkeley
ID: 24237 · Report as offensive
Petit Soleil
Avatar

Send message
Joined: 17 Feb 03
Posts: 1497
Credit: 70,934
RAC: 0
Canada
Message 24238 - Posted: 9 Sep 2004, 22:19:04 UTC - in response to Message 24233.  
Last modified: 9 Sep 2004, 22:26:57 UTC

> Go Figure...

lol
Try to move the box arround. move the left box to the right or switch their hub's connectors. You can also try some voodoo stuff but be carefull.
ID: 24238 · Report as offensive
Petit Soleil
Avatar

Send message
Joined: 17 Feb 03
Posts: 1497
Credit: 70,934
RAC: 0
Canada
Message 24240 - Posted: 9 Sep 2004, 22:25:46 UTC
Last modified: 9 Sep 2004, 23:38:31 UTC

Thanks Rom.

To all Celeron users !!! In order to complete your work in time, be sure to run the benchmarks manually with your box fully loaded. Launch a defrag or play doom while you BOINC benchmarking...

Friendly
Marc

//testing sig

-.-. --.- -.. -..- . - --... ...-- .-.-. -.-
ID: 24240 · Report as offensive
1 · 2 · 3 · Next

Message boards : Number crunching : How Are Benchmarks Calculated?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.