Please bear with me as I try to explain why I'm asking this question.

In late October, I switched one of my boxes (#6979886) from standard apps over to Lunatics, just sort of cautiously dipping my toe in the water, so to speak. I made no other performance-related changes and didn't touch the default app_info file created by the Lunatics installer. (I was already using app_config, mbcuda, and ap_cmdline parameters, so I just copied them over to the new Lunatics files, where necessary.)

Now, with 6 weeks of Lunatics tasks completed, it seems like the results should be stabilized enough to do some comparisons. I found the following:

RAC: Stock (a/o 10/25) = 8,611.29; Lunatics (a/o 12/6) = 9,193.27; +6.8% (a modest increase)
Total Credits (6 week period): Stock (9/13-10/25) = 359,730; Lunatics (10/25-12/6) = 391,400; +8.9% (again, a modest increase)

Since the increases in those measures seemed rather modest and not as great as the extravagant claims that some folks have made regarding Lunatics, I drilled a bit deeper, trying to look at the 4 individual task categories (AP on CPU, AP on GPU, MB on CPU, and MB on GPU). It was my understanding that the first 3 of those should show improvement under Lunatics, while the last should remain essentially the same.

Using that "Average Processing Rate" metric (from Application details, a/o 12/6), I found:

AP CPU: Stock APR = 14.49195, Lunatics APR = 42.78317 (appears to be a VERY nice increase)
AP GPU: Stock APR = 215.78144, Lunatics APR = 389.14612 (also a nice increase)
MB CPU: Stock APR = 11.33956, Lunatics APR = 14.33899 (modest, but certainly an increase)
MB GPU (cuda50): Stock APR = 95.67301, Lunatics APR = 83.25852 (say WHAT??)

Obviously, this last APR comparison is what got my attention, especially since the bulk of the work that this machine does is likely MB tasks on the GTX 550 Ti GPU. Might that be dragging down my overall results?

Thinking that I might be able to confirm what the APR was showing by comparing a few numbers from task result reports for similar MB GPU tasks, sampled before and after the changeover, I came up with the following pairings (where AR=angle range, FC=flopcounter, RT=run time in seconds, CT=CPU time in seconds):

Stock: AR=0.382054; FC=53,072,655,244,881; RT=2,107.65; CT=201.40
Lunatics: AR=0.381663; FC=53,110,735,033,713; RT=2,078.04; CT = 251.21

Stock: AR=0.448108; FC=38,024,781,934,260; RT=1,797.78; CT=179.85
Lunatics: AR=0.448017, FC=38,027,145,108,247; RT=1,754.80; CT=217.34

Stock: AR=2.765495; FC=16,349,089,995,284; RT=1,119.74; CT=127.23
Lunatics: AR=2.717296; FC=16,354,049,619,830; RT=1,456.81; CT=129.36

Stock: AR=7.236807; FC=15,952,040,115,690; RT=1,119.74; CT=128.39
Lunatics: AR=7.813126; FC=15,950,270,023,272; RT=2,136.19; CT=146.98

Now, from my rather limited knowledge of what's going on in these apps (always a dangerous thing), it does seem to me that these comparisons generally confirm my original understanding, that the MB GPU (cuda50) Lunatics app is essentially performing the same as it was as a stock app, and not at the diminished level that the APR would suggest.

So, in view of the "Average Processing Rate" numbers shown for MB GPU tasks, I finally come back around to my "title" question: "What, exactly, does APR measure, and is it meaningful?" If it is, how can that significant drop in APR be explained?

With my GTX 550Ti's I run the CUDA42 app and this is likely why you're going a bit backwards running the CUDA50 app (CUDA50 is more meant for latter cards).

Cheers.

With my GTX 550Ti's I run the CUDA42 app and this is likely why you're going a bit backwards running the CUDA50 app (CUDA50 is more meant for latter cards).

Cheers.

I chose the CUDA50 because that's what the scheduler had finally settled on for the stock app. It tried out CUDA32, CUDA42, and CUDA50 when v7 first rolled out and eventually just sent CUDA50. On the Application details page, you can see a much higher APR for the CUDA50 over CUDA32 and CUDA42, also. Which perhaps comes back to my original question.

Edit: Ah, it occurs to me, though, that when v7 first rolled out, that box had a GT 640 in it. It wasn't replaced with the GTX 550 Ti until August 5th, so I think by that time the scheduler had already settled on CUDA50 and never retried the other apps with the new GPU. In any event, CUDA50 has been running both as stock and Lunatics, so I would think that comparison would still be valid.

On my systems the GTX 460 is running Cuda50. The GTX 560Ti is running Cuda42- they were the ones settled on by the stock application.
When i set the video cards to run 2 at a time, the APR number dropped- even though the number of WUs processesed per hour went up, this being due to the increase in time it takes to process each WU even though overall throughput is improved.

---

Grant
Darwin NT

With my GTX 550Ti's I run the CUDA42 app and this is likely why you're going a bit backwards running the CUDA50 app (CUDA50 is more meant for latter cards).

Cheers.

I chose the CUDA50 because that's what the scheduler had finally settled on for the stock app. It tried out CUDA32, CUDA42, and CUDA50 when v7 first rolled out and eventually just sent CUDA50. On the Application details page, you can see a much higher APR for the CUDA50 over CUDA32 and CUDA42, also. Which perhaps comes back to my original question.

Edit: Ah, it occurs to me, though, that when v7 first rolled out, that box had a GT 640 in it. It wasn't replaced with the GTX 550 Ti until August 5th, so I think by that time the scheduler had already settled on CUDA50 and never retried the other apps with the new GPU. In any event, CUDA50 has been running both as stock and Lunatics, so I would think that comparison would still be valid.

A 550Ti running cuda 5.0 will almost certainly be slower than a 640 runnignn cuda 5.0 - lower APR.

you can probbaly get a bit faster by using 4.2 on the 550Ti.

IOW you might be comparing green apples to green pears but you are still comparing apples to pears (difefrent cards).

If you don't change anything else APR btween stock and lunatics will be more or less the same because it's the same app.

I'd recommend using the Cuda 4.2 falvour on the 550Ti and see if you get extra throughput by running 2 tasks at once.


APR is a part of CreditNew. It measures, in a not very good way, how fast a particular app is on your system.
Once established it is very difficult to shift. So if you change something in the system (like changing the card) I'd trust APR less than I can throw David, while balancing a ball on my nose and singing the Chipmunk song.

---

A person who won't read has no advantage over one who can't read. (Mark Twain)

If you don't change anything else APR btween stock and lunatics will be more or less the same because it's the same app.

That's just it. I didn't change anything upon switching from stock to Lunatics. The change from GT 640 to GTX 550 Ti occurred way back in early August, after the scheduler had already established Cuda50 as the app of choice, while my switch to Lunatics wasn't until nearly the end of October. The 550 Ti has never run anything other than Cuda50 under stock or under Lunatics. Yet the APR appears to be considerably lower under Lunatics.

I'd recommend using the Cuda 4.2 falvour on the 550Ti and see if you get extra throughput by running 2 tasks at once.

I already do run 2 tasks at once (and have been since June 8th), either 2 MB or 1 MB + 1 AP, using the app_config.xml settings (which is why I haven't touched the default app_info file.) I might try switching to Cuda42, though, perhaps later this week.

APR is a part of CreditNew. It measures, in a not very good way, how fast a particular app is on your system.
Once established it is very difficult to shift.

Actually, it appears to me as if the APR figure is quite volatile. For instance, in my original post, I noted that the figure for Lunatics MB GPU as of last Friday evening was 83.25852. This morning I see that it's down to 76.72412. I think I've seen it as high as 89, but that still falls quite a bit short of the stock figure of 95.67301. Which is what led me to this question.

What APR measures - or attempts to measure - is the 'speed' with which the application, running on whatever combination of hardware you have available (CPU, motherboard, memory, hard disks - as well as the GPU) processes real life SETI workunits.

That's not as easy as it sounds. How big is a workunit? So is a given number of computational seconds for the unit fast or slow?

As you will have noticed, SETI tasks come in different sizes - some take longer to process than others. That's because we're analysing radio recordings made at the Arecibo radio telescope: the telescope is used in different ways by different researchers: sometimes they are staring at a single point in the sky, sometimes they are conducting a slow, detailled survey of the sky, and sometimes they are conducting a fast, broad-brush survey. And sometimes they switch between two or three of those during the time we're recording.

All of those telescopic variations are boiled down into a single number, the true angle range or 'AR', which is the difference between the first and last telescope aiming point in the sky for the particular chunk of data you're working on.

A few years ago, several of us - notably Joe Segur - tried to check how long tasks would take at various ARs. You can read about it in Estimates and Deadlines revisited, and in particular get an idea of how much variability there is in the system.

The 'estimated' size (time to complete) of the tasks you get today is still based on that work. The AR of the task is used to look up - using an updated version of Joe's formula - the expected number of 'Floating Point Operations' needed to complete the task. That number is embedded in each task you receive as <rsc_fpops_est>.

But that work was done in 2007/8, and things have changed since then.

* CPUs have got better, and new capabilities have been added
* GPUs have been invented
* Extra processing ('autocorrelations') has been added to the SETI search.
* Extra optimisation has been added to - or in some cases taken away from - the applications we use.

As a result, the single <rsc_fpops_est> curve can't, realistically, represent a true estimated running time for every different task running with every different available app on every variation of hardware. APR is an attempt to get a reasonable, usable, value for the hardware/application combination in question, but it doesn't attempt to correct for any inaccuracies in the AR/fpops curve.

What you will find is that if you divide the size of the task (<rsc_fops_est> - the number of floating point ops) by the APR (billions of ops per second), you will see BOINC's initial estimate of how long the task will take, in seconds.

In all likelihood, that estimate will be wrong, and an adjustment will be fed back into the APR on completion. In the early stages of using a new app, the adjustments will make a large and noticeable difference, because BOINC has less experience (fewer completed tasks) to base its averages on. As the number of completed tasks increases, each individual result makes a smaller impact on the 'average so far', so you'll stop noticing the changes.

When i set the video cards to run 2 at a time, the APR number dropped- even though the number of WUs processesed per hour went up, this being due to the increase in time it takes to process each WU even though overall throughput is improved.

I think there might be a clue here. I've been running 2 at a time for about 6 months, so that didn't change when I migrated from stock to Lunatics. However, I'm wondering if the increased efficiency of the AP GPU app under Lunatics might have a corresponding dampening effect on MB tasks when the GPU is running both an AP and an MB task at the same time. (I have it set up to run either 2 MB tasks or 1 MB + 1 AP concurrently.) I don't get all that many AP tasks (just 111 APs vs. 3484 MBs on the GPU since changing to Lunatics), so I wouldn't really think the effect would be that dramatic, but still possible. Anybody have any thoughts on this conjecture?

Thank you, Richard! That's a very comprehensive explanation. I had noticed very early on, after rejoining the project in February, that there was a definite relationship between angle range and task run time, but I had never understood what "angle range" actually meant. I feel smarter already .... and it's only Monday!

I gather, then, that the APR shown on the Application details page should be taken with a rather large grain (chunk?) of salt. However, if the scheduler is using that figure to ultimately determine which Cuda app should go to a given machine, and if APR is recommended as the metric to use for Cuda selection for those switching from stock to Lunatics, isn't that a bit shaky?

For instance, even though the performance improvement I've seen on the one box is far more modest than I might have expected given some of the claims I've read, it is an improvement nonetheless, and I will probably gradually convert over some or all of my other machines to Lunatics. But choosing the correct Cuda app based on the APR, (or on the scheduler's ultimate decision, based again on the APR) seems a bit suspect in some cases.

I have one machine (#7057115) running 2 GTX 660s, to which the scheduler had seemingly decided to send only Cuda42 tasks for many weeks, and the APR seemed to support that, by a narrow margin. However, over the last several days, it has suddenly flip-flopped, and at the moment is now sending only Cuda50 tasks. Now the APR on the Application details page appears to strongly support that new choice. In short, if I had switched from stock to Lunatics on that machine last week, I would have selected Cuda42, but this week it would be Cuda50. Yet nothing in the setup of that machine has changed in many months (although there is a GTX 670 in its future).

I have just the opposite situation on another machine (#6980751), which was getting nothing but Cuda50 for a very long time and is now getting only Cuda42. However, that machine is rather schizophrenic anyway, with a GTX 660, GTX 650, and two GT 640s (the 2nd of which was only added about 3 weeks ago).

In short, I'm still rather uncertain as to whether APR is actually a meaningful tool for an end user trying to compare performance or make configuration decisions.

Another reason APR is volatile (at least for the x41 Cuda apps) is that the Cuda app do the autocorrelations portion inefficiently, meaning that VHAR Wu's (which have highest proportion of autocorrelations) cause the APR to go down,
(It's all to do with the sheer number of synchs and memory transfers), it's also one of the reason's why the APRs are lower for v7 Cuda work than v6 Cuda work.

This also means the theoretical fastest Cuda app version can it's APR driven down by VHARs (or VLARs, as they were issued to Keplers for a while) to a lower value than a theoretical slower Cuda app version causing them to switch over.

A trick to stop that happening is do an app_config.xml to run your unfavoured Stock Cuda app at a higher number of instances to drive it's APR further down, and your favoured Stock Cuda app at a lesser number of instances to drive it's APR up,
with the Currently Recommended Boinc (7.2.28, 7.2.31 or 7.2.33) you'd need to suspend all the Wu's of the app_version you don't what to change, then switch gpu_usage, then do them, rince and repeat.

With the Development versions of Boinc (7.2.29, 7.2.30, 7.2.32 and 7.2.34 at present), you can put different ngpus values in for different plan classes, so driving the APR the way you want it,
for example here I've set the Cuda32 and Cuda42 apps to run at 4 tasks at once and Cuda50 at 2 tasks at once:

<app_config>
   <app_version>
       <app_name>setiathome_v7</app_name>
       <plan_class>cuda32</plan_class>
       <avg_ncpus>0.04</avg_ncpus>
       <ngpus>0.25</ngpus>
   </app_version>
   <app_version>
       <app_name>setiathome_v7</app_name>
       <plan_class>cuda42</plan_class>
       <avg_ncpus>0.04</avg_ncpus>
       <ngpus>0.25</ngpus>
   </app_version>
   <app_version>
       <app_name>setiathome_v7</app_name>
       <plan_class>cuda50</plan_class>
       <avg_ncpus>0.04</avg_ncpus>
       <ngpus>0.5</ngpus>
   </app_version>
</app_config>

Claggy

A trick to stop that happening is do an app_config.xml to run your unfavoured Stock Cuda app at a higher number of instances to drive it's APR further down, and your favoured Stock Cuda app at a lesser number of instances to drive it's APR up

Seems like that would be an interesting exercise, but I don't think it would serve any purpose at present for me to try to artificially manipulate the APR when I'm really just trying to figure out if the naturally-generated values it presents are useful. Only if I was already certain of which stock Cuda app should be favored would I want to do that, I think (and then only if the scheduler disagreed).

...
In short, I'm still rather uncertain as to whether APR is actually a meaningful tool for an end user trying to compare performance or make configuration decisions.

IMO APR is the best indicator BOINC supplies, and a user could usually get a near-optimum configuration by using its indications. But I agree it's loose enough that making decisions requires extended observations.

The averaging is performed using "Brown's Simple Exponential Smoothing" with a 0.01 parameter. In effect each new sample is multiplied by 0.01 and added to the previous average multiplied by 1 - 0.01 (though the actual code uses a different form). That kind of exponential average only needs a single saved value. With that 0.01 value, if a user changes something which affects production it takes about 69 validated results from the new configuration for APR to adjust halfway to the productivity change, and over 300 to adjust within 5% of final.

The same of course applies when there's variation caused by mismatch between the rsc_fpops_est values and actual performance. The tendency of S@H work delivery to provide batches of tasks with similar AR exacerbates that effect. That's why for a GPU configuration APR is fairly volatile. At the other end of the scale, it changes glacially slowly for the slowest 100000 hosts which do less than 2 tasks per day.

An alternative approach is to do offline testing. The Optimize your GPU. Find the value the easy way. thread discusses a tool Fred put together to aid that, but as supplied it contained applications for Sah Enhanced and Fred's downloads are temporarily unavailable anyway. BilBg's extended version in post 1300787 may be available, but it too would need v7 applications and test WUs added. At Lunatics there are test WUs and simpler test packages which don't attempt to check performance with multiple tasks simultaneously.

                                                                  Joe

Thanks for the explanation, Joe. I think I understand some of it! ;^)

it takes about 69 validated results from the new configuration for APR to adjust halfway to the productivity change, and over 300 to adjust within 5% of final.

The tendency of S@H work delivery to provide batches of tasks with similar AR exacerbates that effect. That's why for a GPU configuration APR is fairly volatile.

Okay, so I guess that should tell me that a GPU that processes close to 100 tasks a day could see some fairly wide APR swings in just a few days when S@h gets in a rut with WUs from one portion of the AR scale for a while, then gets into a different rut with a very different set of ARs. I suppose that could explain the much higher Cuda50 APR on the Application details page if the last several hundred tasks that were processed under the stock app were at one extreme of the Angle Range scale, leaving that "final" APR at the top of the roller coaster. Still, though, I don't believe I've seen the MB GPU APR under Lunatics come close to that peak, if that was what it was. Oh, well.

An alternative approach is to do offline testing.

I don't think I need to go that route. My main concern was that apparent APR decline for MB GPU tasks between stock and Lunatics apps, but I think I've learned enough to go ahead and ignore that discrepancy, trust that the examples I first presented with results from similar ARs under stock and Lunatics actually show that the performance is essentially the same after all, and move forward from there. Which, in this case, probably means following the earlier suggestions to switch from Cuda50 to Cuda42 for the GTX 550 Ti, despite what the scheduler had been sending under the stock app. I imagine I'll make that change in a day or two, and see what happens.

Edit: Going forward, if I can't entirely trust the APR or the scheduler's decisions, what would the "voices of experience" suggest for Cuda apps on my box with the two GTX 660s and then the mongrel with GTX 660, GTX 650, and two GT 640s?

An alternative approach is to do offline testing. The Optimize your GPU. Find the value the easy way. thread discusses a tool Fred put together to aid that, but as supplied it contained applications for Sah Enhanced and Fred's downloads are temporarily unavailable anyway. BilBg's extended version in post 1300787 may be available, but it too would need v7 applications and test WUs added. At Lunatics there are test WUs and simpler test packages which don't attempt to check performance with multiple tasks simultaneously.

                                                                  Joe

You made me update the package ;)

SetiPerformance 1.8 + Added apps from Lunatics v0.41
http://setiathome.berkeley.edu/forum_thread.php?id=73524

---

 


- ALF - "Find out what you don't do well ..... then don't do it!" :)
 

For what it's worth, I wanted to provide a final (?) update to this thread. Following the recommendations of Wiggo and William, I switched from Cuda50 to Cuda42 on the GTX 550Ti on December 10, 2013. Since I used a 6 week sample period for the stats I listed in my first post in this thread, it seemed reasonable that a 6-week run of Cuda42 should provide a comparable sample. No other changes were made to either the GPU or CPU processing. (Recapping a bit to put the overall numbers in perspective, this is host 6979886, which runs only 138 hours per week, with a dual-core CPU and running 2 tasks at a time on the GPU.) So here goes.

My original comparisons between Stock (running Cuda50 on the GPU) and Lunatics (also running Cuda50):

RAC: Stock (a/o 10/25) = 8,611.29; Lunatics (a/o 12/6) = 9,193.27; +6.8% (a modest increase)
Total Credits (6 week period): Stock (9/13-10/25) = 359,730; Lunatics (10/25-12/6) = 391,400; +8.9% (again, a modest increase)
MB GPU (cuda50): Stock APR = 95.67301, Lunatics APR = 83.25852

Now, comparing Stock and Lunatics running Cuda42:

RAC: Stock (a/o 10/25) = 8,611.29; Lunatics (a/o 1/22) = 9,098.01; +5.7% (a modest increase from Stock, but actually a decrease from the Lunatics Cuda50 period)
Total Credits (6 week period): Stock (9/13-10/25) = 359,730; Lunatics (12/11-1/22) = 394,827; +9.8% (less than 1% better than Lunatics Cuda50)

The other metric, and the original subject of the thread, was the APR, which I see now shows the measurement unit of GFLOPS. As of the moment, it appears to have climbed up to 91.03 GFLOPS for MB GPU. This is as close as I've seen it come to where Cuda50 was at under stock, but checking it periodically over the last 6 weeks, I've still seen a high degree of volatility, hitting a low of around 77 GFLOPS as recently as a week or two ago. That volatility still makes it hard for me to think of it as a terribly reliable decision-making tool, either for scheduler or human use, but I also realize that it's probably the best we have at present.

In short, it appears that the change from Cuda50 to Cuda42 for the GTX550Ti made very little difference in productivity; a tiny decrease in RAC, which I don't think is of any significance, and a tiny increase in Total Credits, which is probably the best measure of overall performance.

I have noticed that when running 1 task a timne APR is HIGH.
When running 2 tasks at a time APR almost halves.
When running 3 tasks at a time the APR is a bit hihger than a third of the 1 task value.

--> Multiply the APR by the number of tasks you are running at time to get an idea how your GPU is doing.

My rigs APRs at the moment are

SETI@home v7 (anonyymi alusta, CPU)
Keskimääräinen suoritusnopeus (APR) 31.66 GFLOPS

SETI@home v7 (anonyymi alusta, NVIDIAn GPU)
Keskimääräinen suoritusnopeus (APR) 153.66 GFLOPS

AstroPulse v6 (anonyymi alusta, CPU)
Keskimääräinen suoritusnopeus (APR) 135.64 GFLOPS

AstroPulse v6 (anonyymi alusta, NVIDIAn GPU)
Keskimääräinen suoritusnopeus (APR) 497.47 GFLOPS

I run ...
6 CPU tasks at a time.
3 GPU MBv7 tasks at a time.
4 GPU AP tasks at a time.

CPU=3930K@4.3GHz ht on, use 50% of the processors.
GPU=GTX780. To get an idea of total performance multiply by number of tasks.
so CPU APRs are MBv7 = 189.96 and AP = 813.84
and GPU APRs are MBv7 = 460.98 and AP = 1989.88

---

To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones