Observation of CreditNew Impact (2)


log in

Advanced search

Message boards : Number crunching : Observation of CreditNew Impact (2)

1 · 2 · 3 · 4 . . . 20 · Next
Author Message
Lionel
Send message
Joined: 25 Mar 00
Posts: 545
Credit: 227,107,632
RAC: 197,417
Australia
Message 1386253 - Posted: 1 Jul 2013, 0:04:09 UTC

Note: Data for last 9 days added with additional commentary.


I’ve had a look at the data around v7 and v6 WUs as well as AP WUs and below is a quick observational analysis of my data and the impact of v7 in relation to granted credits.

Under v6, I was roughly averaging 100 credits per Work Unit (WU).
Under v7, it seems that the average is sitting around 75-80 credits per WU.
In looking at run-times and taking the outliers out, cpu run time was around 600-660 seconds (10-11 minutes) per WU for v6, and appears to be around 800-1100 seconds (13+ to 18+ minutes) for a v7 WU. CPU time seems to have gone up by a factor of 2-3 from 50-60 seconds for v6 to 90-180 seconds for v7.

So doing a quick Back of the Envelope (10.5/15.83=0.66) shows that from a WU processing/throughput capability, I can expect to do roughly 66% of the volume of WUs that I did before (for example, if I was doing 400 WUs per day under v6, I can now expect to do around 264 WUs per day under v7).

Looking at the impact on credit gives 0.66*0.775 = 0.514 or 51.4%. In essence I can expect that daily credit for v7 will drop to circa 51% of what I was getting under v6.

I am aware of the comments around “that the system needs time to settle down” and that “it thinks all the WUs coming back at the moment are easy, hence the low credit” however, if the system continues to perform as is, then I can expect to see no change from current trajectory.

To test the assumption, I have looked at credit per day pre v7 and post v7 implementation (the data is contained below).

The average daily credit prior to migration was 221,878. Following migration on 1st June, the average daily credit is showing as 102,376 which is circa 46.1% of the previous daily average under v6. Up to and including 22 June, I saw no real change in daily total even though was comment that the credit system had been tweaked.

As part of my thinking, I decided to “benchmark” v7 credits against Astropulse credits. I started this on 23 June with the migration of a single box. Over the next day or two, I noticed what appeared to be an increase in daily totals, so decided to migrate the other two boxes to AP only to see what the full impact would be.

The assumption that I was running with was that pre v7, v6 and AP should have been fairly well benchmarked against each other and that the granting of credits would be in approximate equilibrium based on a daily basis.

If this was the case, then I would expect to see a rise in daily totals from circa 102k to circa 220k and for the daily total to remain around 220k. From 23 June onwards, totals increased on a daily basis towards the peak of 222k on 29 June. Unfortunately there has been a lack of AP WUs starting around 30 June and so daily totals have declined over the last few days.

My observation in here (albeit based on a small amount of data) is that v6 and AP seemed to be fairly well benchmarked against each other. The issue is with v7. V7 is not well benchmarked against v6 WUs, nor is it well benchmarked against AP WUs. In fact, one can almost double their existing daily run rate through only processing AP WUs.


Daily run rates:

2013.05.16 – 244,130
2013.05.17 – 220,168
2013.05.18 – 231,098
2013.05.19 – 226,353
2013.05.20 – 224,723
2013.05.21 – 210,477
2013.05.22 - 0
2013.05.23 – 431,485
2013.05.24 – 229,312
2013.05.25 – 228,767
2013.05.26 – 239,021
2013.05.27 – 231,271
2013.05.28 – 231,050
2013.05.29 – 0
2013.05.30 – 392,635
2013.05.31 – 209,556

Migration of all boxes to v7 on “day 1”

2013.06.01 – 123,072
2013.06.02 – 94,061
2013.06.03 – 102,333
2013.06.04 – 99,896
2013.06.05 - 65,653
2013.06.06 - 112,209
2013.06.07 - 102,538
2013.06.08 - 110,760
2013.06.09 - 89,757
2013.06.10 - 96,018
2013.06.11 - 111,653
2013.06.12 - 90,091
2013.06.13 - 119,848
2013.06.14 - 99,884
2013.06.15 - 104,561
2013.06.16 - 110,566
2013.06.17 - 110,603
2013.06.18 - 102,856
2013.06.19 - 85,268
2013.06.20 - 140,694
2013.06.21 - 70,247
2013.06.22 - 109,698

Migration of all boxes away from v7 to AP only (over period of ~4 days)

2013.06.23 – 126,873
2013.06.24 – 141,847
2013.06.25 – 169,625
2013.06.26 – 169,517
2013.06.27 – 183,693
2013.06.28 – 206,019
2013.06.29 – 222,126
2013.06.30 – 211,468 (Lack of AP wus from here)
2013.07.01 – 182,394


Comment on Recognition:

Whilst some are focusing just on credit, the issue is not about credit as such. It’s about recognition.

There are many distributed computing projects to which people contribute resources. The manner and means in which those projects recognise individual contribution is through a system that is based on and allocates credits. Some projects choose to recognise a person’s contribution more than other projects, thus they grant a higher credit rate per contribution for that project. In short, credits are effectively an indication of a person’s contribution to a project.

In the case of "the New Credit System" implemented by Seti, recognition of personal contribution has been reduced. At present, the indication is that recognition for effort is effectively half that of what it was prior to the new recognition system being employed.


____________

Profile James SotherdenProject donor
Avatar
Send message
Joined: 16 May 99
Posts: 8788
Credit: 34,021,423
RAC: 58,387
United States
Message 1386259 - Posted: 1 Jul 2013, 0:20:18 UTC

And other projects have told DR.A. to stuff his newcredit. Now if he said you will run newcredit or you cant run Boinc, Then you will see parity.

I run Seti@Home beacause I believe in it.
____________

Old James

Profile Michael W.F. Miles
Avatar
Send message
Joined: 24 Mar 07
Posts: 238
Credit: 28,289,338
RAC: 19,887
Canada
Message 1386589 - Posted: 1 Jul 2013, 20:33:09 UTC - in response to Message 1386253.

From what I am seeing running v7 MB cpu tasks are just not worth it.
The time it takes a gpu to run AP is one hour on average for a 700 credit
the same time for a MB tasks may get you 70
big difference

Why so little credit for a MB task.
It sure looks like AP is king again for credits

Michael Miles

Lionel
Send message
Joined: 25 Mar 00
Posts: 545
Credit: 227,107,632
RAC: 197,417
Australia
Message 1386659 - Posted: 2 Jul 2013, 0:24:26 UTC - in response to Message 1386589.

James, Michael, I agree. The new credit system is an absolute stuff up.

I also agree that running v7 on a cpu is now virtually not worth it. v7 has taken this over the tipping point. You are far better running AP on a cpu and getting at least twice the credit per core per hour.

Also, when AP wus come back you are far better to only processing AP wus, killing off all the others till they are no APs then flipping back to enhanced and v7 to fill the lack of work void.

But you know what staggers me is the deafening silence coming from Anderson, Korpela and co. You would think that given the angst around over v7 credits that they would communicate more with the volunteers and briefing us on what they are doing to fix the problem, rather than just appear to be sitting on their hands and putting their heads in the sand.



____________

Profile Blurf
Volunteer tester
Send message
Joined: 2 Sep 06
Posts: 7545
Credit: 6,803,287
RAC: 8,405
United States
Message 1386677 - Posted: 2 Jul 2013, 1:51:47 UTC - in response to Message 1386659.
Last modified: 2 Jul 2013, 1:52:16 UTC

But you know what staggers me is the deafening silence coming from Anderson, Korpela and co. You would think that given the angst around over v7 credits that they would communicate more with the volunteers and briefing us on what they are doing to fix the problem, rather than just appear to be sitting on their hands and putting their heads in the sand.


Lionel---

1) Dr Anderson very rarely--if ever--posts on the forums. Don't expect to see posts from here.

2) Dr. Korpela and his staff do not engage in "sitting on their hands"--they're extremely short both budget & staff-wise so they can either take time to post on the forums with every little thing they do, or they can work on keeping the servers. Please cut them some slack. Matt does post updates when he can.
____________


Profile betregerProject donor
Avatar
Send message
Joined: 29 Jun 99
Posts: 2303
Credit: 4,870,031
RAC: 9,354
United States
Message 1386683 - Posted: 2 Jul 2013, 2:18:44 UTC - in response to Message 1386677.

Well stated.
____________

Profile {BDC} Thomas DupontProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Dec 11
Posts: 3758
Credit: 1,310,933
RAC: 289
France
Message 1386758 - Posted: 2 Jul 2013, 7:40:57 UTC - in response to Message 1386253.

Thanks very much for the analysis of changes in your RAC Lionel !
Very interesting to see a big cruncher like you.
It's very instructive especially for me because I can compare your stalls in KC compared to my team.
However, I totally agree with Blurf : don't forget that the staff and budget of the project are minimal and that the project is still working in the emergency.
You're here for over 13 years ! You should know... We do what we can with what we have.

FYI, here's the RAC of my team before and after v7

2013.05.01 >> 505 404
2013.05.02 >> 525 118
2013.05.03 >> 510 465
2013.05.04 >>543 912
2013.05.05 >> 489 550
2013.05.06 >> 510 695
2013.05.07 >> 0
2013.05.08 >> 962 063 >> Maintenance
2013.05.09 >> 565 545
2013.05.10 >> 552 424
2013.05.11 >> 502 682
2013.05.12 >> 556 887
2013.05.13 >> 567 531
2013.05.14 >> 517 180
2013.05.15 >> 599 158
2013.05.16 >> 553 868
2013.05.17 >> 575 511
2013.05.18 >> 602 858
2013.05.19 >> 560 806
2013.05.20 >> 578 527
2013.05.21 >> 0
2013.05.22 >> 1 225 668 >> Maintenance
2013.05.23 >> 637 222
2013.05.24 >> 599 316
2013.05.25 >> 613 521
2013.05.26 >> 593 240
2013.05.27 >> 587 166
2013.05.28 >> 0
2013.05.29 >> 1 080 200 >> Maintenance
2013.05.30 >> 559 477
2013.05.31 >> 356 910

Average >> 550 KC


2013.06.01 >> 320 867 >> Migration V7
2013.06.02 >> 442 288
2013.06.03 >> 287 334
2013.06.04 >> 181 137
2013.06.05 >> 308 772
2013.06.06 >> 268 995
2013.06.07 >> 289 123
2013.06.08 >> 268 049
2013.06.09 >> 259 008
2013.06.10 >> 300 288
2013.06.11 >> 264 932
2013.06.12 >> 271 672
2013.06.13 >> 256 083
2013.06.14 >> 273 654
2013.06.15 >> 263 289
2013.06.16 >> 276 272
2013.06.17 >> 259 107
2013.06.18 >> 214 800
2013.06.19 >> 311 603
2013.06.20 >> 389 124
2013.06.21 >> 246 658
2013.06.22 >> 284 413
2013.06.23 >> 283 260
2013.06.24 >> 305 561
2013.06.25 >> 276 445
2013.06.26 >> 260 408
2013.06.27 >> 310 254
2013.06.28 >> 296 944
2013.06.29 >> 325 508
2013.06.30 >> 355 612
2013.07.01 >> 311 340

Average >> 289 KC

Before/After
-47%

I also found that many members who use LUNATICS have not yet made the switch from 0.40 to 0.41
They are not supplied with WU's until they are not switching
This certainly has an influence on the amount of WU's treaties and therefore on the global RAC
____________
Team Founder BRIGADE DU COSMOS




BRIGADE DU COSMOS is proudly sponsored by Zenovia Digital Exchange

Profile {BDC} Thomas DupontProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Dec 11
Posts: 3758
Credit: 1,310,933
RAC: 289
France
Message 1386764 - Posted: 2 Jul 2013, 7:56:12 UTC

PS : however, I think it's a mistake not to accept all types of WU's.
If everyone did the same, we'll never get to stabilize the situation.
And we are here primarily to advance the science. Our RAC is secondary.
____________
Team Founder BRIGADE DU COSMOS




BRIGADE DU COSMOS is proudly sponsored by Zenovia Digital Exchange

bill
Send message
Joined: 16 Jun 99
Posts: 860
Credit: 23,330,929
RAC: 24,068
United States
Message 1386768 - Posted: 2 Jul 2013, 8:05:26 UTC - in response to Message 1386764.

PS : however, I think it's a mistake not to accept all types of WU's.
If everyone did the same, we'll never get to stabilize the situation.
And we are here primarily to advance the science. Our RAC is secondary.


Yep. Plus looking at the project from the
lab's side, nothing is broken.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5811
Credit: 58,722,118
RAC: 48,317
Australia
Message 1386769 - Posted: 2 Jul 2013, 8:08:42 UTC - in response to Message 1386768.


RAC still falling, although it might (finally) be starting to bottom out.
____________
Grant
Darwin NT.

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3411
Credit: 46,462,687
RAC: 7,810
Russia
Message 1386773 - Posted: 2 Jul 2013, 9:05:40 UTC - in response to Message 1386659.
Last modified: 2 Jul 2013, 9:07:35 UTC


I also agree that running v7 on a cpu is now virtually not worth it. v7 has taken this over the tipping point. You are far better running AP on a cpu and getting at least twice the credit per core per hour.



Quite strange conclusion considering elative speeds of stock CPU (and even opt CPU) AP and GPU AP.
I would recommend directly opposite config: to crunch MB on CPU and AP on GPU. At least for ATi. For NV hosts better choice would be MB-only setup of CPU and CUDA opt apps. And avoid CPU AP for both configs.

EDIT: and this recommendation has nothing with definitely broken credits awarding. It bases on relative apps performance per se.
____________

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 4161
Credit: 113,841,370
RAC: 148,252
United States
Message 1386822 - Posted: 2 Jul 2013, 14:23:41 UTC

I see normal AR tasks now providing the ~100 credits per tasks instead of the 50-80 I was seeing initially. While many of my MB CPU only machines have a RAC that is still falling I imagine this will correct itself in a few more weeks.

Granted the credit system used is still rather broken. Perhaps the BOINC team could get a math major to intern for them or something.
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Ingleside
Volunteer developer
Send message
Joined: 4 Feb 03
Posts: 1546
Credit: 4,245,147
RAC: 8,074
Norway
Message 1386827 - Posted: 2 Jul 2013, 14:38:02 UTC - in response to Message 1386253.

In the case of "the New Credit System" implemented by Seti, recognition of personal contribution has been reduced. At present, the indication is that recognition for effort is effectively half that of what it was prior to the new recognition system being employed.

I've no idea how much faster optimized v7 is compared to stock v7 and neither how much faster optimized v6 was compared to stock v6. But to get some numbers, let's say optimized v6 was 2x faster, and optimized v7 is 1.1x faster.

Where's basically 3 ways to give credit:
1: Give same credit/wu even wu is different.
2: Give same credit/second for stock application.
3: Give same credit/second for optimized application.

Let's also compare two identical computers, except one runs stock and the other optimized. Under v6, the stock did 100 wu/day and got 10000 credit/day while optimized did 200 wu/day and got 20000 credit/day.

Let's also say, with stock v7 each wu takes 2x longer to run, meaning with v7 stock does 50 wu/day and optimized 55 wu/day. This means:
If #1: stock gets 5000 credit/day, optimized gets 5500 credit/day.
If #2: stock gets 10000 credit/day, optimized gets 11000 credit/day.
If #3: stock gets 18181 credit/day, optimized gets 20000 credit/day.

On top of this, let's say the two computers ran v6 for 1000 days, meaning the stock had 10 M credits and optimized 20 M credits and let's say the optimized didn't upgrade to v7 but stopped crunching. This means:
If #1: stock needs 2000 days to catch-up.
If #2: stock needs 1000 days to catch-up.
If #3: stock needs 550 days to catch-up.


Atleast to me, #2 is the system what most correctly shows optimized v6 was really 2x faster than stock meaning needs double the time to catch-up with a computer stopped crunching. If #1, stock gets 1/2 the credit as before while optimized gets roughly 1/4 the credit as before. If #3, stock gets roughly 82% more credit as before while optimized has no change. With #3 even someone ran optimized for 2.7 years if they stop running they'll be overtaken by someone running stock the whole time in only 1.4 years and atleast to me this is not a good "thank you for being 2x faster".


But, what about creditNew? As has been mentioned, this gives roughly 10% lower credit for stock application than should, meaning:
If #1: stock gets 4500 credit/day, optimized 4950 credit/day.
If #2: stock gets 9000 credit/day, optimized gets 9900 credit/day.
If #3: stock gets 16364 credit/day, optimized gets 18000 credit/day.

This gives the additional effect if optimized computer doesn't run v7:
If #1: stock needs 2222 days to catch-up.
If #2: stock needs 1111 days to catch-up.
If #3: stock needs 611 days to catch-up.


____________
"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."

Tom*
Send message
Joined: 12 Aug 11
Posts: 114
Credit: 4,810,378
RAC: 4,325
United States
Message 1386856 - Posted: 2 Jul 2013, 19:27:57 UTC

Before everyone talks about V6 stock V6 optimized apps V7 stock and V7 optimized
apps, we need to get a baseline vis a vis stock vs optimized apps both GPU and CPU
Everything following is only related to MB apps

My understanding (please correct me if I am wrong:-)) on day 1 The stock CPU apps for MBV7 are the optimized apps used for V6 plus AVX for even faster Stock MB application and VLAR's were moved to NVIDIA GPU's so we had a triple whammy.

1. V7 apps do more science so take longer across the board CPU and GPU.

2. V7 CPU apps stock are now the old optimized apps for V6 so ALL CPU based systems are now running optimized CPU apps CreditNew now has a New baseline
of NO CPU apps (too soon to calculate needs weeks to stabilize on new application)
but GPU's are all returning many results against each other and not the CPU's

3. VLAR's which only ran on CPU's (for the most part) in V6 and take three times longer on GPU's in V7 are now really confusing CreditNew

Summary - CPU apps in V7 are now twice as fast as in V6 and not processing as many VLAR's. Since the CPU app is used as the baseline in CREDITNEW?? it appears all the GPU's have slowed down plus they are taking longer on each job.

Questions -

1. Since VLAR's have moved from NVIDIA to CPU's have we seen any improvement in
credits for GPU tasks?

2. Are the Optimized Applications for both GPU and CPU in V6 pretty much the same as the stock apps for V7? At least until we get another breakthru in processing speed of Newer optimized applications??

3. Is it true to say that the GPU's are getting less credit per hour of work
on SETI and is it true to say CPU applications are getting more credit per hour of work?

Bottom line
Objectives of V7 SETI

Get rid of Shorties - accomplished for the most part so far at least
Increase optimizations across the board - accomplished
Increase Science by having each workunit start the NITPECKER AutoCorrelation

I am sure there are more these are just my observations

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8408
Credit: 4,125,187
RAC: 1,336
United Kingdom
Message 1386878 - Posted: 2 Jul 2013, 20:54:15 UTC - in response to Message 1386856.

Before everyone talks about V6 stock V6 optimized apps V7 stock and V7 optimized
apps, we need to get a baseline vis a vis stock vs optimized apps both GPU and CPU...

So what and how do we calibrate to something tangible that works across all hardware?...


Cobblestones come unstuck when banged across different projects because they do not directly measure the resources utilised...

As an alternative, one idea could be to count/estimate transistor or bit transitions usefully utilised. That would give useful counts for even non-compute projects. However, there would need to be another count of some kind to reward for such as availability and storage.

And then, for any scheme, how/who for the implementation...?


Happy fast crunchin',
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Josef W. SegurProject donor
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4241
Credit: 1,046,990
RAC: 302
United States
Message 1387022 - Posted: 3 Jul 2013, 4:44:58 UTC - in response to Message 1386856.

...
My understanding (please correct me if I am wrong:-)) on day 1 The stock CPU apps for MBV7 are the optimized apps used for V6 plus AVX for even faster Stock MB application...

The stock Sah v7 CPU app primarily differs from the stock Sah v6 CPU app by the addition of code to do the Autocorrelation search. There were also some AVX functions added and it is linked to a newer version of FFTW which also supports AVX. Both stock CPU apps do have considerable optimization, but that is done in a manner allowing the same application to run well on a wide variety of systems.

The relative speed of the stock apps is certainly pertinent to a discussion of CreditNew, because CPU efficiency is higher than GPU efficiency as Dr. Anderson conceives the term. Two quotes from the CreditNew wiki page are pertinent:

GPUs typically have a higher (10-100X) peak FLOPS than CPUs. However, application efficiency is typically lower (very roughly, 10% for GPUs, 50% for CPUs).

If an application has both CPU and GPU versions, the version normalization mechanism figures out which version is most efficient and uses that to reduce the credit granted to less-efficient versions.

The v6 and v7 CPU apps are about the same speed when doing v6 tasks, with a small advantage to the v7 version on hosts with AVX capability. The added time of Autocorrelations on CPU is somewhat less than a 10% increase averaged over all angle ranges. On GPU the impact of Autocorrelations is much greater, which translates into a serious loss of efficiency as Dr. Anderson conceives it.
Joe

Profile jason_geeProject donor
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 24 Nov 06
Posts: 4978
Credit: 73,335,232
RAC: 16,210
Australia
Message 1387059 - Posted: 3 Jul 2013, 6:16:17 UTC - in response to Message 1387022.
Last modified: 3 Jul 2013, 6:17:01 UTC

...
My understanding (please correct me if I am wrong:-)) on day 1 The stock CPU apps for MBV7 are the optimized apps used for V6 plus AVX for even faster Stock MB application...

The stock Sah v7 CPU app primarily differs from the stock Sah v6 CPU app by the addition of code to do the Autocorrelation search. There were also some AVX functions added and it is linked to a newer version of FFTW which also supports AVX. Both stock CPU apps do have considerable optimization, but that is done in a manner allowing the same application to run well on a wide variety of systems.

The relative speed of the stock apps is certainly pertinent to a discussion of CreditNew, because CPU efficiency is higher than GPU efficiency as Dr. Anderson conceives the term. Two quotes from the CreditNew wiki page are pertinent:

GPUs typically have a higher (10-100X) peak FLOPS than CPUs. However, application efficiency is typically lower (very roughly, 10% for GPUs, 50% for CPUs).

If an application has both CPU and GPU versions, the version normalization mechanism figures out which version is most efficient and uses that to reduce the credit granted to less-efficient versions.

The v6 and v7 CPU apps are about the same speed when doing v6 tasks, with a small advantage to the v7 version on hosts with AVX capability. The added time of Autocorrelations on CPU is somewhat less than a 10% increase averaged over all angle ranges. On GPU the impact of Autocorrelations is much greater, which translates into a serious loss of efficiency as Dr. Anderson conceives it.
Joe


For my two cents in that context, I'm pleased to see Dr A's theories matching reality, since algorithmically both the CPU & GPU autocorelations are order O(nlogn) , however my baseline/reference 'get it working' GPU autcorrelation implementation uses the 4nfft approach, so becomes 4x(nlogn). It's then pretty easy to see how 10% becomes 40%. A tasty Type 2 DCT kernel with attention to max bandwidth and cache locality should improve that handily, once all the dust has settled and other fires extinguished.
____________
"It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change."
Charles Darwin

Profile jason_geeProject donor
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 24 Nov 06
Posts: 4978
Credit: 73,335,232
RAC: 16,210
Australia
Message 1387063 - Posted: 3 Jul 2013, 6:22:34 UTC - in response to Message 1387062.
Last modified: 3 Jul 2013, 6:23:28 UTC

Now, could you express that in kibbles so the kitties could understand what you just said?


how about 'Quick & dirty gets the job done, but perfection[purrrrfection ?] takes a little more time & effort' ?
____________
"It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change."
Charles Darwin

1 · 2 · 3 · 4 . . . 20 · Next

Message boards : Number crunching : Observation of CreditNew Impact (2)

Copyright © 2014 University of California