Oh, hello. I was out of town most of last week on vacation, then Jeff and I were at OSCON 2009 down in San Jose until today. Despite being billed as an open source developer conference we got all kinds of linux sysadmin and mysql tips and tricks from various experts that we may apply towards better diagnosing of system/network/database issues in the future.

That all said, I haven't had the time to catch up on the lengthy discussions here in this forum during my absence. I imagine it has been mostly about our continuing network struggles. This may all become quite moot quite fast as Eric started rolling out the updated scientific analysis configuration, which is an easy knob to turn as we can increase sensitivity, thus improving our science, with the additional happy side benefit of reducing demand on our servers. I think, though, that we have now just reached the limits of that particular knob before getting diminishing returns.

Apparently there were a few servers that needed to be kicked while I was away. Jeff and Eric took care of all that. Mount issues and the like. We also seem to have our new disk arrays set up both at Arecibo and here, so the raw data pipeline should be kicking into full swing again soon. This is good as we're down to our last 10 files that we've been bringing up from the archives (there are a lot more files, but they require the radar blanking software to work in order to be processed, and I haven't gotten around to that yet).

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

I wonder why you are moving to higher sensitivity MB rather than weaning us off MB and onto AP, because I thought AP gave a richer science content. Perhaps it is that we can't actually get any more AP's from Arecibo.

I wonder why you are moving to higher sensitivity MB rather than weaning us off MB and onto AP, because I thought AP gave a richer science content.

AP and MB use the same exact data from Arecibo. However, the science from AP and MB are quite different, so it's not like one is an improvement over the other, and both are doing valid, useful work. AP also processes much faster than we can apparently feed it, so we are still dominated by MB processing.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

what happen to the result-page?, I can't view my pending credit for quite a time.

what happen to the result-page?, I can't view my pending credit for quite a time.

patience; it'll be back soon.... while your waiting for it, play a game like throw out a random # of credits each until it's back on and the day you were the closets; consider that the smartest you've been in the past month. bet against yourself... LOL!

thanks for the update Matt. Although it ain't much of an update.

thanks for the update Matt. Although it ain't much of an update.

Actually, the news from OSCON is pretty interesting. The rest will come.

---

Regarding the results page, I just turned that back on (I guess somebody forgot to turn it back on after the outage).

If trends continue, the result table will be sufficient small enough again that we won't need to turn that off during weekly outage recovery.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

All I know is that last post "Working as Expected (Jul 13 2009)" with 210 replies gave me a headache once I was done reading it. If we can put a man on the moon then we can fix this bandwith database issue. We just need the NASA $1 billion funding. Oh wait....DAMN! In the mean time use the force Matt!



Get crunchin'

Dave

---

Sir,

REF

I think, though, that we have now just reached the limits of that particular knob before getting diminishing returns.


it it gives you 1% better returns and it adds 24 hrs to the crunch time... Turn the knob to 11. :)

---

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

---

Matt's back, Matt's back, YAYYYY we're saved. Don't ever go away again!!!!

Only joking, hope you had a great time and forgot all about us while you were away.

---

PROUD MEMBER OF Team Starfire World BOINC

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.

I think a count would be less stress than showing all of the workunits.. there numbers.. credit.. etc etc etc

Dont you agree?

---

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.

I think it was David Anderson who took out the task count from the host summary pages for that reason as part of a BOINC-wide redesign, at much the same time as Matt said that the database could handle the individual load of people clicking on web pages just fine, it was the bulk load of handling the results (and later the automated load of stats scrapers hitting the website) that caused problems.

Hooray.. I can finally check my pending credit..
Don't ever go on holiday Matt, we need you to look over SEI@HOME...

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.

I think a count would be less stress than showing all of the workunits.. there numbers.. credit.. etc etc etc

Dont you agree?

Actually, no. A count has to go through all of the working of determining which rows to include in the count. This is just as much work as determining which rows to have in the result data set. The only savings is in building and transporting the result data set. Since the result data set is fairly small in this case, the savings are fairly small.

---

BOINC WIKI

Not very big question but why is there still a separate pending credit page accessible from the account page when there is the option of showing just the pending on the tasks page(s)

---

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I download the pending page (right click; save as) then pass it through
this awk script to get total pending as well as by individual computer:

BEGIN { print "\nComputer\tPend\t Total Average Max"; flag=0;}
(flag==1) {flag=0; split($0,a,">"); split(a[2],b,"<");
npend++; no[host]++; gsub(",","",b[1]); tot[host] += b[1]; pend += b[1];
if (mx[host]<b[1]) { mx[host]=b[1];}}
/show_host_detail/ { split($0,a,">"); split(a[3],b,"<"); host= b[1]; flag=1;}
END { j=1; for (x in no) { a[j++]=x;}; j=asort(a);
for (i=1; i<=j; ++i) { x=a[i];
printf("%8d\t%4d\t%9.2f %6.2f %7.2f\n",x,n[x],tot[x],tot[x]/no[x],mx[x]);};
printf(" Total =\t%4d\t%9.2f %6.2f\n\n",npend,pend,pend/npend);}

---

for your information:
the amount of time to display all data regarding the pending credit is MUCH BIGGER then the amount of time needed to count and display total pending credit.

and by the way..
if the current web page has capability to display all records regarding the pending credit, then logically it only need to add few rows of code, to calculate the total pending credit.

however, assuming the huge amount of users, that simultaneously viewing the result page, actually even adding few rows of code, will consume a lot of processing power.

for your information:
the amount of time to display all data regarding the pending credit is MUCH BIGGER then the amount of time needed to count and display total pending credit.

and by the way..
if the current web page has capability to display all records regarding the pending credit, then logically it only need to add few rows of code, to calculate the total pending credit.

however, assuming the huge amount of users, that simultaneously viewing the result page, actually even adding few rows of code, will consume a lot of processing power.

I think it should take less to count than display.

The problem is, I can't test, and I'm pretty sure the various BOINC projects know the effect this has on performance.

That is why I won't state as a fact that there is a time difference.

---