Hello Again (Jul 23 2009)


log in

Advanced search

Message boards : Technical News : Hello Again (Jul 23 2009)

1 · 2 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 920766 - Posted: 23 Jul 2009, 20:21:14 UTC

Oh, hello. I was out of town most of last week on vacation, then Jeff and I were at OSCON 2009 down in San Jose until today. Despite being billed as an open source developer conference we got all kinds of linux sysadmin and mysql tips and tricks from various experts that we may apply towards better diagnosing of system/network/database issues in the future.

That all said, I haven't had the time to catch up on the lengthy discussions here in this forum during my absence. I imagine it has been mostly about our continuing network struggles. This may all become quite moot quite fast as Eric started rolling out the updated scientific analysis configuration, which is an easy knob to turn as we can increase sensitivity, thus improving our science, with the additional happy side benefit of reducing demand on our servers. I think, though, that we have now just reached the limits of that particular knob before getting diminishing returns.

Apparently there were a few servers that needed to be kicked while I was away. Jeff and Eric took care of all that. Mount issues and the like. We also seem to have our new disk arrays set up both at Arecibo and here, so the raw data pipeline should be kicking into full swing again soon. This is good as we're down to our last 10 files that we've been bringing up from the archives (there are a lot more files, but they require the radar blanking software to work in order to be processed, and I haven't gotten around to that yet).

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 22,372,905
RAC: 6,103
United States
Message 920772 - Posted: 23 Jul 2009, 20:46:59 UTC - in response to Message 920766.

I wonder why you are moving to higher sensitivity MB rather than weaning us off MB and onto AP, because I thought AP gave a richer science content. Perhaps it is that we can't actually get any more AP's from Arecibo.

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 920782 - Posted: 23 Jul 2009, 21:07:10 UTC - in response to Message 920772.

I wonder why you are moving to higher sensitivity MB rather than weaning us off MB and onto AP, because I thought AP gave a richer science content.


AP and MB use the same exact data from Arecibo. However, the science from AP and MB are quite different, so it's not like one is an improvement over the other, and both are doing valid, useful work. AP also processes much faster than we can apparently feed it, so we are still dominated by MB processing.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Edywison
Avatar
Send message
Joined: 31 May 09
Posts: 13
Credit: 20,563
RAC: 0
Indonesia
Message 920788 - Posted: 23 Jul 2009, 21:30:52 UTC - in response to Message 920766.

what happen to the result-page?, I can't view my pending credit for quite a time.

zpm
Volunteer tester
Avatar
Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,601,490
RAC: 152
United States
Message 920817 - Posted: 23 Jul 2009, 22:38:41 UTC - in response to Message 920788.

what happen to the result-page?, I can't view my pending credit for quite a time.



patience; it'll be back soon.... while your waiting for it, play a game like throw out a random # of credits each until it's back on and the day you were the closets; consider that the smartest you've been in the past month. bet against yourself... LOL!

thanks for the update Matt. Although it ain't much of an update.

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 920819 - Posted: 23 Jul 2009, 22:40:11 UTC - in response to Message 920817.

thanks for the update Matt. Although it ain't much of an update.

Actually, the news from OSCON is pretty interesting. The rest will come.

____________

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 920824 - Posted: 23 Jul 2009, 22:52:46 UTC - in response to Message 920817.

Regarding the results page, I just turned that back on (I guess somebody forgot to turn it back on after the outage).

If trends continue, the result table will be sufficient small enough again that we won't need to turn that off during weekly outage recovery.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile BigDaddyDave
Avatar
Send message
Joined: 8 Oct 00
Posts: 65
Credit: 6,716,968
RAC: 4,933
United States
Message 920831 - Posted: 23 Jul 2009, 23:05:10 UTC - in response to Message 920766.

All I know is that last post "Working as Expected (Jul 13 2009)" with 210 replies gave me a headache once I was done reading it. If we can put a man on the moon then we can fix this bandwith database issue. We just need the NASA $1 billion funding. Oh wait....DAMN! In the mean time use the force Matt!



Get crunchin'

Dave
____________

Profile cliff west
Send message
Joined: 7 May 01
Posts: 197
Credit: 11,361,285
RAC: 209
United States
Message 920845 - Posted: 23 Jul 2009, 23:37:02 UTC

Sir,

REF

I think, though, that we have now just reached the limits of that particular knob before getting diminishing returns.


it it gives you 1% better returns and it adds 24 hrs to the crunch time... Turn the knob to 11. :)
____________

RAdZer0
Send message
Joined: 20 Jan 03
Posts: 49
Credit: 2,749,129
RAC: 0
United States
Message 920859 - Posted: 24 Jul 2009, 0:20:00 UTC - in response to Message 920845.

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.
____________

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 15,941,065
RAC: 11,993
United States
Message 920863 - Posted: 24 Jul 2009, 0:37:06 UTC

Matt's back, Matt's back, YAYYYY we're saved. Don't ever go away again!!!!

Only joking, hope you had a great time and forgot all about us while you were away.
____________


PROUD MEMBER OF Team Starfire World BOINC

Aurora Borealis
Volunteer tester
Avatar
Send message
Joined: 14 Jan 01
Posts: 2981
Credit: 5,089,440
RAC: 2,014
Canada
Message 920905 - Posted: 24 Jul 2009, 2:22:51 UTC - in response to Message 920859.

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.

RAdZer0
Send message
Joined: 20 Jan 03
Posts: 49
Credit: 2,749,129
RAC: 0
United States
Message 920943 - Posted: 24 Jul 2009, 6:26:38 UTC - in response to Message 920905.

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.



I think a count would be less stress than showing all of the workunits.. there numbers.. credit.. etc etc etc

Dont you agree?
____________

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8633
Credit: 51,583,570
RAC: 48,710
United Kingdom
Message 920972 - Posted: 24 Jul 2009, 7:35:28 UTC - in response to Message 920905.

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.

I think it was David Anderson who took out the task count from the host summary pages for that reason as part of a BOINC-wide redesign, at much the same time as Matt said that the database could handle the individual load of people clicking on web pages just fine, it was the bulk load of handling the results (and later the automated load of stats scrapers hitting the website) that caused problems.

Profile Edywison
Avatar
Send message
Joined: 31 May 09
Posts: 13
Credit: 20,563
RAC: 0
Indonesia
Message 920978 - Posted: 24 Jul 2009, 7:54:30 UTC - in response to Message 920766.

Hooray.. I can finally check my pending credit..
Don't ever go on holiday Matt, we need you to look over SEI@HOME...

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24694
Credit: 522,659
RAC: 19
United States
Message 920998 - Posted: 24 Jul 2009, 11:25:25 UTC - in response to Message 920943.

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.



I think a count would be less stress than showing all of the workunits.. there numbers.. credit.. etc etc etc

Dont you agree?

Actually, no. A count has to go through all of the working of determining which rows to include in the count. This is just as much work as determining which rows to have in the result data set. The only savings is in building and transporting the result data set. Since the result data set is fairly small in this case, the savings are fairly small.
____________


BOINC WIKI

KWSN Sir Clark
Volunteer tester
Avatar
Send message
Joined: 17 Aug 02
Posts: 128
Credit: 218,109
RAC: 25
United Kingdom
Message 921016 - Posted: 24 Jul 2009, 13:03:27 UTC

Not very big question but why is there still a separate pending credit page accessible from the account page when there is the option of showing just the pending on the tasks page(s)
____________

Profile ivan
Volunteer tester
Avatar
Send message
Joined: 5 Mar 01
Posts: 624
Credit: 143,456,491
RAC: 148,956
United Kingdom
Message 921017 - Posted: 24 Jul 2009, 13:09:36 UTC - in response to Message 920859.

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.


I download the pending page (right click; save as) then pass it through
this awk script to get total pending as well as by individual computer:

BEGIN { print "\nComputer\tPend\t Total Average Max"; flag=0;}
(flag==1) {flag=0; split($0,a,">"); split(a[2],b,"<");
npend++; no[host]++; gsub(",","",b[1]); tot[host] += b[1]; pend += b[1];
if (mx[host]<b[1]) { mx[host]=b[1];}}
/show_host_detail/ { split($0,a,">"); split(a[3],b,"<"); host= b[1]; flag=1;}
END { j=1; for (x in no) { a[j++]=x;}; j=asort(a);
for (i=1; i<=j; ++i) { x=a[i];
printf("%8d\t%4d\t%9.2f %6.2f %7.2f\n",x,n[x],tot[x],tot[x]/no[x],mx[x]);};
printf(" Total =\t%4d\t%9.2f %6.2f\n\n",npend,pend,pend/npend);}

____________

Profile Edywison
Avatar
Send message
Joined: 31 May 09
Posts: 13
Credit: 20,563
RAC: 0
Indonesia
Message 921047 - Posted: 24 Jul 2009, 15:38:08 UTC - in response to Message 920998.

for your information:
the amount of time to display all data regarding the pending credit is MUCH BIGGER then the amount of time needed to count and display total pending credit.

and by the way..
if the current web page has capability to display all records regarding the pending credit, then logically it only need to add few rows of code, to calculate the total pending credit.

however, assuming the huge amount of users, that simultaneously viewing the result page, actually even adding few rows of code, will consume a lot of processing power.

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 921065 - Posted: 24 Jul 2009, 16:11:41 UTC - in response to Message 921047.

for your information:
the amount of time to display all data regarding the pending credit is MUCH BIGGER then the amount of time needed to count and display total pending credit.

and by the way..
if the current web page has capability to display all records regarding the pending credit, then logically it only need to add few rows of code, to calculate the total pending credit.

however, assuming the huge amount of users, that simultaneously viewing the result page, actually even adding few rows of code, will consume a lot of processing power.


I think it should take less to count than display.

The problem is, I can't test, and I'm pretty sure the various BOINC projects know the effect this has on performance.

That is why I won't state as a fact that there is a time difference.
____________

1 · 2 · Next

Message boards : Technical News : Hello Again (Jul 23 2009)

Copyright © 2014 University of California