Hello Again (Jul 23 2009)

Message boards : Technical News : Hello Again (Jul 23 2009)
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 920766 - Posted: 23 Jul 2009, 20:21:14 UTC

Oh, hello. I was out of town most of last week on vacation, then Jeff and I were at OSCON 2009 down in San Jose until today. Despite being billed as an open source developer conference we got all kinds of linux sysadmin and mysql tips and tricks from various experts that we may apply towards better diagnosing of system/network/database issues in the future.

That all said, I haven't had the time to catch up on the lengthy discussions here in this forum during my absence. I imagine it has been mostly about our continuing network struggles. This may all become quite moot quite fast as Eric started rolling out the updated scientific analysis configuration, which is an easy knob to turn as we can increase sensitivity, thus improving our science, with the additional happy side benefit of reducing demand on our servers. I think, though, that we have now just reached the limits of that particular knob before getting diminishing returns.

Apparently there were a few servers that needed to be kicked while I was away. Jeff and Eric took care of all that. Mount issues and the like. We also seem to have our new disk arrays set up both at Arecibo and here, so the raw data pipeline should be kicking into full swing again soon. This is good as we're down to our last 10 files that we've been bringing up from the archives (there are a lot more files, but they require the radar blanking software to work in order to be processed, and I haven't gotten around to that yet).

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 920766 · Report as offensive
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,658,217
RAC: 1
United States
Message 920772 - Posted: 23 Jul 2009, 20:46:59 UTC - in response to Message 920766.  

I wonder why you are moving to higher sensitivity MB rather than weaning us off MB and onto AP, because I thought AP gave a richer science content. Perhaps it is that we can't actually get any more AP's from Arecibo.
ID: 920772 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 920782 - Posted: 23 Jul 2009, 21:07:10 UTC - in response to Message 920772.  

I wonder why you are moving to higher sensitivity MB rather than weaning us off MB and onto AP, because I thought AP gave a richer science content.


AP and MB use the same exact data from Arecibo. However, the science from AP and MB are quite different, so it's not like one is an improvement over the other, and both are doing valid, useful work. AP also processes much faster than we can apparently feed it, so we are still dominated by MB processing.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 920782 · Report as offensive
Profile Edywison
Avatar

Send message
Joined: 31 May 09
Posts: 13
Credit: 20,563
RAC: 0
Indonesia
Message 920788 - Posted: 23 Jul 2009, 21:30:52 UTC - in response to Message 920766.  

what happen to the result-page?, I can't view my pending credit for quite a time.
ID: 920788 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 920817 - Posted: 23 Jul 2009, 22:38:41 UTC - in response to Message 920788.  

what happen to the result-page?, I can't view my pending credit for quite a time.



patience; it'll be back soon.... while your waiting for it, play a game like throw out a random # of credits each until it's back on and the day you were the closets; consider that the smartest you've been in the past month. bet against yourself... LOL!

thanks for the update Matt. Although it ain't much of an update.
ID: 920817 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 920819 - Posted: 23 Jul 2009, 22:40:11 UTC - in response to Message 920817.  

thanks for the update Matt. Although it ain't much of an update.

Actually, the news from OSCON is pretty interesting. The rest will come.

ID: 920819 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 920824 - Posted: 23 Jul 2009, 22:52:46 UTC - in response to Message 920817.  

Regarding the results page, I just turned that back on (I guess somebody forgot to turn it back on after the outage).

If trends continue, the result table will be sufficient small enough again that we won't need to turn that off during weekly outage recovery.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 920824 · Report as offensive
Profile BigDaddyDave
Avatar

Send message
Joined: 8 Oct 00
Posts: 67
Credit: 20,337,995
RAC: 15
United States
Message 920831 - Posted: 23 Jul 2009, 23:05:10 UTC - in response to Message 920766.  

All I know is that last post "Working as Expected (Jul 13 2009)" with 210 replies gave me a headache once I was done reading it. If we can put a man on the moon then we can fix this bandwith database issue. We just need the NASA $1 billion funding. Oh wait....DAMN! In the mean time use the force Matt!



Get crunchin'

Dave
ID: 920831 · Report as offensive
Profile cliff west

Send message
Joined: 7 May 01
Posts: 211
Credit: 16,180,728
RAC: 15
United States
Message 920845 - Posted: 23 Jul 2009, 23:37:02 UTC

Sir,

REF

I think, though, that we have now just reached the limits of that particular knob before getting diminishing returns.


it it gives you 1% better returns and it adds 24 hrs to the crunch time... Turn the knob to 11. :)
ID: 920845 · Report as offensive
RAdZer0

Send message
Joined: 20 Jan 03
Posts: 49
Credit: 2,749,129
RAC: 0
United States
Message 920859 - Posted: 24 Jul 2009, 0:20:00 UTC - in response to Message 920845.  

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.
ID: 920859 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 920863 - Posted: 24 Jul 2009, 0:37:06 UTC

Matt's back, Matt's back, YAYYYY we're saved. Don't ever go away again!!!!

Only joking, hope you had a great time and forgot all about us while you were away.


PROUD MEMBER OF Team Starfire World BOINC
ID: 920863 · Report as offensive
Aurora Borealis
Volunteer tester
Avatar

Send message
Joined: 14 Jan 01
Posts: 3075
Credit: 5,631,463
RAC: 0
Canada
Message 920905 - Posted: 24 Jul 2009, 2:22:51 UTC - in response to Message 920859.  

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.
ID: 920905 · Report as offensive
RAdZer0

Send message
Joined: 20 Jan 03
Posts: 49
Credit: 2,749,129
RAC: 0
United States
Message 920943 - Posted: 24 Jul 2009, 6:26:38 UTC - in response to Message 920905.  

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.



I think a count would be less stress than showing all of the workunits.. there numbers.. credit.. etc etc etc

Dont you agree?
ID: 920943 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14645
Credit: 200,643,578
RAC: 874
United Kingdom
Message 920972 - Posted: 24 Jul 2009, 7:35:28 UTC - in response to Message 920905.  

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.

I think it was David Anderson who took out the task count from the host summary pages for that reason as part of a BOINC-wide redesign, at much the same time as Matt said that the database could handle the individual load of people clicking on web pages just fine, it was the bulk load of handling the results (and later the automated load of stats scrapers hitting the website) that caused problems.
ID: 920972 · Report as offensive
Profile Edywison
Avatar

Send message
Joined: 31 May 09
Posts: 13
Credit: 20,563
RAC: 0
Indonesia
Message 920978 - Posted: 24 Jul 2009, 7:54:30 UTC - in response to Message 920766.  

Hooray.. I can finally check my pending credit..
Don't ever go on holiday Matt, we need you to look over SEI@HOME...
ID: 920978 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 920998 - Posted: 24 Jul 2009, 11:25:25 UTC - in response to Message 920943.  

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.

I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.



I think a count would be less stress than showing all of the workunits.. there numbers.. credit.. etc etc etc

Dont you agree?

Actually, no. A count has to go through all of the working of determining which rows to include in the count. This is just as much work as determining which rows to have in the result data set. The only savings is in building and transporting the result data set. Since the result data set is fairly small in this case, the savings are fairly small.


BOINC WIKI
ID: 920998 · Report as offensive
KWSN Sir Clark
Volunteer tester

Send message
Joined: 17 Aug 02
Posts: 139
Credit: 1,002,493
RAC: 8
United Kingdom
Message 921016 - Posted: 24 Jul 2009, 13:03:27 UTC

Not very big question but why is there still a separate pending credit page accessible from the account page when there is the option of showing just the pending on the tasks page(s)
ID: 921016 · Report as offensive
Profile ivan
Volunteer tester
Avatar

Send message
Joined: 5 Mar 01
Posts: 783
Credit: 348,560,338
RAC: 223
United Kingdom
Message 921017 - Posted: 24 Jul 2009, 13:09:36 UTC - in response to Message 920859.  

Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus.


I download the pending page (right click; save as) then pass it through
this awk script to get total pending as well as by individual computer:

BEGIN { print "\nComputer\tPend\t Total Average Max"; flag=0;}
(flag==1) {flag=0; split($0,a,">"); split(a[2],b,"<");
npend++; no[host]++; gsub(",","",b[1]); tot[host] += b[1]; pend += b[1];
if (mx[host]<b[1]) { mx[host]=b[1];}}
/show_host_detail/ { split($0,a,">"); split(a[3],b,"<"); host= b[1]; flag=1;}
END { j=1; for (x in no) { a[j++]=x;}; j=asort(a);
for (i=1; i<=j; ++i) { x=a[i];
printf("%8d\t%4d\t%9.2f %6.2f %7.2f\n",x,n[x],tot[x],tot[x]/no[x],mx[x]);};
printf(" Total =\t%4d\t%9.2f %6.2f\n\n",npend,pend,pend/npend);}

ID: 921017 · Report as offensive
Profile Edywison
Avatar

Send message
Joined: 31 May 09
Posts: 13
Credit: 20,563
RAC: 0
Indonesia
Message 921047 - Posted: 24 Jul 2009, 15:38:08 UTC - in response to Message 920998.  

for your information:
the amount of time to display all data regarding the pending credit is MUCH BIGGER then the amount of time needed to count and display total pending credit.

and by the way..
if the current web page has capability to display all records regarding the pending credit, then logically it only need to add few rows of code, to calculate the total pending credit.

however, assuming the huge amount of users, that simultaneously viewing the result page, actually even adding few rows of code, will consume a lot of processing power.
ID: 921047 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 921065 - Posted: 24 Jul 2009, 16:11:41 UTC - in response to Message 921047.  

for your information:
the amount of time to display all data regarding the pending credit is MUCH BIGGER then the amount of time needed to count and display total pending credit.

and by the way..
if the current web page has capability to display all records regarding the pending credit, then logically it only need to add few rows of code, to calculate the total pending credit.

however, assuming the huge amount of users, that simultaneously viewing the result page, actually even adding few rows of code, will consume a lot of processing power.


I think it should take less to count than display.

The problem is, I can't test, and I'm pretty sure the various BOINC projects know the effect this has on performance.

That is why I won't state as a fact that there is a time difference.
ID: 921065 · Report as offensive
1 · 2 · Next

Message boards : Technical News : Hello Again (Jul 23 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.