Hello Again (Jul 23 2009)

Message boards : Technical News : Hello Again (Jul 23 2009)

Author	Message
Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 920766 - Posted: 23 Jul 2009, 20:21:14 UTC
	Oh, hello. I was out of town most of last week on vacation, then Jeff and I were at OSCON 2009 down in San Jose until today. Despite being billed as an open source developer conference we got all kinds of linux sysadmin and mysql tips and tricks from various experts that we may apply towards better diagnosing of system/network/database issues in the future. That all said, I haven't had the time to catch up on the lengthy discussions here in this forum during my absence. I imagine it has been mostly about our continuing network struggles. This may all become quite moot quite fast as Eric started rolling out the updated scientific analysis configuration, which is an easy knob to turn as we can increase sensitivity, thus improving our science, with the additional happy side benefit of reducing demand on our servers. I think, though, that we have now just reached the limits of that particular knob before getting diminishing returns. Apparently there were a few servers that needed to be kicked while I was away. Jeff and Eric took care of all that. Mount issues and the like. We also seem to have our new disk arrays set up both at Arecibo and here, so the raw data pipeline should be kicking into full swing again soon. This is good as we're down to our last 10 files that we've been bringing up from the archives (there are a lot more files, but they require the radar blanking software to work in order to be processed, and I haven't gotten around to that yet). - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 920766 ·

PhonAcq Send message Joined: 14 Apr 01 Posts: 1620 Credit: 20,856,091 RAC: 4,477	Message 920772 - Posted: 23 Jul 2009, 20:46:59 UTC - in response to Message 920766.
	I wonder why you are moving to higher sensitivity MB rather than weaning us off MB and onto AP, because I thought AP gave a richer science content. Perhaps it is that we can't actually get any more AP's from Arecibo.
	ID: 920772 ·

Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 920782 - Posted: 23 Jul 2009, 21:07:10 UTC - in response to Message 920772.
	I wonder why you are moving to higher sensitivity MB rather than weaning us off MB and onto AP, because I thought AP gave a richer science content. AP and MB use the same exact data from Arecibo. However, the science from AP and MB are quite different, so it's not like one is an improvement over the other, and both are doing valid, useful work. AP also processes much faster than we can apparently feed it, so we are still dominated by MB processing. - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 920782 ·

Edywison Send message Joined: 31 May 09 Posts: 13 Credit: 20,563 RAC: 0	Message 920788 - Posted: 23 Jul 2009, 21:30:52 UTC - in response to Message 920766.
	what happen to the result-page?, I can't view my pending credit for quite a time.
	ID: 920788 ·

zpm Volunteer tester Send message Joined: 25 Apr 08 Posts: 284 Credit: 975,677 RAC: 0	Message 920817 - Posted: 23 Jul 2009, 22:38:41 UTC - in response to Message 920788.
	what happen to the result-page?, I can't view my pending credit for quite a time. patience; it'll be back soon.... while your waiting for it, play a game like throw out a random # of credits each until it's back on and the day you were the closets; consider that the smartest you've been in the past month. bet against yourself... LOL! thanks for the update Matt. Although it ain't much of an update.
	ID: 920817 ·

1mp0£173 Volunteer tester Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0	Message 920819 - Posted: 23 Jul 2009, 22:40:11 UTC - in response to Message 920817.
	thanks for the update Matt. Although it ain't much of an update. Actually, the news from OSCON is pretty interesting. The rest will come. ____________
	ID: 920819 ·

Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 920824 - Posted: 23 Jul 2009, 22:52:46 UTC - in response to Message 920817.
	Regarding the results page, I just turned that back on (I guess somebody forgot to turn it back on after the outage). If trends continue, the result table will be sufficient small enough again that we won't need to turn that off during weekly outage recovery. - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 920824 ·

BigDaddyDave Send message Joined: 8 Oct 00 Posts: 64 Credit: 4,877,690 RAC: 3,401	Message 920831 - Posted: 23 Jul 2009, 23:05:10 UTC - in response to Message 920766.
	All I know is that last post "Working as Expected (Jul 13 2009)" with 210 replies gave me a headache once I was done reading it. If we can put a man on the moon then we can fix this bandwith database issue. We just need the NASA $1 billion funding. Oh wait....DAMN! In the mean time use the force Matt! Get crunchin' Dave ____________
	ID: 920831 ·

cliff west Send message Joined: 7 May 01 Posts: 196 Credit: 10,409,706 RAC: 4,956	Message 920845 - Posted: 23 Jul 2009, 23:37:02 UTC
	Sir, REF I think, though, that we have now just reached the limits of that particular knob before getting diminishing returns. it it gives you 1% better returns and it adds 24 hrs to the crunch time... Turn the knob to 11. :) ____________
	ID: 920845 ·

RAdZer0 Send message Joined: 20 Jan 03 Posts: 49 Credit: 2,749,129 RAC: 0	Message 920859 - Posted: 24 Jul 2009, 0:20:00 UTC - in response to Message 920845.
	Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus. ____________
	ID: 920859 ·

perryjay Send message Joined: 20 Aug 02 Posts: 3359 Credit: 9,611,335 RAC: 10,023	Message 920863 - Posted: 24 Jul 2009, 0:37:06 UTC
	Matt's back, Matt's back, YAYYYY we're saved. Don't ever go away again!!!! Only joking, hope you had a great time and forgot all about us while you were away. ____________ PROUD MEMBER OF Team Starfire World BOINC
	ID: 920863 ·

Aurora Borealis Volunteer tester Send message Joined: 14 Jan 01 Posts: 2975 Credit: 4,228,080 RAC: 2,727	Message 920905 - Posted: 24 Jul 2009, 2:22:51 UTC - in response to Message 920859.
	Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus. I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available.
	ID: 920905 ·

RAdZer0 Send message Joined: 20 Jan 03 Posts: 49 Credit: 2,749,129 RAC: 0	Message 920943 - Posted: 24 Jul 2009, 6:26:38 UTC - in response to Message 920905.
	Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus. I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available. I think a count would be less stress than showing all of the workunits.. there numbers.. credit.. etc etc etc Dont you agree? ____________
	ID: 920943 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7453 Credit: 39,544,877 RAC: 32,252	Message 920972 - Posted: 24 Jul 2009, 7:35:28 UTC - in response to Message 920905.
	Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus. I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available. I think it was David Anderson who took out the task count from the host summary pages for that reason as part of a BOINC-wide redesign, at much the same time as Matt said that the database could handle the individual load of people clicking on web pages just fine, it was the bulk load of handling the results (and later the automated load of stats scrapers hitting the website) that caused problems.
	ID: 920972 ·

Edywison Send message Joined: 31 May 09 Posts: 13 Credit: 20,563 RAC: 0	Message 920978 - Posted: 24 Jul 2009, 7:54:30 UTC - in response to Message 920766.
	Hooray.. I can finally check my pending credit.. Don't ever go on holiday Matt, we need you to look over SEI@HOME...
	ID: 920978 ·

John McLeod VII Volunteer developer Volunteer tester Send message Joined: 15 Jul 99 Posts: 22343 Credit: 437,061 RAC: 263	Message 920998 - Posted: 24 Jul 2009, 11:25:25 UTC - in response to Message 920943.
	Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus. I recall someone (Matt??) saying that keeping a running WU count on all the accounts was too hard on the database which is why it is no longer available. I think a count would be less stress than showing all of the workunits.. there numbers.. credit.. etc etc etc Dont you agree? Actually, no. A count has to go through all of the working of determining which rows to include in the count. This is just as much work as determining which rows to have in the result data set. The only savings is in building and transporting the result data set. Since the result data set is fairly small in this case, the savings are fairly small. ____________ BOINC WIKI
	ID: 920998 ·

KWSN Sir Clark Volunteer tester Send message Joined: 17 Aug 02 Posts: 127 Credit: 178,674 RAC: 64	Message 921016 - Posted: 24 Jul 2009, 13:03:27 UTC
	Not very big question but why is there still a separate pending credit page accessible from the account page when there is the option of showing just the pending on the tasks page(s) ____________
	ID: 921016 ·

ivan Volunteer tester Send message Joined: 5 Mar 01 Posts: 381 Credit: 93,014,724 RAC: 118,613	Message 921017 - Posted: 24 Jul 2009, 13:09:36 UTC - in response to Message 920859.
	Matt. Just a small tip, i dont know much of the stuff on your end as im a regular tech. But i only use the pending credit page to see the total number not a per result page. Which im sure alot of people only look at the total also. Maybe you can make a total than you can click thru to see individual results. I dont know what kinda power it takes to pull up those pages on some people (mine is 26,000 pending) But if it lowers the load on things by 1% its a bonus. I download the pending page (right click; save as) then pass it through this awk script to get total pending as well as by individual computer: BEGIN { print "\nComputer\tPend\t Total Average Max"; flag=0;} (flag==1) {flag=0; split($0,a,">"); split(a[2],b,"<"); npend++; no[host]++; gsub(",","",b[1]); tot[host] += b[1]; pend += b[1]; if (mx[host]<b[1]) { mx[host]=b[1];}} /show_host_detail/ { split($0,a,">"); split(a[3],b,"<"); host= b[1]; flag=1;} END { j=1; for (x in no) { a[j++]=x;}; j=asort(a); for (i=1; i<=j; ++i) { x=a[i]; printf("%8d\t%4d\t%9.2f %6.2f %7.2f\n",x,n[x],tot[x],tot[x]/no[x],mx[x]);}; printf(" Total =\t%4d\t%9.2f %6.2f\n\n",npend,pend,pend/npend);} ____________
	ID: 921017 ·

Edywison Send message Joined: 31 May 09 Posts: 13 Credit: 20,563 RAC: 0	Message 921047 - Posted: 24 Jul 2009, 15:38:08 UTC - in response to Message 920998.
	for your information: the amount of time to display all data regarding the pending credit is MUCH BIGGER then the amount of time needed to count and display total pending credit. and by the way.. if the current web page has capability to display all records regarding the pending credit, then logically it only need to add few rows of code, to calculate the total pending credit. however, assuming the huge amount of users, that simultaneously viewing the result page, actually even adding few rows of code, will consume a lot of processing power.
	ID: 921047 ·

1mp0£173 Volunteer tester Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0	Message 921065 - Posted: 24 Jul 2009, 16:11:41 UTC - in response to Message 921047.
	for your information: the amount of time to display all data regarding the pending credit is MUCH BIGGER then the amount of time needed to count and display total pending credit. and by the way.. if the current web page has capability to display all records regarding the pending credit, then logically it only need to add few rows of code, to calculate the total pending credit. however, assuming the huge amount of users, that simultaneously viewing the result page, actually even adding few rows of code, will consume a lot of processing power. I think it should take less to count than display. The problem is, I can't test, and I'm pretty sure the various BOINC projects know the effect this has on performance. That is why I won't state as a fact that there is a time difference. ____________
	ID: 921065 ·

1 · 2 · Next

Message boards : Technical News : Hello Again (Jul 23 2009)