Data Laundering (Oct 12 2009)

Message boards : Technical News : Data Laundering (Oct 12 2009)

Author	Message
Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 939469 - Posted: 12 Oct 2009, 23:02:21 UTC
	The latest software blanking tests were also a success, so we'll start putting older pre-hardware-blanked data into production, now that we can remove the radar. Yay! May take a few days to rev up this engine. Meanwhile Eric has been making progress on the "zone RFI" rejection software/algorithms, so we can start getting rid of the garbage that makes up our current "top candidates." The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors! - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 939469 ·

Pappa Volunteer tester Send message Joined: 9 Jan 00 Posts: 2562 Credit: 12,301,681 RAC: 0	Message 939483 - Posted: 13 Oct 2009, 0:04:08 UTC - in response to Message 939469.
	The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors! - Matt In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup. ____________ Please consider a Donation to the Seti Project.
	ID: 939483 ·

Gary Charpentier Volunteer tester Send message Joined: 25 Dec 00 Posts: 10331 Credit: 5,103,953 RAC: 6,273	Message 939489 - Posted: 13 Oct 2009, 0:19:36 UTC - in response to Message 939483.
	The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors! - Matt In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup. Good idea, except I'm sure the DB is a bit big for backups on CD. Perhaps a couple of removable TB drives though. Thanks for the update Matt ____________
	ID: 939489 ·

Berserker Volunteer tester Send message Joined: 2 Jun 99 Posts: 105 Credit: 5,353,952 RAC: 385	Message 939496 - Posted: 13 Oct 2009, 0:42:54 UTC - in response to Message 939483.
	In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup. That (usually) works for MyISAM tables, but it's a bit more risky with InnoDB tables (which I think BOINC prefers for throughput reasons). Also, if you install a DB software upgrade which makes some important internal structural change, your backup - and even the live tables - might be rendered useless (though this is only likely with a major version upgrade). ____________ Stats site - http://www.teamocuk.co.uk - still alive and (just about) kicking.
	ID: 939496 ·

Speedy Volunteer tester Send message Joined: 26 Jun 04 Posts: 579 Credit: 3,490,077 RAC: 8,752	Message 939505 - Posted: 13 Oct 2009, 1:24:20 UTC Last modified: 13 Oct 2009, 1:30:31 UTC
	The latest software blanking tests were also a success, so we'll start putting older pre-hardware-blanked data into production This is great news. Does this mean that there will be more ap work so that people that like ap work can crunch that instead of mb work? Or have I missed the point as to why the demand for ap work is a lot higher? Thanks feel the update. New data that we will be processing soon will it be around the same processing speed or will there be a slight speed increase due to the angle it was recordered on? ____________ Live in NZ y not join Smile City?
	ID: 939505 ·

CryptokiD Send message Joined: 2 Dec 00 Posts: 134 Credit: 2,814,936 RAC: 0	Message 939510 - Posted: 13 Oct 2009, 1:33:56 UTC
	matt, as always, thankyou for the update. i always enjoy reading them. can i axk you a question. i have heard talk of seti possibly running out of work unless the blanking deal gets worked out. if you by chance do not get it worked out, is there any rough timeframe for when the work will run out. and if you do get the radar blanking deal working and fully operational, where would that put us in a timeframe for running out of data? reason i axk is most my machines only run a few hours per day. i have my work queue set to 2 days on the temp machines. if were going to run out od workunits anytime soon i would propbably bump this up to 10. my cuda machine is the only one that really runs 24/7.
	ID: 939510 ·

[seti.international] Philip J. Fry Volunteer tester Send message Joined: 6 Apr 07 Posts: 6571 Credit: 50,818,961 RAC: 20,438	Message 939597 - Posted: 13 Oct 2009, 10:13:09 UTC
	After the todays maintenance, we will see again the task lists? And maybe also the pending credit list? ____________ >Das Deutsche Cafe. The German Cafe.<
	ID: 939597 ·

Gundolf Jahn Send message Joined: 19 Sep 00 Posts: 3076 Credit: 341,886 RAC: 96	Message 939602 - Posted: 13 Oct 2009, 11:04:13 UTC - in response to Message 939597.
	After the todays maintenance, we will see again the task lists? And maybe also the pending credit list? That depends, as Matt said: The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). Gruß, Gundolf
	ID: 939602 ·

[seti.international] Philip J. Fry Volunteer tester Send message Joined: 6 Apr 07 Posts: 6571 Credit: 50,818,961 RAC: 20,438	Message 939638 - Posted: 13 Oct 2009, 13:59:14 UTC - in response to Message 939602.
	After the todays maintenance, we will see again the task lists? And maybe also the pending credit list? That depends, as Matt said: The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). Gruß, Gundolf I have no big knowledge about the things behind the curtain. I didn't know that this should mean this. ____________ >Das Deutsche Cafe. The German Cafe.<
	ID: 939638 ·

Biffa Volunteer tester Send message Joined: 27 Oct 99 Posts: 41 Credit: 22,388,169 RAC: 1,242	Message 939692 - Posted: 14 Oct 2009, 1:56:56 UTC
	Yep I'd say the replica database is where all that stuff runs to keep the load down on the main db. ____________
	ID: 939692 ·

Johnney Guinness Volunteer tester Send message Joined: 11 Sep 06 Posts: 2944 Credit: 2,305,966 RAC: 330	Message 939707 - Posted: 14 Oct 2009, 2:25:51 UTC - in response to Message 939469.
	Matt, Thats really great news. This is science progress, SETI science progress. It gives me great hope in my heart to believe that yourself and Eric are getting to a stage where real data analysis can be carried out. Gives me great hope! Thank you for your tireless efforts Matt, John. ____________
	ID: 939707 ·

DJStarfox Send message Joined: 23 May 01 Posts: 1040 Credit: 485,117 RAC: 71	Message 939725 - Posted: 14 Oct 2009, 3:18:30 UTC Last modified: 14 Oct 2009, 3:18:41 UTC
	Yeah, now SETI is back up after a long outage, but no replica DB. :( They must have spent all day working on it....
	ID: 939725 ·

edjcox Send message Joined: 20 May 99 Posts: 49 Credit: 3,122,385 RAC: 2,400	Message 939728 - Posted: 14 Oct 2009, 3:36:46 UTC
	Matt: Thanks for a decade of work on SETI. Realizing your overly busy can you point me to location where I could get a Diagram of SETI's data processing/flow? I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system. I'm sure a response may be awhile so don't worry if it takes some time to respond. Thank you, E J Cox ____________
	ID: 939728 ·

C. Barbie Volunteer tester Send message Joined: 1 Jun 99 Posts: 7 Credit: 5,374,706 RAC: 11,506	Message 939764 - Posted: 14 Oct 2009, 6:25:07 UTC - in response to Message 939728.
	Matt: Thanks for a decade of work on SETI. Realizing your overly busy can you point me to location where I could get a Diagram of SETI's data processing/flow? I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system. I'm sure a response may be awhile so don't worry if it takes some time to respond. Thank you, E J Cox Try this... http://setiathome.berkeley.edu/forum_thread.php?id=54748 ____________
	ID: 939764 ·

52 Aces Send message Joined: 7 Jan 02 Posts: 494 Credit: 11,300,589 RAC: 3,595	Message 939794 - Posted: 14 Oct 2009, 11:38:55 UTC - in response to Message 939764. Last modified: 14 Oct 2009, 11:50:09 UTC
	where I could get a Diagram of SETI's data processing/flow? I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system. For a very high level overview, see the diagrams in this write-up --> http://setiathome.berkeley.edu/sah_papers/cacm.php or this. Also, some good video explanations at the 10 year meeting and other stuff here --> http://setiathome.berkeley.edu/sah_about.php If you want deeper details, esp on implimentation, reading the source code and looking at the mysql layout will get you the quickest answer. Boinc -- http://boinc.berkeley.edu/trac/wiki/SourceCode S@H -- http://setiathome.berkeley.edu/sah_porting.php
	ID: 939794 ·

[seti.international] Philip J. Fry Volunteer tester Send message Joined: 6 Apr 07 Posts: 6571 Credit: 50,818,961 RAC: 20,438	Message 939798 - Posted: 14 Oct 2009, 11:51:01 UTC
	I know, that the task lists come from the replica DB.. ;-) But I didn't understood that Matt meant, that he will enable again the task lists.. BTW. I was interested about other GPUs, for to decide which will be the next for me.. But no joy, task lists are still disabled. I know it's a hard work to manage a lot of servers. Matt, good luck! ____________ >Das Deutsche Cafe. The German Cafe.<
	ID: 939798 ·

Message boards : Technical News : Data Laundering (Oct 12 2009)