Message boards :
Technical News :
Data Laundering (Oct 12 2009)
Message board moderation
Author | Message |
---|---|
Matt Lebofsky Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 |
The latest software blanking tests were also a success, so we'll start putting older pre-hardware-blanked data into production, now that we can remove the radar. Yay! May take a few days to rev up this engine. Meanwhile Eric has been making progress on the "zone RFI" rejection software/algorithms, so we can start getting rid of the garbage that makes up our current "top candidates." The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors! - Matt -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
Pappa Send message Joined: 9 Jan 00 Posts: 2562 Credit: 12,301,681 RAC: 0 |
The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors! In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup. Please consider a Donation to the Seti Project. |
Gary Charpentier Send message Joined: 25 Dec 00 Posts: 30925 Credit: 53,134,872 RAC: 32 |
The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors! Good idea, except I'm sure the DB is a bit big for backups on CD. Perhaps a couple of removable TB drives though. Thanks for the update Matt |
Berserker Send message Joined: 2 Jun 99 Posts: 105 Credit: 5,440,087 RAC: 0 |
In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup. That (usually) works for MyISAM tables, but it's a bit more risky with InnoDB tables (which I think BOINC prefers for throughput reasons). Also, if you install a DB software upgrade which makes some important internal structural change, your backup - and even the live tables - might be rendered useless (though this is only likely with a major version upgrade). Stats site - http://www.teamocuk.co.uk - still alive and (just about) kicking. |
Speedy Send message Joined: 26 Jun 04 Posts: 1643 Credit: 12,921,799 RAC: 89 |
The latest software blanking tests were also a success, so we'll start putting older pre-hardware-blanked data into production This is great news. Does this mean that there will be more ap work so that people that like ap work can crunch that instead of mb work? Or have I missed the point as to why the demand for ap work is a lot higher? Thanks feel the update. New data that we will be processing soon will it be around the same processing speed or will there be a slight speed increase due to the angle it was recordered on? |
CryptokiD Send message Joined: 2 Dec 00 Posts: 150 Credit: 3,216,632 RAC: 0 |
matt, as always, thankyou for the update. i always enjoy reading them. can i axk you a question. i have heard talk of seti possibly running out of work unless the blanking deal gets worked out. if you by chance do not get it worked out, is there any rough timeframe for when the work will run out. and if you do get the radar blanking deal working and fully operational, where would that put us in a timeframe for running out of data? reason i axk is most my machines only run a few hours per day. i have my work queue set to 2 days on the temp machines. if were going to run out od workunits anytime soon i would propbably bump this up to 10. my cuda machine is the only one that really runs 24/7. |
Dirk Sadowski Send message Joined: 6 Apr 07 Posts: 7105 Credit: 147,663,825 RAC: 5 |
After the todays maintenance, we will see again the task lists? And maybe also the pending credit list? |
Gundolf Jahn Send message Joined: 19 Sep 00 Posts: 3184 Credit: 446,358 RAC: 0 |
After the todays maintenance, we will see again the task lists? That depends, as Matt said: The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). Gruß, Gundolf |
Dirk Sadowski Send message Joined: 6 Apr 07 Posts: 7105 Credit: 147,663,825 RAC: 5 |
After the todays maintenance, we will see again the task lists? I have no big knowledge about the things behind the curtain. I didn't know that this should mean this. |
Biffa Send message Joined: 27 Oct 99 Posts: 41 Credit: 22,750,323 RAC: 0 |
Yep I'd say the replica database is where all that stuff runs to keep the load down on the main db. |
Johnney Guinness Send message Joined: 11 Sep 06 Posts: 3093 Credit: 2,652,287 RAC: 0 |
Matt, Thats really great news. This is science progress, SETI science progress. It gives me great hope in my heart to believe that yourself and Eric are getting to a stage where real data analysis can be carried out. Gives me great hope! Thank you for your tireless efforts Matt, John. |
DJStarfox Send message Joined: 23 May 01 Posts: 1066 Credit: 1,226,053 RAC: 2 |
Yeah, now SETI is back up after a long outage, but no replica DB. :( They must have spent all day working on it.... |
edjcox Send message Joined: 20 May 99 Posts: 96 Credit: 5,878,353 RAC: 0 |
Matt: Thanks for a decade of work on SETI. Realizing your overly busy can you point me to location where I could get a Diagram of SETI's data processing/flow? I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system. I'm sure a response may be awhile so don't worry if it takes some time to respond. Thank you, E J Cox Never engage stupid people at their level, they then have the home court advantage..... |
C. Barbie Send message Joined: 1 Jun 99 Posts: 7 Credit: 9,256,876 RAC: 0 |
Matt: Try this... http://setiathome.berkeley.edu/forum_thread.php?id=54748 |
52 Aces Send message Joined: 7 Jan 02 Posts: 497 Credit: 14,261,068 RAC: 67 |
where I could get a Diagram of SETI's data processing/flow? For a very high level overview, see the diagrams in this write-up --> http://setiathome.berkeley.edu/sah_papers/cacm.php or this. Also, some good video explanations at the 10 year meeting and other stuff here --> http://setiathome.berkeley.edu/sah_about.php If you want deeper details, esp on implimentation, reading the source code and looking at the mysql layout will get you the quickest answer. |
Dirk Sadowski Send message Joined: 6 Apr 07 Posts: 7105 Credit: 147,663,825 RAC: 5 |
I know, that the task lists come from the replica DB.. ;-) But I didn't understood that Matt meant, that he will enable again the task lists.. BTW. I was interested about other GPUs, for to decide which will be the next for me.. But no joy, task lists are still disabled. I know it's a hard work to manage a lot of servers. Matt, good luck! |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.