Data Laundering (Oct 12 2009)

Message boards : Technical News : Data Laundering (Oct 12 2009)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 939469 - Posted: 12 Oct 2009, 23:02:21 UTC

The latest software blanking tests were also a success, so we'll start putting older pre-hardware-blanked data into production, now that we can remove the radar. Yay! May take a few days to rev up this engine. Meanwhile Eric has been making progress on the "zone RFI" rejection software/algorithms, so we can start getting rid of the garbage that makes up our current "top candidates."

The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors!

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 939469 · Report as offensive
Profile Pappa
Volunteer tester
Avatar

Send message
Joined: 9 Jan 00
Posts: 2562
Credit: 12,301,681
RAC: 0
United States
Message 939483 - Posted: 13 Oct 2009, 0:04:08 UTC - in response to Message 939469.  

The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors!

- Matt


In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup.

Please consider a Donation to the Seti Project.

ID: 939483 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30967
Credit: 53,134,872
RAC: 32
United States
Message 939489 - Posted: 13 Oct 2009, 0:19:36 UTC - in response to Message 939483.  

The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors!

- Matt


In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup.

Good idea, except I'm sure the DB is a bit big for backups on CD. Perhaps a couple of removable TB drives though.

Thanks for the update Matt


ID: 939489 · Report as offensive
Berserker
Volunteer tester

Send message
Joined: 2 Jun 99
Posts: 105
Credit: 5,440,087
RAC: 0
United Kingdom
Message 939496 - Posted: 13 Oct 2009, 0:42:54 UTC - in response to Message 939483.  

In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup.


That (usually) works for MyISAM tables, but it's a bit more risky with InnoDB tables (which I think BOINC prefers for throughput reasons). Also, if you install a DB software upgrade which makes some important internal structural change, your backup - and even the live tables - might be rendered useless (though this is only likely with a major version upgrade).
Stats site - http://www.teamocuk.co.uk - still alive and (just about) kicking.
ID: 939496 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 939505 - Posted: 13 Oct 2009, 1:24:20 UTC
Last modified: 13 Oct 2009, 1:30:31 UTC

The latest software blanking tests were also a success, so we'll start putting older pre-hardware-blanked data into production

This is great news. Does this mean that there will be more ap work so that people that like ap work can crunch that instead of mb work? Or have I missed the point as to why the demand for ap work is a lot higher? Thanks feel the update. New data that we will be processing soon will it be around the same processing speed or will there be a slight speed increase due to the angle it was recordered on?
ID: 939505 · Report as offensive
CryptokiD
Avatar

Send message
Joined: 2 Dec 00
Posts: 150
Credit: 3,216,632
RAC: 0
United States
Message 939510 - Posted: 13 Oct 2009, 1:33:56 UTC

matt, as always, thankyou for the update. i always enjoy reading them.

can i axk you a question. i have heard talk of seti possibly running out of work unless the blanking deal gets worked out. if you by chance do not get it worked out, is there any rough timeframe for when the work will run out. and if you do get the radar blanking deal working and fully operational, where would that put us in a timeframe for running out of data?

reason i axk is most my machines only run a few hours per day. i have my work queue set to 2 days on the temp machines. if were going to run out od workunits anytime soon i would propbably bump this up to 10.

my cuda machine is the only one that really runs 24/7.
ID: 939510 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 939597 - Posted: 13 Oct 2009, 10:13:09 UTC


After the todays maintenance, we will see again the task lists?

And maybe also the pending credit list?

ID: 939597 · Report as offensive
Profile Gundolf Jahn

Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 446,358
RAC: 0
Germany
Message 939602 - Posted: 13 Oct 2009, 11:04:13 UTC - in response to Message 939597.  

After the todays maintenance, we will see again the task lists?

And maybe also the pending credit list?

That depends, as Matt said:
The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope).

Gruß,
Gundolf
ID: 939602 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 939638 - Posted: 13 Oct 2009, 13:59:14 UTC - in response to Message 939602.  

After the todays maintenance, we will see again the task lists?

And maybe also the pending credit list?

That depends, as Matt said:
The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope).

Gruß,
Gundolf


I have no big knowledge about the things behind the curtain.

I didn't know that this should mean this.

ID: 939638 · Report as offensive
Biffa
Volunteer tester
Avatar

Send message
Joined: 27 Oct 99
Posts: 41
Credit: 22,750,323
RAC: 0
United Kingdom
Message 939692 - Posted: 14 Oct 2009, 1:56:56 UTC

Yep I'd say the replica database is where all that stuff runs to keep the load down on the main db.
ID: 939692 · Report as offensive
Profile Johnney Guinness
Volunteer tester
Avatar

Send message
Joined: 11 Sep 06
Posts: 3093
Credit: 2,652,287
RAC: 0
Ireland
Message 939707 - Posted: 14 Oct 2009, 2:25:51 UTC - in response to Message 939469.  

Matt,
Thats really great news. This is science progress, SETI science progress. It gives me great hope in my heart to believe that yourself and Eric are getting to a stage where real data analysis can be carried out.

Gives me great hope!
Thank you for your tireless efforts Matt,
John.
ID: 939707 · Report as offensive
DJStarfox

Send message
Joined: 23 May 01
Posts: 1066
Credit: 1,226,053
RAC: 2
United States
Message 939725 - Posted: 14 Oct 2009, 3:18:30 UTC
Last modified: 14 Oct 2009, 3:18:41 UTC

Yeah, now SETI is back up after a long outage, but no replica DB. :(

They must have spent all day working on it....
ID: 939725 · Report as offensive
Profile edjcox
Avatar

Send message
Joined: 20 May 99
Posts: 96
Credit: 5,878,353
RAC: 0
United States
Message 939728 - Posted: 14 Oct 2009, 3:36:46 UTC

Matt:

Thanks for a decade of work on SETI.

Realizing your overly busy can you point me to location where I could get a Diagram of SETI's data processing/flow?

I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system.

I'm sure a response may be awhile so don't worry if it takes some time to respond.

Thank you,

E J Cox
Never engage stupid people at their level, they then have the home court advantage.....
ID: 939728 · Report as offensive
C. Barbie
Volunteer tester

Send message
Joined: 1 Jun 99
Posts: 7
Credit: 9,256,876
RAC: 0
United States
Message 939764 - Posted: 14 Oct 2009, 6:25:07 UTC - in response to Message 939728.  

Matt:

Thanks for a decade of work on SETI.

Realizing your overly busy can you point me to location where I could get a Diagram of SETI's data processing/flow?

I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system.

I'm sure a response may be awhile so don't worry if it takes some time to respond.

Thank you,

E J Cox



Try this... http://setiathome.berkeley.edu/forum_thread.php?id=54748
ID: 939764 · Report as offensive
Profile 52 Aces
Avatar

Send message
Joined: 7 Jan 02
Posts: 497
Credit: 14,261,068
RAC: 67
United States
Message 939794 - Posted: 14 Oct 2009, 11:38:55 UTC - in response to Message 939764.  
Last modified: 14 Oct 2009, 11:50:09 UTC

where I could get a Diagram of SETI's data processing/flow?

I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system.


For a very high level overview, see the diagrams in this write-up --> http://setiathome.berkeley.edu/sah_papers/cacm.php or this.

Also, some good video explanations at the 10 year meeting and other stuff here --> http://setiathome.berkeley.edu/sah_about.php

If you want deeper details, esp on implimentation, reading the source code and looking at the mysql layout will get you the quickest answer.

ID: 939794 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 939798 - Posted: 14 Oct 2009, 11:51:01 UTC


I know, that the task lists come from the replica DB.. ;-)
But I didn't understood that Matt meant, that he will enable again the task lists..

BTW. I was interested about other GPUs, for to decide which will be the next for me..
But no joy, task lists are still disabled.

I know it's a hard work to manage a lot of servers.
Matt, good luck!

ID: 939798 · Report as offensive

Message boards : Technical News : Data Laundering (Oct 12 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.