Data Laundering (Oct 12 2009)


log in

Advanced search

Message boards : Technical News : Data Laundering (Oct 12 2009)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1388
Credit: 74,079
RAC: 0
United States
Message 939469 - Posted: 12 Oct 2009, 23:02:21 UTC

The latest software blanking tests were also a success, so we'll start putting older pre-hardware-blanked data into production, now that we can remove the radar. Yay! May take a few days to rev up this engine. Meanwhile Eric has been making progress on the "zone RFI" rejection software/algorithms, so we can start getting rid of the garbage that makes up our current "top candidates."

The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors!

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Pappa
Volunteer tester
Avatar
Send message
Joined: 9 Jan 00
Posts: 2562
Credit: 12,301,681
RAC: 0
United States
Message 939483 - Posted: 13 Oct 2009, 0:04:08 UTC - in response to Message 939469.

The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors!

- Matt


In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup.

____________
Please consider a Donation to the Seti Project.

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12057
Credit: 6,374,127
RAC: 8,651
United States
Message 939489 - Posted: 13 Oct 2009, 0:19:36 UTC - in response to Message 939483.

The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope). We are still concerned that we suddenly don't have a reliable backup mechanism, if mysqldump occasionally gives us dumps containing hidden syntax errors!

- Matt


In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup.

Good idea, except I'm sure the DB is a bit big for backups on CD. Perhaps a couple of removable TB drives though.

Thanks for the update Matt


____________

Berserker
Volunteer tester
Send message
Joined: 2 Jun 99
Posts: 105
Credit: 5,386,463
RAC: 0
United Kingdom
Message 939496 - Posted: 13 Oct 2009, 0:42:54 UTC - in response to Message 939483.

In the past had several machines that required the same updated database (a fairly large database) in multiple locations. I found that it was easier update the Master, then shutdown MySQL and make a copy of the DB and then burn to a CD for transport. Then mount the DB on the machines at the other sites. In that case the CD became my backup.


That (usually) works for MyISAM tables, but it's a bit more risky with InnoDB tables (which I think BOINC prefers for throughput reasons). Also, if you install a DB software upgrade which makes some important internal structural change, your backup - and even the live tables - might be rendered useless (though this is only likely with a major version upgrade).
____________
Stats site - http://www.teamocuk.co.uk - still alive and (just about) kicking.

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 644
Credit: 5,327,076
RAC: 5,817
New Zealand
Message 939505 - Posted: 13 Oct 2009, 1:24:20 UTC
Last modified: 13 Oct 2009, 1:30:31 UTC

The latest software blanking tests were also a success, so we'll start putting older pre-hardware-blanked data into production

This is great news. Does this mean that there will be more ap work so that people that like ap work can crunch that instead of mb work? Or have I missed the point as to why the demand for ap work is a lot higher? Thanks feel the update. New data that we will be processing soon will it be around the same processing speed or will there be a slight speed increase due to the angle it was recordered on?
____________

Live in NZ y not join Smile City?

CryptokiD
Avatar
Send message
Joined: 2 Dec 00
Posts: 134
Credit: 2,814,936
RAC: 0
United States
Message 939510 - Posted: 13 Oct 2009, 1:33:56 UTC

matt, as always, thankyou for the update. i always enjoy reading them.

can i axk you a question. i have heard talk of seti possibly running out of work unless the blanking deal gets worked out. if you by chance do not get it worked out, is there any rough timeframe for when the work will run out. and if you do get the radar blanking deal working and fully operational, where would that put us in a timeframe for running out of data?

reason i axk is most my machines only run a few hours per day. i have my work queue set to 2 days on the temp machines. if were going to run out od workunits anytime soon i would propbably bump this up to 10.

my cuda machine is the only one that really runs 24/7.

Profile [seti.international] Dirk Sadowski
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7018
Credit: 59,143,060
RAC: 20,636
Germany
Message 939597 - Posted: 13 Oct 2009, 10:13:09 UTC


After the todays maintenance, we will see again the task lists?

And maybe also the pending credit list?

____________
BR



>Das Deutsche Cafe. The German Cafe.<

Profile Gundolf Jahn
Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 356,746
RAC: 15
Germany
Message 939602 - Posted: 13 Oct 2009, 11:04:13 UTC - in response to Message 939597.

After the todays maintenance, we will see again the task lists?

And maybe also the pending credit list?

That depends, as Matt said:
The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope).

Gruß,
Gundolf

Profile [seti.international] Dirk Sadowski
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7018
Credit: 59,143,060
RAC: 20,636
Germany
Message 939638 - Posted: 13 Oct 2009, 13:59:14 UTC - in response to Message 939602.

After the todays maintenance, we will see again the task lists?

And maybe also the pending credit list?

That depends, as Matt said:
The mysql replica was pretty much rendered useless by all our poking and prodding last week. We'll recreate it from scratch tomorrow (we hope).

Gruß,
Gundolf


I have no big knowledge about the things behind the curtain.

I didn't know that this should mean this.

____________
BR



>Das Deutsche Cafe. The German Cafe.<

Biffa
Volunteer tester
Avatar
Send message
Joined: 27 Oct 99
Posts: 41
Credit: 22,694,516
RAC: 101
United Kingdom
Message 939692 - Posted: 14 Oct 2009, 1:56:56 UTC

Yep I'd say the replica database is where all that stuff runs to keep the load down on the main db.
____________

Profile Johnney Guinness
Volunteer tester
Avatar
Send message
Joined: 11 Sep 06
Posts: 3076
Credit: 2,386,369
RAC: 2,369
Ireland
Message 939707 - Posted: 14 Oct 2009, 2:25:51 UTC - in response to Message 939469.

Matt,
Thats really great news. This is science progress, SETI science progress. It gives me great hope in my heart to believe that yourself and Eric are getting to a stage where real data analysis can be carried out.

Gives me great hope!
Thank you for your tireless efforts Matt,
John.
____________

DJStarfox
Send message
Joined: 23 May 01
Posts: 1040
Credit: 533,933
RAC: 154
United States
Message 939725 - Posted: 14 Oct 2009, 3:18:30 UTC
Last modified: 14 Oct 2009, 3:18:41 UTC

Yeah, now SETI is back up after a long outage, but no replica DB. :(

They must have spent all day working on it....

Profile edjcox
Avatar
Send message
Joined: 20 May 99
Posts: 63
Credit: 3,967,004
RAC: 1,081
United States
Message 939728 - Posted: 14 Oct 2009, 3:36:46 UTC

Matt:

Thanks for a decade of work on SETI.

Realizing your overly busy can you point me to location where I could get a Diagram of SETI's data processing/flow?

I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system.

I'm sure a response may be awhile so don't worry if it takes some time to respond.

Thank you,

E J Cox
____________
Never engage stupid people at their level, they then have the home court advantage.....

C. Barbie
Volunteer tester
Send message
Joined: 1 Jun 99
Posts: 7
Credit: 9,254,854
RAC: 0
United States
Message 939764 - Posted: 14 Oct 2009, 6:25:07 UTC - in response to Message 939728.

Matt:

Thanks for a decade of work on SETI.

Realizing your overly busy can you point me to location where I could get a Diagram of SETI's data processing/flow?

I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system.

I'm sure a response may be awhile so don't worry if it takes some time to respond.

Thank you,

E J Cox



Try this... http://setiathome.berkeley.edu/forum_thread.php?id=54748
____________

Profile 52 Aces
Avatar
Send message
Joined: 7 Jan 02
Posts: 497
Credit: 12,875,212
RAC: 6,874
United States
Message 939794 - Posted: 14 Oct 2009, 11:38:55 UTC - in response to Message 939764.
Last modified: 14 Oct 2009, 11:50:09 UTC

where I could get a Diagram of SETI's data processing/flow?

I'm interested in an overview of the projects data flow and analysis efforts as well as how this is implemented on your system.


For a very high level overview, see the diagrams in this write-up --> http://setiathome.berkeley.edu/sah_papers/cacm.php or this.

Also, some good video explanations at the 10 year meeting and other stuff here --> http://setiathome.berkeley.edu/sah_about.php

If you want deeper details, esp on implimentation, reading the source code and looking at the mysql layout will get you the quickest answer.

Profile [seti.international] Dirk Sadowski
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7018
Credit: 59,143,060
RAC: 20,636
Germany
Message 939798 - Posted: 14 Oct 2009, 11:51:01 UTC


I know, that the task lists come from the replica DB.. ;-)
But I didn't understood that Matt meant, that he will enable again the task lists..

BTW. I was interested about other GPUs, for to decide which will be the next for me..
But no joy, task lists are still disabled.

I know it's a hard work to manage a lot of servers.
Matt, good luck!

____________
BR



>Das Deutsche Cafe. The German Cafe.<

Message boards : Technical News : Data Laundering (Oct 12 2009)

Copyright © 2014 University of California