June Update (Jun 02 2010)


log in

Advanced search

Message boards : Technical News : June Update (Jun 02 2010)

1 · 2 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1000083 - Posted: 2 Jun 2010, 19:31:08 UTC

Another monthly-ish report.

First the good news, before it seems like I only want to blather about the funny/annoying stuff. Jeff has been hammering on the NTPCkr to incorporate the newer RFI removal code. Before the plumbing was of the form: signals come in, pixels are scored, the best ones are displayed for the public to see. Now the plumbing is: signals come in, pixels are scored, the best ones are displayed in a sort of "preview" form and sent into the RFI loop, which then forces the pixels to be rescored (after bad signals are removed), and if they still happen to be in the top ten they'll have all the associated plots for the public to analyze.

I'm also still pecking away at data integrity tests. I have the "birdie injector" (which sticks fake signals in the raw data) working to some extent. After some full tests we're finding these birdies in the results reported back from the clients - though it seems that we might have to add another retroactive signal correction in the future. Don't worry - if this in fact true it's not a big deal. It's easy to fix and there's no lost scientific integrity. Other than that, there's continuing testing happening in my copious free time. I also wrote up a scientific newsletter about radar blanking.

Of course, our server woes have peaked a bit recently, coinciding nicely with the holiday weekend and a mass e-mail. The mass mail was part of the problem actually - there was a link to several video files which were much larger than I assumed. Like hundreds of megabytes larger. So that made our web site (and the whole lab's internet connection) a little bit sluggish. Oopsie.

But the two machines prone to random crashes did just that. First mork went down taking the BOINC user database with it. Recovery was easy enough, but then the next day ptolemy went down taking everything with it. Dan actually came up on Sunday to power cycle the thing by hand (with my guidance via phone).

Yeah - it's on our list of over 200 critical things to get a remote power strip installed on ptolemy. I'd rather we just have systems that didn't crash. Unfortunately the functionalities of these machines are such that transferring them to other machines is impossible. However, we do have thumper... and marvin...

I've been hoping to reorganize thumper and upgrade its OS for some time now. We finally had the window to do that this past month, but there was one hangup after another postponing this project. Meanwhile we have marvin set up for test database purposes. It's more or less a functional equivalent of thumper, but with a lot less drive spindles. Still, the plan now is to burn marvin in, move the science database there (temporarily if not permanently), and then thumper is free to be completely wiped clean. Maybe we'll make thumper the new ptolemy and retire old ptolemy. That'll be one less crashy server to deal with. As for replacing mork... well we need another system with a bunch of CPUs, many disk spindles, and at least 64GB of memory. Not happening any time soon.

Let's see... other projects... oh yeah - we're now merging the spike tables. We had to split the spike signal a while back due to reaching some logical constraint in the database. After the dust settled on various other projects we're ready to make that one whole spike table again. Easier said than done, but what isn't around her? Anywho.. this was why the spike signal counts on the science status page seemed a little off for a while.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Chris S
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 31117
Credit: 11,278,940
RAC: 19,850
United Kingdom
Message 1000098 - Posted: 2 Jun 2010, 20:23:47 UTC

Thanks for the update Matt it is appreciated. May I ask, how large is the Seti Informix database compared to other scientific or similar applications out there?

____________
Damsel Rescuer, Kitty Patron, Uli Devotee, Julie Supporter
ES99 Admirer, Raccoon Friend, Anniet fan, 1% badge, HSA


Claggy
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4048
Credit: 32,693,129
RAC: 604
United Kingdom
Message 1000107 - Posted: 2 Jun 2010, 20:41:22 UTC - in response to Message 1000083.

Thanks for the update Matt, and for all the hard work keeping the Servers running,

Claggy

Luke
Volunteer developer
Avatar
Send message
Joined: 31 Dec 06
Posts: 2546
Credit: 817,560
RAC: 0
New Zealand
Message 1000126 - Posted: 2 Jun 2010, 21:09:48 UTC

Thank you Matt for the update, any is appreciated. All the servers are apparently running normally, we can upload, download and report tasks. All is fine and happy.

On a completely unrelated note... It was mentioned on the old list of things SETI@home would desire to implement is a sky search in the southern hemisphere, possibly using te parks observatory in Australia? Was this idea shelved? Or is it still in the back of your minds if Arecibo runs out of funding?
____________
- Luke.

Profile Byron Leigh Hatch @ team Carl Sagan
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 3501
Credit: 11,813,565
RAC: 457
Canada
Message 1000143 - Posted: 2 Jun 2010, 22:19:44 UTC

Thanks for the update Matt. You scientist At SETI@home are the best.
____________

Profile Bill Walker
Avatar
Send message
Joined: 4 Sep 99
Posts: 3330
Credit: 1,966,412
RAC: 2,199
Canada
Message 1000146 - Posted: 2 Jun 2010, 22:38:03 UTC

Thank you Matt, and please tell Dan thanks too.
____________

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12126
Credit: 6,409,188
RAC: 8,333
United States
Message 1000162 - Posted: 3 Jun 2010, 0:35:38 UTC - in response to Message 1000083.

Thanks for the update. It is much appreciated.

Went by the NTPCKR page a couple days ago but there are no links to the waterfall plots available, in case Jeff didn't know that.

Keep up the good work.

____________

Profile gizbar
Avatar
Send message
Joined: 7 Jan 01
Posts: 586
Credit: 21,087,774
RAC: 0
United Kingdom
Message 1000273 - Posted: 3 Jun 2010, 9:07:12 UTC

Thanks Matt, and Dan for dragging himself in on a Sunday!

Nice to get an update, and hope your productivity has increased since you're not posting here all the time...

But, who's Marvin? He's not on the server list!

regards, Gizbar.

____________


A proud GPU User Server Donor!

Fred W
Volunteer tester
Send message
Joined: 13 Jun 99
Posts: 2524
Credit: 11,954,210
RAC: 0
United Kingdom
Message 1000289 - Posted: 3 Jun 2010, 10:18:38 UTC - in response to Message 1000273.

But, who's Marvin? He's not on the server list!



He's hiding. That's what paranoia does... ;-P

F.
____________

ront
Send message
Joined: 25 Aug 01
Posts: 77
Credit: 386,336
RAC: 0
United States
Message 1000312 - Posted: 3 Jun 2010, 13:19:41 UTC

May I add my thanks and gratitude for the update and the outstanding dedication of all of you to keep this project online.

Blessing and Peace

ront
____________

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1000362 - Posted: 3 Jun 2010, 16:48:36 UTC

Answering some questions:

1. Not sure about other projects, but the SETI@home science database is roughly 1.25 terabytes and the Astropulse database is just under 0.5 terabytes. Both are of course still growing. There are billions of signals in there, which is why matching them together in various permutations is such a slow, arduous problem.

2. Southern hemisphere search is on indefinite hold. Not enough resources. That, or other telescopes, could be used if Arecibo goes away, but we'd prefer to keep it up at Arecibo as long as possible.

3. No waterfall plots on the NTPCkr page because these "top candidates" haven't gone through RFI excision yet, and when they do they'll be rescored. If they manage to still be in the top ten (highly unlikely) *then* waterfalls will be made (it's silly to keep making waterfalls full of RFI a computer can get rid of eventually on its own).

4. Marvin is a functional equivalent of thumper but with only 8 drives (instead of 48) - one of many Intel donations. We used it as a compute server, then set it up as a database test server, now maybe an actual database server. I only post the servers on the status page that are public facing (or process public data). We actually have about, let's see... over 20 servers around here (including desktops and backend compute servers).

5. To be clear, I personally am not a scientist, though I learn a lot via osmosis hanging around them that sometimes people think I know what I'm talking about. I more of a math/computer geek.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Byron Leigh Hatch @ team Carl Sagan
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 3501
Credit: 11,813,565
RAC: 457
Canada
Message 1000384 - Posted: 3 Jun 2010, 18:30:05 UTC

Thank you Matt. You scientist At SETI@home show outstanding dedication to keep this project on line. Best wishes and peace Byron.

Profile Geek@Play
Volunteer tester
Avatar
Send message
Joined: 31 Jul 01
Posts: 2463
Credit: 85,116,244
RAC: 15,502
United States
Message 1000390 - Posted: 3 Jun 2010, 19:04:54 UTC

I believe you are being too modest Matt. You have the "Project Scientist" title so you are one.

Thanks for everything you do to keep us updated.

____________
Boinc....Boinc....Boinc....Boinc....

Profile SciManStev
Volunteer tester
Avatar
Send message
Joined: 20 Jun 99
Posts: 4792
Credit: 79,688,418
RAC: 34,257
United States
Message 1000466 - Posted: 4 Jun 2010, 0:02:27 UTC

Thank you very much for the news. Knowing a little of what goes on gives us high motivation to continue being a part of this fantastic search. I see every one of you as a scientist and a problem solver. We will crunch on!

Steve
____________
Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

Profile Norm Hill
Send message
Joined: 6 Sep 99
Posts: 9
Credit: 10,395,688
RAC: 141
United States
Message 1000467 - Posted: 4 Jun 2010, 0:03:19 UTC - in response to Message 1000362.

Matt,
Half the reason I continue to contribute my spare CPU cycles to SETI is that I am truly interested in what is going on behind the scenes. I enjoyed hearing the daily goings on around the lab. You were a little frustrated in your posts sometimes but that goes with being a Type-A personality. Please reconsider going back to a daily or weekly posting. There a lot SETI fans who are actually interested in what is going on!

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12126
Credit: 6,409,188
RAC: 8,333
United States
Message 1000481 - Posted: 4 Jun 2010, 0:48:03 UTC - in response to Message 1000362.

5. To be clear, I personally am not a scientist, though I learn a lot via osmosis hanging around them that sometimes people think I know what I'm talking about. I more of a math/computer geek.

- Matt

What makes a scientist? A fancy degree? Can name a long list that didn't have one. Make the majority of money at it? Can name another long list that didn't. Publishing papers? Haven't you done that?

Matt, you may not be an astronomer, but you are a scientist, perhaps computer scientist with a specialty of distributed processing. Anyone who can keep Seti going is a cutting edge wonder.


____________

Profile Byron Leigh Hatch @ team Carl Sagan
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 3501
Credit: 11,813,565
RAC: 457
Canada
Message 1000489 - Posted: 4 Jun 2010, 1:33:17 UTC - in response to Message 1000362.

Answering some questions:

.....

5. To be clear, I personally am not a scientist, though I learn a lot via osmosis hanging around them that sometimes people think I know what I'm talking about. I more of a math/computer geek.

- Matt

I know that Matt, but still ... I consider you to be a scientist. Thank you Matt, David A, Dan, Jeff, Josh Von Korff, Robert Bankay, Kevin Douglas, and anyone I forgot ... You scientist At SETI@home show outstanding dedication to keep this project on line. Best wishes and peace Byron.

Profile Jon Golding
Avatar
Send message
Joined: 20 Apr 00
Posts: 56
Credit: 365,064
RAC: 7
United Kingdom
Message 1000593 - Posted: 4 Jun 2010, 11:43:43 UTC

Jeff has been hammering on the NTPCkr to incorporate the newer RFI removal code. Before the plumbing was of the form: signals come in, pixels are scored, the best ones are displayed for the public to see. Now the plumbing is: signals come in, pixels are scored, the best ones are displayed in a sort of "preview" form and sent into the RFI loop, which then forces the pixels to be rescored (after bad signals are removed), and if they still happen to be in the top ten they'll have all the associated plots for the public to analyze.


So, is the RFI removal script now running constantly in the background, or does this script only get implemented every day or so?
____________

Luke
Volunteer developer
Avatar
Send message
Joined: 31 Dec 06
Posts: 2546
Credit: 817,560
RAC: 0
New Zealand
Message 1000601 - Posted: 4 Jun 2010, 12:13:50 UTC

Thanks for answering my question Matt. :)
____________
- Luke.

Richard
Avatar
Send message
Joined: 10 Jul 99
Posts: 19
Credit: 8,474,928
RAC: 9,198
Argentina
Message 1000664 - Posted: 4 Jun 2010, 16:23:20 UTC - in response to Message 1000467.

Matt,
Half the reason I continue to contribute my spare CPU cycles to SETI is that I am truly interested in what is going on behind the scenes. I enjoyed hearing the daily goings on around the lab. You were a little frustrated in your posts sometimes but that goes with being a Type-A personality. Please reconsider going back to a daily or weekly posting. There a lot SETI fans who are actually interested in what is going on!


I agree 100% with this Member. I think there is a lot of people who enjoyed the almost daily posts of Matt, not matter if there are good or bad news, and nowadays whitout this posts, feels a little "abandoned" (to say) by the project. I agree that the Science is the most important part of the project, but I believe that participate is not only crunching numbers but also knowing what is happening behind the curtains with our efforts.
I think that there outside is a lot of people like me who enjoyed to read each morning the "coloured" comments of Matt (he would be a writer) and feels a little frustrated by the lack of them.
____________

1 · 2 · Next

Message boards : Technical News : June Update (Jun 02 2010)

Copyright © 2014 University of California