Back from a long weekend out of town. Didn't seem to miss very much. I checked the network graphs while I was away and saw no dips, so that's a pretty good sign things were generally healthy in my absence. There was another seemingly bogus disk failure on thumper. Is smartd being too sensitive? The drive tagged as potentially faulty was failed/re-added without much ado. Today had the usual outage. Nothing out of the ordinary there.

One funny thing - for an unspecified amount of time nobody on the Berkeley campus (outside of the space lab) was able to connect to our servers to receive/send SETI@home data. This was due to asymmetrical routing - a problem on our public facing servers that send data over our ISP (as opposed to via the campus LAN). Jeff found and fixed the problem and I updated the network scripts to make sure a reboot doesn't break it again.

Jeff just spent an hour or so walking me through the current nitpicker (i.e. the candidate-finder) code. This really is one of those simple concepts that requires a complex solution. I find it frustrating to describe why, as the reasons are hardly obvious, and the problems are nested. We used to do this stuff with our own human brains which can find patterns and detect duplicates and RFI quickly as long as the data fits on a couple pages. This isn't so much the case anymore, and getting the computers to smartly (and efficiently) do the same grouping, comparing, and discarding is difficult. Think of it this way: you have a bunch of friends and you realize two of them are single and, based on many different variables, perhaps quite compatible - so you set them up on a date. Easy, no? Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy. In any case, I might start throwing random output from it on the science status page which is of anecdotal interest. Like extra info about where we're currently pointing and what we've seen there before. Check for that in the next day or so.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy.

That doesn't stop some companies from trying, and selling you the service ;)

---

Contribute to the Wiki!

. . . Thanks for the Update Matt - Each of You are doing an Exemplary Job @ Berkeley

< btw - nice comparisons ;)

---

BOINC Wiki . . .

Science Status Page . . .

Jeff just spent an hour or so walking me through the current nitpicker (i.e. the candidate-finder) code. This really is one of those simple concepts that requires a complex solution. I find it frustrating to describe why, as the reasons are hardly obvious, and the problems are nested. We used to do this stuff with our own human brains which can find patterns and detect duplicates and RFI quickly as long as the data fits on a couple pages. This isn't so much the case anymore, and getting the computers to smartly (and efficiently) do the same grouping, comparing, and discarding is difficult. Think of it this way: you have a bunch of friends and you realize two of them are single and, based on many different variables, perhaps quite compatible - so you set them up on a date. Easy, no? Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy.

Without looking at the data, something like the Quine-McCluskey method for reducing electronic circuits sounds like a solution to the problem. Because you are looking at something more complex that binary, the idea would have to be refined. If you have not heard of this before, a book describing it is Digital Logic and Computer Design by M. Morris Mano or Digital Networks and Computer Systems by Taylor L. Booth.

---

Love the analogy. Look forward to hearing more on how nitpicker is developing. Thanks for the update Matt and thanks to all the unsung heroes behind the scenes.

---

Thanks for the update...When the nitpicker is operational...then the real fun begins.

---

PROUD TO BE TFFE!

I might start throwing random output from it on the science status page which is of anecdotal interest. Like extra info about where we're currently pointing and what we've seen there before. Check for that in the next day or so.

Matt & team thank's for the update's you guy's are doing gr8 work. While you are adding data to the science status page is it possable to add a field on the server status page to show the latest wu tape/disk that has been finished. Evan if it was only shown on page for 6 or 12 hours, is this possible?

One more question. How long is it before say tape/disk '12mr08ab' gets sent back to get filled up with new raw data?

Thank's
Speedy

Back from a long weekend out of town. Didn't seem to miss very much. I checked the network graphs while I was away and saw no dips, so that's a pretty good sign things were generally healthy in my absence. There was another seemingly bogus disk failure on thumper. Is smartd being too sensitive? The drive tagged as potentially faulty was failed/re-added without much ado. Today had the usual outage. Nothing out of the ordinary there.

One funny thing - for an unspecified amount of time nobody on the Berkeley campus (outside of the space lab) was able to connect to our servers to receive/send SETI@home data. This was due to asymmetrical routing - a problem on our public facing servers that send data over our ISP (as opposed to via the campus LAN). Jeff found and fixed the problem and I updated the network scripts to make sure a reboot doesn't break it again.

Jeff just spent an hour or so walking me through the current nitpicker (i.e. the candidate-finder) code. This really is one of those simple concepts that requires a complex solution. I find it frustrating to describe why, as the reasons are hardly obvious, and the problems are nested. We used to do this stuff with our own human brains which can find patterns and detect duplicates and RFI quickly as long as the data fits on a couple pages. This isn't so much the case anymore, and getting the computers to smartly (and efficiently) do the same grouping, comparing, and discarding is difficult. Think of it this way: you have a bunch of friends and you realize two of them are single and, based on many different variables, perhaps quite compatible - so you set them up on a date. Easy, no? Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy. In any case, I might start throwing random output from it on the science status page which is of anecdotal interest. Like extra info about where we're currently pointing and what we've seen there before. Check for that in the next day or so.

- Matt

Matt:

Isn't there a professor or some fellow graduate students in computer science that can provide some pattern matching heuristic or logic to this problem? If you could find the money, maybe hire a graduate assistant to work with you on this? I would consider NPKR as much a research project as an applied computer science problem.

Um, Matt... the stats on the server status page haven't updated since the system came back online from the Tuesday backup, and...

Three splitters show as 'Not running'!

---

.

Hello, from Albany, CA!...

Back from a long weekend out of town. Didn't seem to miss very much. I checked the network graphs while I was away and saw no dips, so that's a pretty good sign things were generally healthy in my absence. There was another seemingly bogus disk failure on thumper. Is smartd being too sensitive? The drive tagged as potentially faulty was failed/re-added without much ado. Today had the usual outage. Nothing out of the ordinary there.

One funny thing - for an unspecified amount of time nobody on the Berkeley campus (outside of the space lab) was able to connect to our servers to receive/send SETI@home data. This was due to asymmetrical routing - a problem on our public facing servers that send data over our ISP (as opposed to via the campus LAN). Jeff found and fixed the problem and I updated the network scripts to make sure a reboot doesn't break it again.

Jeff just spent an hour or so walking me through the current nitpicker (i.e. the candidate-finder) code. This really is one of those simple concepts that requires a complex solution. I find it frustrating to describe why, as the reasons are hardly obvious, and the problems are nested. We used to do this stuff with our own human brains which can find patterns and detect duplicates and RFI quickly as long as the data fits on a couple pages. This isn't so much the case anymore, and getting the computers to smartly (and efficiently) do the same grouping, comparing, and discarding is difficult. Think of it this way: you have a bunch of friends and you realize two of them are single and, based on many different variables, perhaps quite compatible - so you set them up on a date. Easy, no? Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy. In any case, I might start throwing random output from it on the science status page which is of anecdotal interest. Like extra info about where we're currently pointing and what we've seen there before. Check for that in the next day or so.

- Matt

Matt:

Isn't there a professor or some fellow graduate students in computer science that can provide some pattern matching heuristic or logic to this problem? If you could find the money, maybe hire a graduate assistant to work with you on this? I would consider NPKR as much a research project as an applied computer science problem.

. . . i forwarded a Letter to Dr. Korpela @ Berkeley, that i Originally sent to Ranjeet (see Resume + Website)

Ranjeet has Officially Stated that he would 'Volunteer' his time - he just needs to know 'What' Berkeley wants him to do . . .

RANJEET SRIDHAR - Latest Resume January 2008

> @ Matt - You could ask Eric IF he's contacted Ranjeet concerning . . .

---

BOINC Wiki . . .

Science Status Page . . .

Back from a long weekend out of town. Didn't seem to miss very much. I checked the network graphs while I was away and saw no dips, so that's a pretty good sign things were generally healthy in my absence. There was another seemingly bogus disk failure on thumper. Is smartd being too sensitive? The drive tagged as potentially faulty was failed/re-added without much ado. Today had the usual outage. Nothing out of the ordinary there.

One funny thing - for an unspecified amount of time nobody on the Berkeley campus (outside of the space lab) was able to connect to our servers to receive/send SETI@home data. This was due to asymmetrical routing - a problem on our public facing servers that send data over our ISP (as opposed to via the campus LAN). Jeff found and fixed the problem and I updated the network scripts to make sure a reboot doesn't break it again.

Jeff just spent an hour or so walking me through the current nitpicker (i.e. the candidate-finder) code. This really is one of those simple concepts that requires a complex solution. I find it frustrating to describe why, as the reasons are hardly obvious, and the problems are nested. We used to do this stuff with our own human brains which can find patterns and detect duplicates and RFI quickly as long as the data fits on a couple pages. This isn't so much the case anymore, and getting the computers to smartly (and efficiently) do the same grouping, comparing, and discarding is difficult. Think of it this way: you have a bunch of friends and you realize two of them are single and, based on many different variables, perhaps quite compatible - so you set them up on a date. Easy, no? Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy. In any case, I might start throwing random output from it on the science status page which is of anecdotal interest. Like extra info about where we're currently pointing and what we've seen there before. Check for that in the next day or so.

- Matt

Matt:

Isn't there a professor or some fellow graduate students in computer science that can provide some pattern matching heuristic or logic to this problem? If you could find the money, maybe hire a graduate assistant to work with you on this? I would consider NPKR as much a research project as an applied computer science problem.

. . . i forwarded a Letter to Dr. Korpela @ Berkeley, that i Originally sent to Ranjeet (see Resume + Website)

Ranjeet has Officially Stated that he would 'Volunteer' his time - he just needs to know 'What' Berkeley wants him to do . . .

RANJEET SRIDHAR - Latest Resume January 2008

> @ Matt - You could ask Eric IF he's contacted Ranjeet concerning . . .

Thanks for your weekly update, Matt .

Saw someone mention the 'Quine - Mc Cluskey Method', here a link:
Quine-McCluskey
Since the 'NiTPiCKer' is a SETI@home SCIENCE APP., is there any info on this subject, or just follow your report's on that matter, as you stated in your last post.
(Finally got my lappy back, was quite used to read forum articals, from my roof)
FJV.

---

@ Fred: Insight re: Quine-McCluskey

---

BOINC Wiki . . .

Science Status Page . . .

Matt:

Isn't there a professor or some fellow graduate students in computer science that can provide some pattern matching heuristic or logic to this problem? If you could find the money, maybe hire a graduate assistant to work with you on this? I would consider NPKR as much a research project as an applied computer science problem.

I'd have imagined that performance-wise a static table, as the number of sky locations is predetermined & invariant, hashed into by sky location , with a fixed number of slots for Signal/WU-id's of highest power (Chip stacking method) would do the trick, ... I guess that'd be one big table though! good luck with whatever method is determined! (Can't wait to read that paper :D).

Jason

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

@ Fred: Insight re: Quine-McCluskey

Thank you Dr.C.E.T.I., for your link.
And i'am glad i've my LapTop, back from the reseller. Had insurance till may ;)

---

...
One more question. How long is it before say tape/disk '12mr08ab' gets sent back to get filled up with new raw data?

Thank's
Speedy

The disk contained all the 12mr08 data. It should be available to be sent back almost immediately once the smaller sets are created in local storage. If there are at least as many of the custom foam shipping protectors as there are disks, I'd guess a disk wouldn't stay at Berkeley for more than one work day after that.

                                                                     Joe

Re: the technical news post:

Good news, I see "Workunits waiting for db purging" is at a record low for the month.... Hopefully, the post-processing queue can stay ahead of the curve for a while.

I don't think smartd is too sensitive; I thought the point of smartd was to warn you before a disk fails completely. If you run a smartctl -i -A /dev/sda (or whatever your devices are), do any of the parameters look bad? Things to look for are temperature (usually 194 on SATA) and anything with type "pre-fail" with positive raw values.

The disk contained all the 12mr08 data. It should be available to be sent back almost immediately once the smaller sets are created in local storage. If there are at least as many of the custom foam shipping protectors as there are disks, I'd guess a disk wouldn't stay at Berkeley for more than one work day after that.

                                                                     Joe

Joe thank you I understand. How come there are two tapes from the 13th of March one called 13mr08ah & the other one is called 13mr08ai? is this because there was too much data to store on one tape for the 13th of March?
Thanks
Speedy

Joe thank you I understand. How come there are two tapes from the 13th of March one called 13mr08ah & the other one is called 13mr08ai? is this because there was too much data to store on one tape for the 13th of March?
Thanks
Speedy

Correct, there is often more data collected in a day than will fit on a "tape". With those names there should be at least eight "tapes" used on that day, the first one for the day is normally "aa".

---

BOINC WIKI

BOINCing since 2002/12/8

Clarity on "tapes":

Historically we actually used 35GB tapes to collect raw data, and we'd record about 1.5 tapes a day. So they were named with a date followed by a two letter suffix denoting order ("12ja01aa" followed by "12ja01ab" etc.). We have about 2000 of these old tapes sitting on shelves in our other lab.

Hard drives are cheaper now. We have a set of 500GB or 750GB drives we ship to/from Arecibo to record the data. The data is collected in 50GB files (occasionally referred to as "tapes" out of habit and that's what they are still called in our database and some of our old software). Why 50GB files instead of just filling up the whole drive with a single file? Because when we archive the files to HPSS (our off site archival storage) files over that size tend to break during transfer.

So.. we still have the same naming convention (6 character date followed by 2 character suffix denoting order) but with the higher multibeam data rates we record 10+ files in a day at times. A single drive can span multiple days - when it's full the operators put it in a box. When the box is full, it is shipped back to us and we send them empty drives.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thank You Matt for the Information . . . it is Sincerely Appreciated Sir

---

BOINC Wiki . . .

Science Status Page . . .