Message boards :
Technical News :
Walkthrough (Apr 22 2008)
Message board moderation
Author | Message |
---|---|
![]() ![]() Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 ![]() |
Back from a long weekend out of town. Didn't seem to miss very much. I checked the network graphs while I was away and saw no dips, so that's a pretty good sign things were generally healthy in my absence. There was another seemingly bogus disk failure on thumper. Is smartd being too sensitive? The drive tagged as potentially faulty was failed/re-added without much ado. Today had the usual outage. Nothing out of the ordinary there. One funny thing - for an unspecified amount of time nobody on the Berkeley campus (outside of the space lab) was able to connect to our servers to receive/send SETI@home data. This was due to asymmetrical routing - a problem on our public facing servers that send data over our ISP (as opposed to via the campus LAN). Jeff found and fixed the problem and I updated the network scripts to make sure a reboot doesn't break it again. Jeff just spent an hour or so walking me through the current nitpicker (i.e. the candidate-finder) code. This really is one of those simple concepts that requires a complex solution. I find it frustrating to describe why, as the reasons are hardly obvious, and the problems are nested. We used to do this stuff with our own human brains which can find patterns and detect duplicates and RFI quickly as long as the data fits on a couple pages. This isn't so much the case anymore, and getting the computers to smartly (and efficiently) do the same grouping, comparing, and discarding is difficult. Think of it this way: you have a bunch of friends and you realize two of them are single and, based on many different variables, perhaps quite compatible - so you set them up on a date. Easy, no? Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy. In any case, I might start throwing random output from it on the science status page which is of anecdotal interest. Like extra info about where we're currently pointing and what we've seen there before. Check for that in the next day or so. - Matt -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
Nicolas Send message Joined: 30 Mar 05 Posts: 161 Credit: 12,985 RAC: 0 ![]() |
Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy. That doesn't stop some companies from trying, and selling you the service ;) Contribute to the Wiki! |
![]() ![]() Send message Joined: 29 Feb 00 Posts: 16019 Credit: 794,685 RAC: 0 ![]() |
. . . Thanks for the Update Matt - Each of You are doing an Exemplary Job @ Berkeley < btw - nice comparisons ;) ![]() Science Status Page . . . |
Dena Wiltsie Send message Joined: 19 Apr 01 Posts: 1628 Credit: 24,230,968 RAC: 26 ![]() ![]() |
Jeff just spent an hour or so walking me through the current nitpicker (i.e. the candidate-finder) code. This really is one of those simple concepts that requires a complex solution. I find it frustrating to describe why, as the reasons are hardly obvious, and the problems are nested. We used to do this stuff with our own human brains which can find patterns and detect duplicates and RFI quickly as long as the data fits on a couple pages. This isn't so much the case anymore, and getting the computers to smartly (and efficiently) do the same grouping, comparing, and discarding is difficult. Think of it this way: you have a bunch of friends and you realize two of them are single and, based on many different variables, perhaps quite compatible - so you set them up on a date. Easy, no? Now try to run a completely automated dating service trying to accurately pair up every single person on the planet with the best possible mate. Not as easy. Without looking at the data, something like the Quine-McCluskey method for reducing electronic circuits sounds like a solution to the problem. Because you are looking at something more complex that binary, the idea would have to be refined. If you have not heard of this before, a book describing it is Digital Logic and Computer Design by M. Morris Mano or Digital Networks and Computer Systems by Taylor L. Booth. |
![]() ![]() ![]() Send message Joined: 23 Oct 02 Posts: 5862 Credit: 10,957,677 RAC: 18 ![]() ![]() |
|
![]() ![]() Send message Joined: 2 Feb 04 Posts: 14925 Credit: 1,378,607 RAC: 6 ![]() |
Thanks for the update...When the nitpicker is operational...then the real fun begins. ![]() PROUD TO BE TFFE! |
Speedy ![]() Send message Joined: 26 Jun 04 Posts: 1646 Credit: 12,921,799 RAC: 89 ![]() ![]() |
I might start throwing random output from it on the science status page which is of anecdotal interest. Like extra info about where we're currently pointing and what we've seen there before. Check for that in the next day or so. Matt & team thank's for the update's you guy's are doing gr8 work. While you are adding data to the science status page is it possable to add a field on the server status page to show the latest wu tape/disk that has been finished. Evan if it was only shown on page for 6 or 12 hours, is this possible? One more question. How long is it before say tape/disk '12mr08ab' gets sent back to get filled up with new raw data? Thank's Speedy |
DJStarfox Send message Joined: 23 May 01 Posts: 1066 Credit: 1,226,053 RAC: 2 ![]() |
Back from a long weekend out of town. Didn't seem to miss very much. I checked the network graphs while I was away and saw no dips, so that's a pretty good sign things were generally healthy in my absence. There was another seemingly bogus disk failure on thumper. Is smartd being too sensitive? The drive tagged as potentially faulty was failed/re-added without much ado. Today had the usual outage. Nothing out of the ordinary there. Matt: Isn't there a professor or some fellow graduate students in computer science that can provide some pattern matching heuristic or logic to this problem? If you could find the money, maybe hire a graduate assistant to work with you on this? I would consider NPKR as much a research project as an applied computer science problem. |
![]() ![]() Send message Joined: 20 Dec 05 Posts: 3187 Credit: 57,163,290 RAC: 0 ![]() |
Um, Matt... the stats on the server status page haven't updated since the system came back online from the Tuesday backup, and... Three splitters show as 'Not running'! . ![]() Hello, from Albany, CA!... |
![]() ![]() Send message Joined: 29 Feb 00 Posts: 16019 Credit: 794,685 RAC: 0 ![]() |
Back from a long weekend out of town. Didn't seem to miss very much. I checked the network graphs while I was away and saw no dips, so that's a pretty good sign things were generally healthy in my absence. There was another seemingly bogus disk failure on thumper. Is smartd being too sensitive? The drive tagged as potentially faulty was failed/re-added without much ado. Today had the usual outage. Nothing out of the ordinary there. . . . i forwarded a Letter to Dr. Korpela @ Berkeley, that i Originally sent to Ranjeet (see Resume + Website) Ranjeet has Officially Stated that he would 'Volunteer' his time - he just needs to know 'What' Berkeley wants him to do . . . RANJEET SRIDHAR - Latest Resume January 2008 > @ Matt - You could ask Eric IF he's contacted Ranjeet concerning . . . ![]() Science Status Page . . . |
![]() ![]() Send message Joined: 21 Apr 04 Posts: 3252 Credit: 31,903,643 RAC: 0 ![]() |
Back from a long weekend out of town. Didn't seem to miss very much. I checked the network graphs while I was away and saw no dips, so that's a pretty good sign things were generally healthy in my absence. There was another seemingly bogus disk failure on thumper. Is smartd being too sensitive? The drive tagged as potentially faulty was failed/re-added without much ado. Today had the usual outage. Nothing out of the ordinary there. Thanks for your weekly update, Matt . Saw someone mention the 'Quine - Mc Cluskey Method', here a link: Quine-McCluskey Since the 'NiTPiCKer' is a SETI@home SCIENCE APP., is there any info on this subject, or just follow your report's on that matter, as you stated in your last post. (Finally got my lappy back, was quite used to read forum articals, from my roof) FJV. ![]() |
![]() ![]() Send message Joined: 29 Feb 00 Posts: 16019 Credit: 794,685 RAC: 0 ![]() |
|
![]() ![]() Send message Joined: 24 Nov 06 Posts: 7489 Credit: 91,093,184 RAC: 0 ![]() |
I'd have imagined that performance-wise a static table, as the number of sky locations is predetermined & invariant, hashed into by sky location , with a fixed number of slots for Signal/WU-id's of highest power (Chip stacking method) would do the trick, ... I guess that'd be one big table though! good luck with whatever method is determined! (Can't wait to read that paper :D). Jason "Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions. |
![]() ![]() Send message Joined: 21 Apr 04 Posts: 3252 Credit: 31,903,643 RAC: 0 ![]() |
Thank you Dr.C.E.T.I., for your link. And i'am glad i've my LapTop, back from the reseller. Had insurance till may ;) ![]() |
Josef W. Segur Send message Joined: 30 Oct 99 Posts: 4504 Credit: 1,414,761 RAC: 0 ![]() |
... The disk contained all the 12mr08 data. It should be available to be sent back almost immediately once the smaller sets are created in local storage. If there are at least as many of the custom foam shipping protectors as there are disks, I'd guess a disk wouldn't stay at Berkeley for more than one work day after that. Joe |
DJStarfox Send message Joined: 23 May 01 Posts: 1066 Credit: 1,226,053 RAC: 2 ![]() |
Re: the technical news post: Good news, I see "Workunits waiting for db purging" is at a record low for the month.... Hopefully, the post-processing queue can stay ahead of the curve for a while. I don't think smartd is too sensitive; I thought the point of smartd was to warn you before a disk fails completely. If you run a smartctl -i -A /dev/sda (or whatever your devices are), do any of the parameters look bad? Things to look for are temperature (usually 194 on SATA) and anything with type "pre-fail" with positive raw values. |
Speedy ![]() Send message Joined: 26 Jun 04 Posts: 1646 Credit: 12,921,799 RAC: 89 ![]() ![]() |
Joe thank you I understand. How come there are two tapes from the 13th of March one called 13mr08ah & the other one is called 13mr08ai? is this because there was too much data to store on one tape for the 13th of March? Thanks Speedy |
![]() ![]() Send message Joined: 4 Jul 99 Posts: 1575 Credit: 4,152,111 RAC: 1 ![]() |
Correct, there is often more data collected in a day than will fit on a "tape". With those names there should be at least eight "tapes" used on that day, the first one for the day is normally "aa". BOINC WIKI ![]() ![]() BOINCing since 2002/12/8 |
![]() ![]() Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 ![]() |
Clarity on "tapes": Historically we actually used 35GB tapes to collect raw data, and we'd record about 1.5 tapes a day. So they were named with a date followed by a two letter suffix denoting order ("12ja01aa" followed by "12ja01ab" etc.). We have about 2000 of these old tapes sitting on shelves in our other lab. Hard drives are cheaper now. We have a set of 500GB or 750GB drives we ship to/from Arecibo to record the data. The data is collected in 50GB files (occasionally referred to as "tapes" out of habit and that's what they are still called in our database and some of our old software). Why 50GB files instead of just filling up the whole drive with a single file? Because when we archive the files to HPSS (our off site archival storage) files over that size tend to break during transfer. So.. we still have the same naming convention (6 character date followed by 2 character suffix denoting order) but with the higher multibeam data rates we record 10+ files in a day at times. A single drive can span multiple days - when it's full the operators put it in a box. When the box is full, it is shipped back to us and we send them empty drives. - Matt -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
![]() ![]() Send message Joined: 29 Feb 00 Posts: 16019 Credit: 794,685 RAC: 0 ![]() |
Thank You Matt for the Information . . . it is Sincerely Appreciated Sir ![]() Science Status Page . . . |
©2025 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.