Message boards :
Technical News :
Nefarious Designs, Inc. (Nov 25 2008)
Message board moderation
Author | Message |
---|---|
Matt Lebofsky Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 |
Happy Tuesday. We had the usual outage rigamarole today and should be recovering from that in due time. Right after the backup was finished we restarted mysql with full query logging turned on. We knew this would choke the server a bit, and would just be on temporarily. After about a half hour we had over a million queries in the log, so we brought everything back down and turned logging off. We'll parse this log file, and perhaps others we generate over the next 24 hours, in order to find pesky unoptimized queries, anything that would die if we remove all multidimensional indexes, or queries running far more often than we expected. Also during the outage I moved some big directories around - more NAS shell games. Other than that I've been reconfiguring some more web server stuff (internal use pages) and trying to maximize the raw data pipeline plumbing to get as much work online as possible. It doesn't help that a lot of our raw data drives are showing weird signs of corruption. Don't worry - we do checksums at every important transfer to ensure the data are sound, and the splitters cannot operate on garbage (there are keyword strings occurring regularly throughout the files). Nevertheless, we're having to throw away some files, which is sad. My spider sense tells me this has to do with our general SATA enclosure mounting/unmounting woes. For example, we're finding drives that are 500GB thinking they are 750GB when mounted. Was this because a drive previously on that mount point was 750GB and some bookkeeping bits haven't been cleared? I dunno, but I'm sure this isn't good. In a couple hours I get to call a number where an automated voice will tell me if I have to attend jury duty tomorrow or not. I get dragged in for potential jury duty an astonishing amount (pretty much the legal maximum) considering I never actually get selected for trial, and never will. - Matt -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
Blurf Send message Joined: 2 Sep 06 Posts: 8962 Credit: 12,678,685 RAC: 0 |
Thanks for the update. Happy Thanksgiving, Matt! |
DJStarfox Send message Joined: 23 May 01 Posts: 1066 Credit: 1,226,053 RAC: 2 |
If the kernel doesn't think the entire SATA device driver has been reloaded, it may keep the partition table from the previous drive. This is bad. This behavior depends a lot on the driver and hardware you're using, but a reboot should always fix it (if that's the case). Have a happy Thanksgiving! |
ML1 Send message Joined: 25 Nov 01 Posts: 20372 Credit: 7,508,002 RAC: 20 |
If the kernel doesn't think the entire SATA device driver has been reloaded, it may keep the partition table from the previous drive. This is bad. This behavior depends a lot on the driver and hardware you're using, but a reboot should always fix it (if that's the case). If that really is the case, then a reboot isn't needed. Simpler/quicker is just to issue a "hdparm -z /dev/sdX" for whatever the drive is. (Make sure it has a "umount /dev/sdX" first!) Good luck, Martin See new freedom: Mageia Linux Take a look for yourself: Linux Format The Future is what We all make IT (GPLv3) |
Sharkbait Send message Joined: 2 Jun 99 Posts: 11 Credit: 6,333,145 RAC: 0 |
Good luck on the jury duty Matt. I was selected for mine starting Monday. Out of curiosity, I called up the jury line this week to see what their schedule was like. No jurors were called all week, so I expect to be called in a somewhat busier week. |
PhonAcq Send message Joined: 14 Apr 01 Posts: 1656 Credit: 30,658,217 RAC: 1 |
I get dragged in for potential jury duty an astonishing amount (pretty much the legal maximum) considering I never actually get selected for trial, and never will. This quote elicited a chuckle. I guess Matt is not the typical brain dead drone who tends to remain on the juries after the voir dire process. It is sad that juries are constructed from the most 'normal' of people, those who you really don't want to invite to your parties because the conversations would be about grass growing. I guess that is ok from the accused perspective because, if I were one, then I wouldn't want too creative of a thinker judging me, unless of course I were guilty. On the other hand, it is also sad that these folks are viewed as acceptable peers, which says something about the accused, I suppose. |
PhonAcq Send message Joined: 14 Apr 01 Posts: 1656 Credit: 30,658,217 RAC: 1 |
I believe the raw data is put into the high capacity storage ultimately. So why not put it there to begin with directly from Puerto Rico and download it to Berkeley as needed? Shuffling disks around is proving tedious and error prone, it sounds. And since this is a Berkeley project, maybe the case could be made that an upload/download scheme would be greener than physically shipping the disks back and forth. |
DJStarfox Send message Joined: 23 May 01 Posts: 1066 Credit: 1,226,053 RAC: 2 |
I believe the raw data is put into the high capacity storage ultimately. So why not put it there to begin with directly from Puerto Rico and download it to Berkeley as needed? Shuffling disks around is proving tedious and error prone, it sounds. Not quite. If I know islands like I do, the "main pipe" out of the island is quite small in bandwidth compared to mainland networks. All the internet connections must share this network pipe. It's probably much faster to ship drives, especially as you send more data. Now, if we changed SETI's telescope to be somewhere in New Mexico, then there are more options for remote data transfer. |
Matt Lebofsky Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 |
Yeah.. we can collect up to 100s of gigabytes a day. We'd need to be transporting data at (very) roughly 20Mbits a second 24/7 to keep up. We don't have that spare bandwidth, and Arecibo surely does not. About 10 years ago the ENTIRE connection between Arecibo and the rest of the world was a single 56.6 modem. It's much better now, but still not quite up to state-of-the-art speed. - Matt -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
Ghery S. Pettit Send message Joined: 7 Nov 99 Posts: 325 Credit: 28,109,066 RAC: 82 |
Good luck on the jury duty Matt. I was selected for mine starting Monday. Out of curiosity, I called up the jury line this week to see what their schedule was like. No jurors were called all week, so I expect to be called in a somewhat busier week. And I've been called up for a second time this year - Christmas week. I suspect I won't be going in. :-) |
Robi Send message Joined: 24 Oct 00 Posts: 33 Credit: 886,890 RAC: 1 |
I believe the raw data is put into the high capacity storage ultimately. So why not put it there to begin with directly from Puerto Rico and download it to Berkeley as needed? Shuffling disks around is proving tedious and error prone, it sounds. All that needs to be done is "park" a communications satellite in the range of the Arecibo Telescope... et voilà, SETI@home's own GB/s backbone ;) Robi |
John McLeod VII Send message Joined: 15 Jul 99 Posts: 24806 Credit: 790,712 RAC: 0 |
I believe the raw data is put into the high capacity storage ultimately. So why not put it there to begin with directly from Puerto Rico and download it to Berkeley as needed? Shuffling disks around is proving tedious and error prone, it sounds. Which would interfere with the radio astronomy. BOINC WIKI |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.