Gorlin (Mar 25 2009)

Message boards : Technical News : Gorlin (Mar 25 2009)

Author	Message
Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 879199 - Posted: 25 Mar 2009, 21:07:21 UTC
	Mmm-kay. So where are we at with the science database...? The morning today was much like yesterday: me, Eric, and Jeff shouting over the deafening noise of the server closet, taking turns hunched over a monitor attached directly to thumper (the kvm monitor was having separate issues). Lots of reboots and unexpected (and unpleasant) results. Lots of thinking we found the problem only to reboot and (five minutes later) finding we were wrong, then having to reboot again off of DVD (taking another five minutes). Basically our discussions were along the lines of: Why does the boot metadevice disappear when booting off of DVD? And why does the root metadevice disappear when coming up via grub? Didn't we resync these two drives yesterday? Oh look - the grub device map is referring to /dev/sdm, which was how the root drive was ennumerated when there were only 24 drives in the system - it should be referring to /dev/sdy now that we have 48 - so this must be at least one of our problems! Nope. Changing that did nothing. Etc. etc. etc. etc. Well, whatever. It's been a two-day-long game like a demented version Towers-of-Hanoi - swapping drives, installing/reinstalling grub, resyncing devices, reconfiguring mdadm, then going back to step one and trying a different permutation. On hindsight it probably would have been easier to just install a new OS from scratch (though we would have had to recreate a web of informix configuration which also exists on the root drives). Right now the system is actually up (finally) and resyncing one mirror (again) and will have to sync another once that's finished. So we're offline for another day, and we haven't even gotten to the pulse table problems yet. I will stil try to get Astropulse running in some form later on today/tonight. Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week. Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw. - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 879199 ·

TOM Volunteer tester Send message Joined: 5 Apr 01 Posts: 58 Credit: 51,437,507 RAC: 92,701	Message 879281 - Posted: 26 Mar 2009, 1:15:54 UTC - in response to Message 879199.
	Is there a difference on how the drives are enumerated when you boot from grub when you had 24 drives or 48 and between 24 and 48 when you boot from DVD? - Tom ____________
	ID: 879281 ·

VBobier Send message Joined: 30 Nov 03 Posts: 40845 Credit: 32,130,722 RAC: 18,967	Message 879304 - Posted: 26 Mar 2009, 2:39:43 UTC - in response to Message 879199.
	Mmm-kay. So where are we at with the science database...? The morning today was much like yesterday: me, Eric, and Jeff shouting over the deafening noise of the server closet, taking turns hunched over a monitor attached directly to thumper (the kvm monitor was having separate issues). Lots of reboots and unexpected (and unpleasant) results. Lots of thinking we found the problem only to reboot and (five minutes later) finding we were wrong, then having to reboot again off of DVD (taking another five minutes). Basically our discussions were along the lines of: Why does the boot metadevice disappear when booting off of DVD? And why does the root metadevice disappear when coming up via grub? Didn't we resync these two drives yesterday? Oh look - the grub device map is referring to /dev/sdm, which was how the root drive was enumerated when there were only 24 drives in the system - it should be referring to /dev/sdy now that we have 48 - so this must be at least one of our problems! Nope. Changing that did nothing. Etc. etc. etc. etc. Well, whatever. It's been a two-day-long game like a demented version Towers-of-Hanoi - swapping drives, installing/reinstalling grub, resyncing devices, reconfiguring mdadm, then going back to step one and trying a different permutation. On hindsight it probably would have been easier to just install a new OS from scratch (though we would have had to recreate a web of informix configuration which also exists on the root drives). Right now the system is actually up (finally) and resyncing one mirror (again) and will have to sync another once that's finished. So we're offline for another day, and we haven't even gotten to the pulse table problems yet. I will still try to get Astropulse running in some form later on today/tonight. Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week. Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw. - Matt Are You sure they weren't Hunting for Hossenfeffer? ____________ BSG Anthem My Facebook page
	ID: 879304 ·

Gundolf Jahn Send message Joined: 19 Sep 00 Posts: 3076 Credit: 341,746 RAC: 101	Message 879352 - Posted: 26 Mar 2009, 9:16:08 UTC - in response to Message 879304. Last modified: 26 Mar 2009, 9:42:25 UTC
	Are You sure they weren't Hunting for Hossenfeffer? Who or what is Hossenfeffer? GruÃŸ, Gundolf [edit] Hasenpfeffer? [/edit]
	ID: 879352 ·

KWSN Ekky Ekky Ekky Send message Joined: 25 May 99 Posts: 898 Credit: 6,157,149 RAC: 10,878	Message 879399 - Posted: 26 Mar 2009, 14:44:14 UTC - in response to Message 879352.
	Are You sure they weren't Hunting for Hossenfeffer? Who or what is Hossenfeffer? GruÃŸ, Gundolf [edit] Hasenpfeffer? [/edit] I suspect a reference to an old episode of Bugs Bunny "Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc. ____________
	ID: 879399 ·

VBobier Send message Joined: 30 Nov 03 Posts: 40845 Credit: 32,130,722 RAC: 18,967	Message 879406 - Posted: 26 Mar 2009, 14:56:47 UTC - in response to Message 879399. Last modified: 26 Mar 2009, 15:03:27 UTC
	Are You sure they weren't Hunting for Hossenfeffer? Who or what is Hossenfeffer? GruÃŸ, Gundolf [edit] Hasenpfeffer? [/edit] I suspect a reference to an old episode of Bugs Bunny "Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc. Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling). ____________ BSG Anthem My Facebook page
	ID: 879406 ·

Nemesis Send message Joined: 14 Mar 07 Posts: 129 Credit: 29,273,844 RAC: 19,270	Message 879419 - Posted: 26 Mar 2009, 15:31:26 UTC - in response to Message 879406.
	Are You sure they weren't Hunting for Hossenfeffer? Who or what is Hossenfeffer? GruÃŸ, Gundolf [edit] Hasenpfeffer? [/edit] I suspect a reference to an old episode of Bugs Bunny "Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc. Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling). Yup, one of my all time Bugs faves except for maybe "Hocus Pocus / Newport News"
	ID: 879419 ·

VBobier Send message Joined: 30 Nov 03 Posts: 40845 Credit: 32,130,722 RAC: 18,967	Message 879449 - Posted: 26 Mar 2009, 16:22:46 UTC - in response to Message 879419.
	Are You sure they weren't Hunting for Hossenfeffer? Who or what is Hossenfeffer? GruÃŸ, Gundolf [edit] Hasenpfeffer? [/edit] I suspect a reference to an old episode of Bugs Bunny "Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc. Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling). Yup, one of my all time Bugs faves except for maybe "Hocus Pocus / Newport News" Of course this is the One I was thinking of: The Big Snooze which until recently I didn't realize is older than Me. ____________ BSG Anthem My Facebook page
	ID: 879449 ·

msattler Volunteer tester Send message Joined: 9 Jul 00 Posts: 34169 Credit: 345,050,231 RAC: 549,541	Message 879504 - Posted: 26 Mar 2009, 17:59:58 UTC - in response to Message 879199.
	Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week. Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw. - Matt Best of luck getting it all sorted Matt.....I know from my own little crunching farm experience how frustrating it can be when things just don't work for unexplainable reasons....and the complexity of your systems there are multitudes greater. Next time, keep those Einstein boyz outta the server closet, eh? I think they threw ya a mickey....LOL. And nice to see you have a little time left over for some science again. ____________ ****** "Ask not, what your kitty can do for you. Ask what you can do for your kitty." As it is kitten, so shall it be done.
	ID: 879504 ·

VBobier Send message Joined: 30 Nov 03 Posts: 40845 Credit: 32,130,722 RAC: 18,967	Message 879516 - Posted: 26 Mar 2009, 18:18:15 UTC - in response to Message 879504.
	Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week. Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw. - Matt Best of luck getting it all sorted Matt.....I know from my own little crunching farm experience how frustrating it can be when things just don't work for unexplainable reasons....and the complexity of your systems there are multitudes greater. Next time, keep those Einstein boyz outta the server closet, eh? I think they threw ya a mickey....LOL. And nice to see you have a little time left over for some science again. Yeah I agree those Einstein guys like black holes too much, So keep'em away from the servers. ____________ BSG Anthem My Facebook page
	ID: 879516 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11639 Credit: 2,007,175 RAC: 4,126	Message 879801 - Posted: 27 Mar 2009, 18:20:39 UTC - in response to Message 879199.
	Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week. It looks like they were jealous. All I am getting for Einstein is: "The server at einstein.phys.uwm.edu is taking too long to respond." and 27-Mar-09 19:17:47 Einstein@Home Scheduler request failed: HTTP file not found ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 879801 ·

Gundolf Jahn Send message Joined: 19 Sep 00 Posts: 3076 Credit: 341,746 RAC: 101	Message 879804 - Posted: 27 Mar 2009, 18:29:58 UTC - in response to Message 879801.
	Yeah, but the SETI Server status page too hangs since 13:20:10 UTC ;-) GruÃŸ, Gundolf ____________ Computer sind nicht alles im Leben. (Kleiner Scherz) SETI@home classic workunits 3,758 SETI@home classic CPU time 66,520 hours
	ID: 879804 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11639 Credit: 2,007,175 RAC: 4,126	Message 879805 - Posted: 27 Mar 2009, 18:35:44 UTC - in response to Message 879801.
	Einstein is having its own problems, all right: David Hammer wrote: We are having some filesystem problems. I might have it running again in a few hours if all goes well but I don't know if a simple fsck is going to fix the problem yet. No I was not jealous. ;) David ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 879805 ·

speedimic Volunteer tester Send message Joined: 28 Sep 02 Posts: 362 Credit: 16,590,653 RAC: 0	Message 879839 - Posted: 27 Mar 2009, 20:28:24 UTC - in response to Message 879805.
	Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all beeing down or at least having serious problems... ____________ mic.
	ID: 879839 ·

VBobier Send message Joined: 30 Nov 03 Posts: 40845 Credit: 32,130,722 RAC: 18,967	Message 879842 - Posted: 27 Mar 2009, 20:50:09 UTC - in response to Message 879839.
	Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all being down or at least having serious problems... What Folding doesn't have a problem? ____________ BSG Anthem My Facebook page
	ID: 879842 ·

Gary Charpentier Volunteer tester Send message Joined: 25 Dec 00 Posts: 10327 Credit: 5,098,736 RAC: 6,300	Message 879853 - Posted: 27 Mar 2009, 21:17:05 UTC - in response to Message 879839.
	Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all beeing down or at least having serious problems... The little green men have counted and we have too many humans in orbit right now. As soon as the shuttle lands everything will go back to normal. ;) ____________
	ID: 879853 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11639 Credit: 2,007,175 RAC: 4,126	Message 879863 - Posted: 27 Mar 2009, 22:03:15 UTC - in response to Message 879839.
	News from Milkyway@Home: Travis Desell wrote: We're trying to get the 2nd project set up for the GPUs, and it looks like there was some misunderstanding or miscommunication with labstaff in doing it. Hopefully things should be back up shortly. ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 879863 ·

MarkJ Volunteer tester Send message Joined: 17 Feb 08 Posts: 912 Credit: 14,583,555 RAC: 9,785	Message 879890 - Posted: 27 Mar 2009, 23:59:52 UTC - in response to Message 879199.
	Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week. It must be sympathy pains, apparently their file server has crashed too and is being rebuilt. Sounds a lot like Thumper. When you try and logon to their website it comes up with this message (after about 3 mins): Einstein@Home is down due to a fileserver crash. We are working to bring the project back online ASAP. UPDATE: Fri Mar 27 22:39:13 UTC 2009 The filesystem needs to be repaired so the project will likely be down for at least 12 more hours. Thank you for your patience. ____________ BOINC blog
	ID: 879890 ·

SATAN Send message Joined: 27 Aug 06 Posts: 835 Credit: 2,094,367 RAC: 0	Message 879892 - Posted: 28 Mar 2009, 0:02:10 UTC
	Ageless, thanks for the update from Travis. ____________
	ID: 879892 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11639 Credit: 2,007,175 RAC: 4,126	Message 879893 - Posted: 28 Mar 2009, 0:05:17 UTC - in response to Message 879892.
	Since I seem to have all the news on the space themed projects, here's another one. ;-) Cosmology@Home news update. Scott Kruger wrote: Jord- It's been incredibly frustrating. We've gotten a bunch of the hardware replaced, but we're still having a bunch of IO errors. I suspect that there are some bad sectors on one of the RAID drives, so I'm going to have to back up the entire array and reformat it. If all goes well, it may be up tonight (at least the website). -Scott ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 879893 ·

1 · 2 · Next

Message boards : Technical News : Gorlin (Mar 25 2009)