Gorlin (Mar 25 2009)


log in

Advanced search

Message boards : Technical News : Gorlin (Mar 25 2009)

1 · 2 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 879199 - Posted: 25 Mar 2009, 21:07:21 UTC

Mmm-kay. So where are we at with the science database...? The morning today was much like yesterday: me, Eric, and Jeff shouting over the deafening noise of the server closet, taking turns hunched over a monitor attached directly to thumper (the kvm monitor was having separate issues). Lots of reboots and unexpected (and unpleasant) results. Lots of thinking we found the problem only to reboot and (five minutes later) finding we were wrong, then having to reboot again off of DVD (taking another five minutes).

Basically our discussions were along the lines of: Why does the boot metadevice disappear when booting off of DVD? And why does the root metadevice disappear when coming up via grub? Didn't we resync these two drives yesterday? Oh look - the grub device map is referring to /dev/sdm, which was how the root drive was ennumerated when there were only 24 drives in the system - it should be referring to /dev/sdy now that we have 48 - so this must be at least one of our problems! Nope. Changing that did nothing. Etc. etc. etc. etc.

Well, whatever. It's been a two-day-long game like a demented version Towers-of-Hanoi - swapping drives, installing/reinstalling grub, resyncing devices, reconfiguring mdadm, then going back to step one and trying a different permutation. On hindsight it probably would have been easier to just install a new OS from scratch (though we would have had to recreate a web of informix configuration which also exists on the root drives). Right now the system is actually up (finally) and resyncing one mirror (again) and will have to sync another once that's finished. So we're offline for another day, and we haven't even gotten to the pulse table problems yet. I will stil try to get Astropulse running in some form later on today/tonight.

Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile TOM
Volunteer tester
Avatar
Send message
Joined: 5 Apr 01
Posts: 59
Credit: 54,930,063
RAC: 0
Germany
Message 879281 - Posted: 26 Mar 2009, 1:15:54 UTC - in response to Message 879199.

Is there a difference on how the drives are enumerated when you boot from grub when you had 24 drives or 48 and between 24 and 48 when you boot from DVD?
- Tom
____________

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46126
Credit: 36,599,330
RAC: 5,286
Message 879304 - Posted: 26 Mar 2009, 2:39:43 UTC - in response to Message 879199.

Mmm-kay. So where are we at with the science database...? The morning today was much like yesterday: me, Eric, and Jeff shouting over the deafening noise of the server closet, taking turns hunched over a monitor attached directly to thumper (the kvm monitor was having separate issues). Lots of reboots and unexpected (and unpleasant) results. Lots of thinking we found the problem only to reboot and (five minutes later) finding we were wrong, then having to reboot again off of DVD (taking another five minutes).

Basically our discussions were along the lines of: Why does the boot metadevice disappear when booting off of DVD? And why does the root metadevice disappear when coming up via grub? Didn't we resync these two drives yesterday? Oh look - the grub device map is referring to /dev/sdm, which was how the root drive was enumerated when there were only 24 drives in the system - it should be referring to /dev/sdy now that we have 48 - so this must be at least one of our problems! Nope. Changing that did nothing. Etc. etc. etc. etc.

Well, whatever. It's been a two-day-long game like a demented version Towers-of-Hanoi - swapping drives, installing/reinstalling grub, resyncing devices, reconfiguring mdadm, then going back to step one and trying a different permutation. On hindsight it probably would have been easier to just install a new OS from scratch (though we would have had to recreate a web of informix configuration which also exists on the root drives). Right now the system is actually up (finally) and resyncing one mirror (again) and will have to sync another once that's finished. So we're offline for another day, and we haven't even gotten to the pulse table problems yet. I will still try to get Astropulse running in some form later on today/tonight.

Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw.

- Matt

Are You sure they weren't Hunting for Hossenfeffer?
____________
My Facebook, War Commander, 2015

Profile Gundolf Jahn
Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 357,953
RAC: 18
Germany
Message 879352 - Posted: 26 Mar 2009, 9:16:08 UTC - in response to Message 879304.
Last modified: 26 Mar 2009, 9:42:25 UTC

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

Profile KWSN Ekky Ekky Ekky
Avatar
Send message
Joined: 25 May 99
Posts: 922
Credit: 11,384,611
RAC: 13,101
United Kingdom
Message 879399 - Posted: 26 Mar 2009, 14:44:14 UTC - in response to Message 879352.

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

I suspect a reference to an old episode of Bugs Bunny

"Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc.

____________

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46126
Credit: 36,599,330
RAC: 5,286
Message 879406 - Posted: 26 Mar 2009, 14:56:47 UTC - in response to Message 879399.
Last modified: 26 Mar 2009, 15:03:27 UTC

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

I suspect a reference to an old episode of Bugs Bunny

"Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc.

Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling).
____________
My Facebook, War Commander, 2015

Nemesis
Send message
Joined: 14 Mar 07
Posts: 129
Credit: 31,295,655
RAC: 1,063
Canada
Message 879419 - Posted: 26 Mar 2009, 15:31:26 UTC - in response to Message 879406.

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

I suspect a reference to an old episode of Bugs Bunny

"Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc.

Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling).


Yup, one of my all time Bugs faves except for maybe "Hocus Pocus / Newport News"

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46126
Credit: 36,599,330
RAC: 5,286
Message 879449 - Posted: 26 Mar 2009, 16:22:46 UTC - in response to Message 879419.

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

I suspect a reference to an old episode of Bugs Bunny

"Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc.

Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling).


Yup, one of my all time Bugs faves except for maybe "Hocus Pocus / Newport News"

Of course this is the One I was thinking of: The Big Snooze which until recently I didn't realize is older than Me.

____________
My Facebook, War Commander, 2015

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38925
Credit: 579,226,955
RAC: 510,080
United States
Message 879504 - Posted: 26 Mar 2009, 17:59:58 UTC - in response to Message 879199.



Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw.

- Matt


Best of luck getting it all sorted Matt.....I know from my own little crunching farm experience how frustrating it can be when things just don't work for unexplainable reasons....and the complexity of your systems there are multitudes greater.
Next time, keep those Einstein boyz outta the server closet, eh? I think they threw ya a mickey....LOL.

And nice to see you have a little time left over for some science again.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46126
Credit: 36,599,330
RAC: 5,286
Message 879516 - Posted: 26 Mar 2009, 18:18:15 UTC - in response to Message 879504.



Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw.

- Matt


Best of luck getting it all sorted Matt.....I know from my own little crunching farm experience how frustrating it can be when things just don't work for unexplainable reasons....and the complexity of your systems there are multitudes greater.
Next time, keep those Einstein boyz outta the server closet, eh? I think they threw ya a mickey....LOL.

And nice to see you have a little time left over for some science again.

Yeah I agree those Einstein guys like black holes too much, So keep'em away from the servers.
____________
My Facebook, War Commander, 2015

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12288
Credit: 2,576,849
RAC: 818
Netherlands
Message 879801 - Posted: 27 Mar 2009, 18:20:39 UTC - in response to Message 879199.

Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

It looks like they were jealous.

All I am getting for Einstein is: "The server at einstein.phys.uwm.edu is taking too long to respond." and 27-Mar-09 19:17:47 Einstein@Home Scheduler request failed: HTTP file not found


____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Profile Gundolf Jahn
Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 357,953
RAC: 18
Germany
Message 879804 - Posted: 27 Mar 2009, 18:29:58 UTC - in response to Message 879801.

Yeah, but the SETI Server status page too hangs since 13:20:10 UTC ;-)

Gruß,
Gundolf
____________
Computer sind nicht alles im Leben. (Kleiner Scherz)

SETI@home classic workunits 3,758
SETI@home classic CPU time 66,520 hours

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12288
Credit: 2,576,849
RAC: 818
Netherlands
Message 879805 - Posted: 27 Mar 2009, 18:35:44 UTC - in response to Message 879801.

Einstein is having its own problems, all right:

David Hammer wrote:
We are having some filesystem problems.

I might have it running again in a few hours if all goes well but I don't know if a simple fsck is going to fix the problem yet.

No I was not jealous. ;)

David

____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Profile speedimic
Volunteer tester
Avatar
Send message
Joined: 28 Sep 02
Posts: 362
Credit: 16,590,653
RAC: 0
Germany
Message 879839 - Posted: 27 Mar 2009, 20:28:24 UTC - in response to Message 879805.

Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all beeing down or at least having serious problems...
____________
mic.


zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46126
Credit: 36,599,330
RAC: 5,286
Message 879842 - Posted: 27 Mar 2009, 20:50:09 UTC - in response to Message 879839.

Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all being down or at least having serious problems...

What Folding doesn't have a problem?
____________
My Facebook, War Commander, 2015

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12410
Credit: 6,717,857
RAC: 8,471
United States
Message 879853 - Posted: 27 Mar 2009, 21:17:05 UTC - in response to Message 879839.

Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all beeing down or at least having serious problems...

The little green men have counted and we have too many humans in orbit right now. As soon as the shuttle lands everything will go back to normal. ;)


____________

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12288
Credit: 2,576,849
RAC: 818
Netherlands
Message 879863 - Posted: 27 Mar 2009, 22:03:15 UTC - in response to Message 879839.

News from Milkyway@Home:

Travis Desell wrote:
We're trying to get the 2nd project set up for the GPUs, and it looks like there was some misunderstanding or miscommunication with labstaff in doing it. Hopefully things should be back up shortly.

____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Profile MarkJProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 08
Posts: 937
Credit: 22,630,588
RAC: 88,405
Australia
Message 879890 - Posted: 27 Mar 2009, 23:59:52 UTC - in response to Message 879199.

Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.


It must be sympathy pains, apparently their file server has crashed too and is being rebuilt. Sounds a lot like Thumper. When you try and logon to their website it comes up with this message (after about 3 mins):

Einstein@Home is down due to a fileserver crash. We are working to bring the project back online ASAP. UPDATE: Fri Mar 27 22:39:13 UTC 2009 The filesystem needs to be repaired so the project will likely be down for at least 12 more hours. Thank you for your patience.




____________
BOINC blog

Profile SATAN
Avatar
Send message
Joined: 27 Aug 06
Posts: 835
Credit: 2,094,367
RAC: 0
United Kingdom
Message 879892 - Posted: 28 Mar 2009, 0:02:10 UTC

Ageless, thanks for the update from Travis.
____________

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12288
Credit: 2,576,849
RAC: 818
Netherlands
Message 879893 - Posted: 28 Mar 2009, 0:05:17 UTC - in response to Message 879892.

Since I seem to have all the news on the space themed projects, here's another one. ;-)

Cosmology@Home news update.

Scott Kruger wrote:
Jord-

It's been incredibly frustrating. We've gotten a bunch of the hardware replaced, but we're still having a bunch of IO errors. I suspect that there are some bad sectors on one of the RAID drives, so I'm going to have to back up the entire array and reformat it.

If all goes well, it may be up tonight (at least the website).

-Scott

____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

1 · 2 · Next

Message boards : Technical News : Gorlin (Mar 25 2009)

Copyright © 2014 University of California