Gorlin (Mar 25 2009)

Message boards : Technical News : Gorlin (Mar 25 2009)
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 879199 - Posted: 25 Mar 2009, 21:07:21 UTC

Mmm-kay. So where are we at with the science database...? The morning today was much like yesterday: me, Eric, and Jeff shouting over the deafening noise of the server closet, taking turns hunched over a monitor attached directly to thumper (the kvm monitor was having separate issues). Lots of reboots and unexpected (and unpleasant) results. Lots of thinking we found the problem only to reboot and (five minutes later) finding we were wrong, then having to reboot again off of DVD (taking another five minutes).

Basically our discussions were along the lines of: Why does the boot metadevice disappear when booting off of DVD? And why does the root metadevice disappear when coming up via grub? Didn't we resync these two drives yesterday? Oh look - the grub device map is referring to /dev/sdm, which was how the root drive was ennumerated when there were only 24 drives in the system - it should be referring to /dev/sdy now that we have 48 - so this must be at least one of our problems! Nope. Changing that did nothing. Etc. etc. etc. etc.

Well, whatever. It's been a two-day-long game like a demented version Towers-of-Hanoi - swapping drives, installing/reinstalling grub, resyncing devices, reconfiguring mdadm, then going back to step one and trying a different permutation. On hindsight it probably would have been easier to just install a new OS from scratch (though we would have had to recreate a web of informix configuration which also exists on the root drives). Right now the system is actually up (finally) and resyncing one mirror (again) and will have to sync another once that's finished. So we're offline for another day, and we haven't even gotten to the pulse table problems yet. I will stil try to get Astropulse running in some form later on today/tonight.

Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 879199 · Report as offensive
Profile TOM
Volunteer tester
Avatar

Send message
Joined: 5 Apr 01
Posts: 53
Credit: 65,422,234
RAC: 86
Germany
Message 879281 - Posted: 26 Mar 2009, 1:15:54 UTC - in response to Message 879199.  

Is there a difference on how the drives are enumerated when you boot from grub when you had 24 drives or 48 and between 24 and 48 when you boot from DVD?
- Tom
ID: 879281 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65709
Credit: 55,293,173
RAC: 49
United States
Message 879304 - Posted: 26 Mar 2009, 2:39:43 UTC - in response to Message 879199.  

Mmm-kay. So where are we at with the science database...? The morning today was much like yesterday: me, Eric, and Jeff shouting over the deafening noise of the server closet, taking turns hunched over a monitor attached directly to thumper (the kvm monitor was having separate issues). Lots of reboots and unexpected (and unpleasant) results. Lots of thinking we found the problem only to reboot and (five minutes later) finding we were wrong, then having to reboot again off of DVD (taking another five minutes).

Basically our discussions were along the lines of: Why does the boot metadevice disappear when booting off of DVD? And why does the root metadevice disappear when coming up via grub? Didn't we resync these two drives yesterday? Oh look - the grub device map is referring to /dev/sdm, which was how the root drive was enumerated when there were only 24 drives in the system - it should be referring to /dev/sdy now that we have 48 - so this must be at least one of our problems! Nope. Changing that did nothing. Etc. etc. etc. etc.

Well, whatever. It's been a two-day-long game like a demented version Towers-of-Hanoi - swapping drives, installing/reinstalling grub, resyncing devices, reconfiguring mdadm, then going back to step one and trying a different permutation. On hindsight it probably would have been easier to just install a new OS from scratch (though we would have had to recreate a web of informix configuration which also exists on the root drives). Right now the system is actually up (finally) and resyncing one mirror (again) and will have to sync another once that's finished. So we're offline for another day, and we haven't even gotten to the pulse table problems yet. I will still try to get Astropulse running in some form later on today/tonight.

Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw.

- Matt

Are You sure they weren't Hunting for Hossenfeffer?
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 879304 · Report as offensive
Profile Gundolf Jahn

Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 446,358
RAC: 0
Germany
Message 879352 - Posted: 26 Mar 2009, 9:16:08 UTC - in response to Message 879304.  
Last modified: 26 Mar 2009, 9:42:25 UTC

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]
ID: 879352 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 944
Credit: 52,956,491
RAC: 67
United Kingdom
Message 879399 - Posted: 26 Mar 2009, 14:44:14 UTC - in response to Message 879352.  

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

I suspect a reference to an old episode of Bugs Bunny

"Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc.


ID: 879399 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65709
Credit: 55,293,173
RAC: 49
United States
Message 879406 - Posted: 26 Mar 2009, 14:56:47 UTC - in response to Message 879399.  
Last modified: 26 Mar 2009, 15:03:27 UTC

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

I suspect a reference to an old episode of Bugs Bunny

"Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc.

Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling).
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 879406 · Report as offensive
Nemesis

Send message
Joined: 14 Mar 07
Posts: 129
Credit: 31,295,655
RAC: 0
Canada
Message 879419 - Posted: 26 Mar 2009, 15:31:26 UTC - in response to Message 879406.  

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

I suspect a reference to an old episode of Bugs Bunny

"Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc.

Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling).


Yup, one of my all time Bugs faves except for maybe "Hocus Pocus / Newport News"
ID: 879419 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65709
Credit: 55,293,173
RAC: 49
United States
Message 879449 - Posted: 26 Mar 2009, 16:22:46 UTC - in response to Message 879419.  

Are You sure they weren't Hunting for Hossenfeffer?

Who or what is Hossenfeffer?

Gruß,
Gundolf

[edit] Hasenpfeffer? [/edit]

I suspect a reference to an old episode of Bugs Bunny

"Bring me my Hossenfeffer!" says the King. Bugs is brought in on a lordly dish with a silver cover, "Mmm, what's up Doc?" etc. etc.

Bingo as My Uncle would say, You win a cookie. Yeah, Rabbit Stew is what Google would find(the recipe, among others due to a misspelling).


Yup, one of my all time Bugs faves except for maybe "Hocus Pocus / Newport News"

Of course this is the One I was thinking of: The Big Snooze which until recently I didn't realize is older than Me.

The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 879449 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 879504 - Posted: 26 Mar 2009, 17:59:58 UTC - in response to Message 879199.  



Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw.

- Matt


Best of luck getting it all sorted Matt.....I know from my own little crunching farm experience how frustrating it can be when things just don't work for unexplainable reasons....and the complexity of your systems there are multitudes greater.
Next time, keep those Einstein boyz outta the server closet, eh? I think they threw ya a mickey....LOL.

And nice to see you have a little time left over for some science again.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 879504 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65709
Credit: 55,293,173
RAC: 49
United States
Message 879516 - Posted: 26 Mar 2009, 18:18:15 UTC - in response to Message 879504.  



Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

Meanwhile, I'm back on radar blanking detail. We're now trying cross-correlations to match radar patterns using fftw.

- Matt


Best of luck getting it all sorted Matt.....I know from my own little crunching farm experience how frustrating it can be when things just don't work for unexplainable reasons....and the complexity of your systems there are multitudes greater.
Next time, keep those Einstein boyz outta the server closet, eh? I think they threw ya a mickey....LOL.

And nice to see you have a little time left over for some science again.

Yeah I agree those Einstein guys like black holes too much, So keep'em away from the servers.
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 879516 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 879801 - Posted: 27 Mar 2009, 18:20:39 UTC - in response to Message 879199.  

Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.

It looks like they were jealous.

All I am getting for Einstein is: "The server at einstein.phys.uwm.edu is taking too long to respond." and 27-Mar-09 19:17:47 Einstein@Home Scheduler request failed: HTTP file not found


ID: 879801 · Report as offensive
Profile Gundolf Jahn

Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 446,358
RAC: 0
Germany
Message 879804 - Posted: 27 Mar 2009, 18:29:58 UTC - in response to Message 879801.  

Yeah, but the SETI Server status page too hangs since 13:20:10 UTC ;-)

Gruß,
Gundolf
Computer sind nicht alles im Leben. (Kleiner Scherz)

SETI@home classic workunits 3,758
SETI@home classic CPU time 66,520 hours
ID: 879804 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 879805 - Posted: 27 Mar 2009, 18:35:44 UTC - in response to Message 879801.  

Einstein is having its own problems, all right:

David Hammer wrote:
We are having some filesystem problems.

I might have it running again in a few hours if all goes well but I don't know if a simple fsck is going to fix the problem yet.

No I was not jealous. ;)

David

ID: 879805 · Report as offensive
Profile speedimic
Volunteer tester
Avatar

Send message
Joined: 28 Sep 02
Posts: 362
Credit: 16,590,653
RAC: 0
Germany
Message 879839 - Posted: 27 Mar 2009, 20:28:24 UTC - in response to Message 879805.  

Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all beeing down or at least having serious problems...
mic.


ID: 879839 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65709
Credit: 55,293,173
RAC: 49
United States
Message 879842 - Posted: 27 Mar 2009, 20:50:09 UTC - in response to Message 879839.  

Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all being down or at least having serious problems...

What Folding doesn't have a problem?
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 879842 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30608
Credit: 53,134,872
RAC: 32
United States
Message 879853 - Posted: 27 Mar 2009, 21:17:05 UTC - in response to Message 879839.  

Must be a space-science-epidemic; Cosmo, Milkyway, Einstein and Seti all beeing down or at least having serious problems...

The little green men have counted and we have too many humans in orbit right now. As soon as the shuttle lands everything will go back to normal. ;)


ID: 879853 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 879863 - Posted: 27 Mar 2009, 22:03:15 UTC - in response to Message 879839.  

News from Milkyway@Home:

Travis Desell wrote:
We're trying to get the 2nd project set up for the GPUs, and it looks like there was some misunderstanding or miscommunication with labstaff in doing it. Hopefully things should be back up shortly.

ID: 879863 · Report as offensive
MarkJ Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 1139
Credit: 80,854,192
RAC: 5
Australia
Message 879890 - Posted: 27 Mar 2009, 23:59:52 UTC - in response to Message 879199.  

Funny thing: Oliver and Bernd of Einstein@home have been visiting from Germany, collaborating with Dave on some general BOINC stuff. They left just a couple hours ago, but we did discuss how when SETI@home is having issues such as this, Einstein@home certainly gets a huge "bump" from the suddenly influx of free CPU time. We joked how the these thumper issues strangely coincided with their arrival last week.


It must be sympathy pains, apparently their file server has crashed too and is being rebuilt. Sounds a lot like Thumper. When you try and logon to their website it comes up with this message (after about 3 mins):

Einstein@Home is down due to a fileserver crash. We are working to bring the project back online ASAP. UPDATE: Fri Mar 27 22:39:13 UTC 2009 The filesystem needs to be repaired so the project will likely be down for at least 12 more hours. Thank you for your patience.




BOINC blog
ID: 879890 · Report as offensive
Profile SATAN
Avatar

Send message
Joined: 27 Aug 06
Posts: 835
Credit: 2,129,006
RAC: 0
United Kingdom
Message 879892 - Posted: 28 Mar 2009, 0:02:10 UTC

Ageless, thanks for the update from Travis.
ID: 879892 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 879893 - Posted: 28 Mar 2009, 0:05:17 UTC - in response to Message 879892.  

Since I seem to have all the news on the space themed projects, here's another one. ;-)

Cosmology@Home news update.

Scott Kruger wrote:
Jord-

It's been incredibly frustrating. We've gotten a bunch of the hardware replaced, but we're still having a bunch of IO errors. I suspect that there are some bad sectors on one of the RAID drives, so I'm going to have to back up the entire array and reformat it.

If all goes well, it may be up tonight (at least the website).

-Scott

ID: 879893 · Report as offensive
1 · 2 · Next

Message boards : Technical News : Gorlin (Mar 25 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.