Loss Leader (Dec 16 2008)


log in

Advanced search

Message boards : Technical News : Loss Leader (Dec 16 2008)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1390
Credit: 74,079
RAC: 0
United States
Message 840652 - Posted: 16 Dec 2008, 23:43:25 UTC

First and foremost, it's snowing outside. This doesn't happen very often around here.

So today was an outage day - with one unexpected surprise: a visit from Court, systems administrator extraordinaire here in our lab a couple years back. Nice to see him again and catch up.

The standard outage stuff was, well, standard. Allow me to remind our new readers: Weekly we "compress" the mysql databases (which bloat from continual inserts/deletes all week, much like disk fragmentation) and back them up. These databases contain all the user/host/team info, and who is working on which workunits - basically all the generic volunteer computing stuff. The science is all kept in a separate database (using an Informix engine) on a different server altogether. The latter doesn't suffer from the same bloat, so we can do simple no-frills backups to disk while the database is live, without much ado. In theory we could do the mysql dumps live as well, but we choose to take things down to ensure the master/replica databases are in sync, and allow us some regular downtime to take care of pending server tasks. For example...

Today we finally turned off the old Network Appliance - a NAS server which worked fast and wonderfully, but (a) was only 3 Terabytes raw storage, (b) took up one third of our server closet, and (c) the individual disks have been failing at an increasing rate. We moved all of its functionality elsewhere already, so it was time to say goodbye. Jeff and I tore it apart shelf by shelf. Any sadness was lost in the joy of now having a completely empty rack full of completely useful shelves (we've had ridiculous problems finding racks/shelves that matched in the past). It's kind of funny the most useful part of that system at this point was its racks/shelves. We put all the recently donated Overland Storage servers into this now-empty rack (containing 10 Terabytes worth of storage), as well as anakin (the scheduling server), and there's still room for a lot more stuff. We still have to configure/employ all this new storage, but it's all plugged in and on line at least.

Recovery from the outage is usually painful. Today seems a little worse. Part of that is our work-to-send queue is at zero and the splitters are waiting for some space to free up before creating new work. I also think server "bruno" is having result storage issues slowing things down (people are connecting okay, which they can push through the usual traffic jam). We might need to reconfigure/rebuild that RAID array sooner than later.

I brought the mini video camera to make a quick video tour of our server closet, but the noise of all the fans is so loud it's basically worthless. I did take some low-quality still photos though - I'll get those up on the web someday.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 13000
Credit: 7,666,058
RAC: 6,265
United States
Message 840661 - Posted: 17 Dec 2008, 0:17:14 UTC - in response to Message 840652.

Keep up the good work. I may be seeing that snow in a few hours down here in LA.

____________

Profile Fred J. Verster
Volunteer tester
Avatar
Send message
Joined: 21 Apr 04
Posts: 3252
Credit: 31,902,797
RAC: 257
Netherlands
Message 840669 - Posted: 17 Dec 2008, 0:47:50 UTC - in response to Message 840661.

Hi Matt, thanks for your update and work .It is nice that more raw storage is available.
It's not snowing, only -2C and it's gonna rain, temps will rise.

____________

Profile Neil Blaikie
Volunteer tester
Avatar
Send message
Joined: 17 May 99
Posts: 142
Credit: 6,632,679
RAC: 1,205
Canada
Message 840686 - Posted: 17 Dec 2008, 1:52:42 UTC
Last modified: 17 Dec 2008, 1:53:33 UTC

Good work as always and Thank you for at least spending some of your time recording the video / images even though the video is worthless.

Saw on the news that you guys were getting snow, probably goes with the very very cold temperatures with windchill of -50 that the western Canadian provinces are currently getting.

Here on the East coast of Canada, Montreal area, we are just waiting on 15+ cm's of snow coming tomorrow.

Thank you for your information and glad you managed to get a lot done today.
____________

Profile Jon (nanoreid)
Avatar
Send message
Joined: 16 Aug 07
Posts: 645
Credit: 583,870
RAC: 0
United States
Message 840699 - Posted: 17 Dec 2008, 2:24:41 UTC - in response to Message 840652.

Any idea when we will start seeing work packages again? My poor little laptop has nothing to do while I asleep now.

Thanks
____________
Hopefully the cosmos is not trying to reverse the charges.
Moderation in all things.

Profile Allie in Vancouver
Volunteer tester
Avatar
Send message
Joined: 16 Mar 07
Posts: 3949
Credit: 1,604,668
RAC: 0
Canada
Message 840736 - Posted: 17 Dec 2008, 4:07:10 UTC

Thanks for the quick background info on what happens on outage day. I had known that things ‘went down’ for a few hours on every Tuesday but never really understood why. (I probably still have no real idea, but your disk-defrag simile is something that even us less technical crunchers can understand.)

Also, on behalf of many others, we appreciate your occasional posting of pictures in the about/picture albums section. I check it out from time to time and find that it helps put a human face on things. :)
____________
Pure mathematics is, in its way, the poetry of logical ideas.

Albert Einstein

Zebra3
Avatar
Send message
Joined: 22 Oct 01
Posts: 186
Credit: 13,658,148
RAC: 0
Canada
Message 840744 - Posted: 17 Dec 2008, 4:46:14 UTC - in response to Message 840686.
Last modified: 17 Dec 2008, 5:00:53 UTC




Here on the East coast of Canada, Montreal area, we are just waiting on 15+ cm's of snow coming tomorrow.


I don't mean to burst your bubble Neil but the last time I looked Quebec was part of Upper Canada..as much as it chooses not to be and in no way could it be classed as the East Coast of Canada...Leave that up to us Bluenosers and the Newfies..I apoligize if that being too harsh,

Profile Neil Blaikie
Volunteer tester
Avatar
Send message
Joined: 17 May 99
Posts: 142
Credit: 6,632,679
RAC: 1,205
Canada
Message 840751 - Posted: 17 Dec 2008, 5:10:55 UTC - in response to Message 840744.

True, my apology. Still it is going to snow and much needed to help the huge ice rink I spent a good part of today making.
____________

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 13000
Credit: 7,666,058
RAC: 6,265
United States
Message 840763 - Posted: 17 Dec 2008, 5:48:34 UTC - in response to Message 840652.

I brought the mini video camera to make a quick video tour of our server closet, but the noise of all the fans is so loud it's basically worthless.


Make a call over to your TV people on campus and ask them if you can borrow a "Lav" short for lavaliere. It is the kind of mike that you clip on your collar or necktie. It should reduce that fan noise a bit and bring up your voice talking. If that isn't available then cheat and just shoot your talking part in a nice quiet room without echos and replace the spoiled audio. As long as your lips aren't on camera who is to know?


____________

Profile JLDun
Volunteer tester
Avatar
Send message
Joined: 21 Apr 06
Posts: 307
Credit: 51,623
RAC: 0
United States
Message 840774 - Posted: 17 Dec 2008, 6:43:25 UTC - in response to Message 840763.

I brought the mini video camera to make a quick video tour of our server closet, but the noise of all the fans is so loud it's basically worthless.
As long as your lips aren't on camera who is to know?

Is it necessary for you to be on camera, or can a Commentary Track be added on later?
____________

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 840821 - Posted: 17 Dec 2008, 13:04:48 UTC


. . . Thanks for the Updates Matt - and Accolades to Each of You @ Berkeley

[just edit the sound out - and, like others mention - Overdub where necessary]


____________
BOINC Wiki . . .

Science Status Page . . .

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1390
Credit: 74,079
RAC: 0
United States
Message 840964 - Posted: 17 Dec 2008, 18:38:57 UTC - in response to Message 840763.
Last modified: 17 Dec 2008, 18:39:16 UTC

Make a call over to your TV people on campus and ask them if you can borrow a "Lav" short for lavaliere.


In my other life I have 24 hour access to a full-fledged recording studio with tons of gear, and do lots of movie editing/production/scoring as well as regular ol' music recording etc. However, whether or not SETI will get any time on that work queue (to do sound overdubs) remains to be seen.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Dr.Argentum
Send message
Joined: 24 Nov 99
Posts: 6
Credit: 3,456,585
RAC: 908
Canada
Message 841089 - Posted: 17 Dec 2008, 21:34:16 UTC - in response to Message 840744.




Here on the East coast of Canada, Montreal area, we are just waiting on 15+ cm's of snow coming tomorrow.


I don't mean to burst your bubble Neil but the last time I looked Quebec was part of Upper Canada..as much as it chooses not to be and in no way could it be classed as the East Coast of Canada...Leave that up to us Bluenosers and the Newfies..I apoligize if that being too harsh,


Yikes! I can't let that one pass. Montreal was part of LOWER Canada, sited on the lower part of the St Lawrence River. I can't be too harsh as I could never remember who were the herring chokers...

Profile Neil Blaikie
Volunteer tester
Avatar
Send message
Joined: 17 May 99
Posts: 142
Credit: 6,632,679
RAC: 1,205
Canada
Message 841141 - Posted: 17 Dec 2008, 22:48:38 UTC - in response to Message 841089.
Last modified: 17 Dec 2008, 22:52:25 UTC

I am British with Canadian Permanent Residency so technical it doesn't really matter which part of Canada it is in :-)

Only thing that matters, is that Quebec is part of Canada (albeit for the foreseeable future at least).

Really is quite annoying though doing computing jobs and having to switch languages all the time, you get used to it though over time. I have one client who insists on sending messages to me and my colleagues in french, when it quite clearly states that all correspondents to the server admins be in ENGLISH. He pays well though so can't complain. Someone attempting to run a DB with no knowledge of IBM Dynamic Servers (Informix based) is really annoying, worst of all the guy gets paid to do it and yet sub-contracts the work out!
____________

The Watery Tart
Send message
Joined: 24 Aug 00
Posts: 1
Credit: 197,230
RAC: 0
Message 841148 - Posted: 17 Dec 2008, 22:58:22 UTC

It's snowing in Vegas, and sticking! Global weirding, here we come...
____________

Profile JLDun
Volunteer tester
Avatar
Send message
Joined: 21 Apr 06
Posts: 307
Credit: 51,623
RAC: 0
United States
Message 841286 - Posted: 18 Dec 2008, 4:08:23 UTC - in response to Message 841148.

It's snowing in Vegas, and sticking! Global weirding, here we come...

Had snow here- and slipped INSIDE the house. (Just a few bumps...)
____________

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 841975 - Posted: 19 Dec 2008, 13:41:29 UTC


. . . while it's snowin' where it usually doesn't - we are gettin' Sping-Like Weather here in the Carolina's ladeedaladeeda ;)

@ Matt - remember to provide a link to the Photographs of the 'closet'

- OR more appropriately put - let us know 'when' You do ;) Thanks mate . . .

added notes: 'most' of us left Quebec - when 'le separatist' movement took over Quebec [middle-late 70's] - and the majority of us ended up in California

> my friends had just begun filmin' 'Scanners' @ that time ;)


____________
BOINC Wiki . . .

Science Status Page . . .

Message boards : Technical News : Loss Leader (Dec 16 2008)

Copyright © 2014 University of California