Exodus (Dec 20 2011)


log in

Advanced search

Message boards : Technical News : Exodus (Dec 20 2011)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1179613 - Posted: 20 Dec 2011, 23:35:51 UTC

Hello, world. Here's another random, non-comprehensive status update regarding our servers, quite possibly the last one before the end of the year.

So what are we dealing with lately. Well, carolyn (the mysql server) seems to have some funky memory. Or maybe it's an overactive watchdog in the kernel. Hard to tell, but the warning messages we're getting aren't given us the warm fuzzies. Operations are more or less normal, so we're just keeping an eye on it for the moment (don't really want to do any surgery before the holidays). Meanwhile, it did have a standard issues CPU lock last week, requiring a hard reset and database recovery. However annoying (and it seems the modern day linux kernels are getting more and more prone to this sort of misbehavior) it's so far easy enough to recover from after hard power cycling the machine. We have to do this a lot on bruno (the upload/compute server) quite often, and oscar (the informix database server) the other day as well. Every time we eventually recover just fine.

Also mysql-wise, we seem to be having performance issues that defy easy understanding and explanation. Maybe this is memory related (hope not), but probably just due to some black-box mysql internal bookkeeping. During some testing/tweaking I turned off the daily stats dump scripts, and (oops) forgot to turn them back on. So there was a period of 5-6 days without stats dumps. Sorry about that.

Another thing we have to keep an eye on is server closet temperature. Seems like (without clear notification) we are already in "holiday energy curtailment" mode. With less people around, lab-wide environmental controls (which assist our server closet cooling) are ramped down to save energy. Makes sense, but that still means temperatures rise in our closet, which isn't happy-making. So far they only went up a degree or two on average. Just one more thing to worry about.

Onto brighter news. The gang over at the GPU Users Group has been incredibly helpful to us thus far. They recently donated a 45 JBOD drive array, and as of today 28 2TB and 6TB drives (for this array and/or data transport to/from Arecibo). We'll use this, along with more drives to come and another whole server, to upgrade various parts of our current server backend in the near future: science database server storage, upload server, download server, and main BOINC admin/compute server... They are still collecting donations over at their site (see our
donation page for a link to their paypal-based donation site) going towards this new hardware.

Happy holidays, safe travels, and all that. See you in the new year...

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4141
Credit: 33,583,845
RAC: 26,700
United Kingdom
Message 1179614 - Posted: 20 Dec 2011, 23:42:09 UTC - in response to Message 1179613.

Thanks for the update Matt, and for all the efforts,

Claggy

Thomas Arnold
Volunteer tester
Send message
Joined: 14 May 99
Posts: 56
Credit: 61,046,144
RAC: 0
United States
Message 1179616 - Posted: 20 Dec 2011, 23:56:34 UTC

+1

Thank you very much for the update. They are always appreciated.

And Happy Holidays, Bah Humbug, and all that too.

And Here's to all the best for our Setiathome community for the coming year. We hope to make a difference.

Tom


____________

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12729
Credit: 7,263,689
RAC: 17,604
United States
Message 1179620 - Posted: 21 Dec 2011, 0:01:47 UTC

Thanks for the update.

Merry Christmas and a Happy New Year.

____________

Bob Giel
Volunteer tester
Send message
Joined: 11 Jan 04
Posts: 55
Credit: 5,044,717
RAC: 449
United States
Message 1179657 - Posted: 21 Dec 2011, 4:10:32 UTC

Thanks for the update Matt.

You and the rest of the crew have a Merry Christmas and Happy New Year!

Profile Chris SProject donor
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 32091
Credit: 13,772,457
RAC: 25,459
United Kingdom
Message 1179681 - Posted: 21 Dec 2011, 10:11:38 UTC

Thanks for the update and the kind words for the GPU Group.

Profile MarkJProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 08
Posts: 942
Credit: 25,110,361
RAC: 26,825
Australia
Message 1179684 - Posted: 21 Dec 2011, 10:46:23 UTC
Last modified: 21 Dec 2011, 10:47:03 UTC

Hi Matt,

Thanks for the updates. Its always interesting to find out what is happening behind the scenes.

Question:
The project is asking for one or more 240Gb (or larger) SSD's. Whatever happened with the Intel X25's? They must have died from too many IOPS by now. What are the new ones likely to be used for?

Cheers,
MarkJ

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 11982
Credit: 14,648,885
RAC: 12,540
United States
Message 1179736 - Posted: 21 Dec 2011, 15:02:40 UTC

Thanks, Matt. We always appreciate updates, the more detailed the better.

And have a happy holiday of your choice.

____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


Profile Slavac
Volunteer tester
Avatar
Send message
Joined: 27 Apr 11
Posts: 1932
Credit: 17,952,639
RAC: 0
United States
Message 1179810 - Posted: 21 Dec 2011, 20:10:31 UTC - in response to Message 1179684.

Hi Matt,

Thanks for the updates. Its always interesting to find out what is happening behind the scenes.

Question:
The project is asking for one or more 240Gb (or larger) SSD's. Whatever happened with the Intel X25's? They must have died from too many IOPS by now. What are the new ones likely to be used for?

Cheers,
MarkJ


Hey Mark:

I'll make a post over in the NC thread soon about the SSD's.
____________


Executive Director GPU Users Group Inc. -
brad@gpuug.org

Profile RottenMutt
Avatar
Send message
Joined: 15 Mar 01
Posts: 992
Credit: 207,654,737
RAC: 0
United States
Message 1181155 - Posted: 28 Dec 2011, 18:23:08 UTC

the project went down for a bit, someone must be lurking.


____________

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3534
Credit: 20,731,628
RAC: 23,084
Sweden
Message 1181158 - Posted: 28 Dec 2011, 18:41:26 UTC - in response to Message 1181155.

the project went down for a bit, someone must be lurking.



Main Page says:

Weekly Outage and Initial Catch Up
Because of the holiday schedule, we are conducting the weekly maintenance outage today (Wednesday, 12/28) 28 Dec 2011 | 15:41:41 UTC
____________

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1961
Credit: 10,479,289
RAC: 10,494
United States
Message 1181372 - Posted: 29 Dec 2011, 15:56:49 UTC

Umm, unless there's a good reason not to, could someone bring beta back up?
____________
.

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4141
Credit: 33,583,845
RAC: 26,700
United Kingdom
Message 1181422 - Posted: 29 Dec 2011, 18:02:21 UTC - in response to Message 1181372.

Umm, unless there's a good reason not to, could someone bring beta back up?

+1

Claggy

Profile PERPLEXER ~ Thomas HuettingerProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Jan 05
Posts: 6
Credit: 261,545,276
RAC: 36,043
Germany
Message 1181538 - Posted: 30 Dec 2011, 3:46:24 UTC

oh oh oh

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1961
Credit: 10,479,289
RAC: 10,494
United States
Message 1181615 - Posted: 30 Dec 2011, 17:22:25 UTC
Last modified: 30 Dec 2011, 17:23:09 UTC

still waiting on Beta — dot, dot, dot...
____________
.

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4141
Credit: 33,583,845
RAC: 26,700
United Kingdom
Message 1181619 - Posted: 30 Dec 2011, 17:49:40 UTC - in response to Message 1181615.
Last modified: 30 Dec 2011, 17:50:00 UTC

still waiting on Beta — dot, dot, dot...

I was hoping they'd left Beta down while they built v7 6.98 apps, But there's no sign of them yet, :-(

Claggy

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1961
Credit: 10,479,289
RAC: 10,494
United States
Message 1181620 - Posted: 30 Dec 2011, 18:00:12 UTC - in response to Message 1181619.


I was hoping they'd left Beta down while they built v7 6.98 apps, But there's no sign of them yet, :-(

Claggy


last I have are 6.97 apps - 14 of which I'm trying to report!
____________
.

Profile PERPLEXER ~ Thomas HuettingerProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Jan 05
Posts: 6
Credit: 261,545,276
RAC: 36,043
Germany
Message 1182041 - Posted: 1 Jan 2012, 1:26:14 UTC - in response to Message 1179613.

me fall down...but me not want... HAPPY NEW YEAR !!!

musicplayer
Send message
Joined: 17 May 10
Posts: 1457
Credit: 712,224
RAC: 960
Message 1182050 - Posted: 1 Jan 2012, 1:57:45 UTC
Last modified: 1 Jan 2012, 1:58:16 UTC

Happy New Year!

Time is now 02:54 here. I had a bottle of vine last evening, so I decided to stay away from it tonight.

Apparently an accident with the public fireworks downtown. Fortunately no-one was injured.

And apparently some Chinese lights in the sky as well. Never mind.

Profile Ed. Krushenski (Mr. K.)
Send message
Joined: 30 May 99
Posts: 1
Credit: 7,507,834
RAC: 6,231
United States
Message 1182577 - Posted: 3 Jan 2012, 11:14:59 UTC - in response to Message 1179613.

Hope you all had a happy holiday.

Here's to the hope for MANY candidates this year. Also hope for solving of the wierd issues you have been having.

Perhaps its voodoo, or Gremlins ????

Seriously, Happy New Year to all you folks.

Message boards : Technical News : Exodus (Dec 20 2011)

Copyright © 2014 University of California