Message boards :
Technical News :
server / AC status
Message board moderation
Author | Message |
---|---|
Jeff Cobb Send message Joined: 1 Mar 99 Posts: 122 Credit: 40,367 RAC: 0 |
This has been a very difficult several days and we are still far from out of the woods. This past Saturday morning the air conditioning in our server closet started acting up, apparently cycling on and off. Around noon that day, we deemed it bad enough to come to the lab. It's a good thing, because the AC was completely down when we got here. We shut most machines down and restarted the AC. It seemed to hold. But later that day our monitors showed the temperature increasing again, even with a small number of machines running. We came back to the lab and shut down everything except the web servers. That small load is OK even with no AC. That's the way it has been, off and on, since. The physical plant people have been here several times. They have been doing a good job, even though low staffing levels have cut into the time that they can give us. The current diagnosis is that the AC has a bad condenser fan. Now it is a mater of getting the part - not trivial, unfortunately. In the meantime, they rigged up a piggyback fan, which did help some. Just not enough to run the project. We're hoping that the new fan gets here soon. |
Dirk Sadowski Send message Joined: 6 Apr 07 Posts: 7105 Credit: 147,663,825 RAC: 5 |
Jeff, thanks for the news. A pity.. ..I hope also the fan will be sent and installed soon. ;-) |
Fred J. Verster Send message Joined: 21 Apr 04 Posts: 3252 Credit: 31,903,643 RAC: 0 |
|
Gary Charpentier Send message Joined: 25 Dec 00 Posts: 31012 Credit: 53,134,872 RAC: 32 |
Thanks for the update Jeff. Maybe you need an ambient temp to the server stats page! :) |
alan Send message Joined: 18 Feb 00 Posts: 131 Credit: 401,606 RAC: 0 |
Fingers crossed for you here. Back when I ran VAX's in an airconditioned computer room (I know, I'm showing my age) we had temperature sensing attached to the UPS which could be set to signal the dependent servers to shut down if a temperature emergency arose (too high for too long). I'd be surprised if there was not something available relatively cheaply nowadays to give a warning to signal the servers to shut down gracefully if a temperature emergency arises. Save you having to dash into work, and would be a safeguard for your data integrity too, removing the need for you guys to have to jump when the rude mechanics says "frog" |
Berserker Send message Joined: 2 Jun 99 Posts: 105 Credit: 5,440,087 RAC: 0 |
I'd be surprised if there was not something available relatively cheaply nowadays to give a warning to signal the servers to shut down gracefully if a temperature emergency arises. Save you having to dash into work, and would be a safeguard for your data integrity too, removing the need for you guys to have to jump when the rude mechanics says "frog" How about free? Most systems have temperature monitoring built in these days, and many Linux distributions have the tools either available or built in. The problem of course is that it's an inexact science. Monitoring a CPU doesn't help you if a hard disk gets too hot (and they tend to be the first to 'die' now as modern CPUs usually have built-in thermal protection). Then there's the lack of standards. There's a communication standard (I2C), but no standard as to what sensors should be implemented or what values represent a particular temperature. I got caught out by this - implemented thermal monitoring on a system which promptly shut itself down because the seemingly reasonable high temperature limit I'd set was misinterpreted as a value that was lower than the ambient temperature in the server room. Anyway, glad to see the project survived the AC woes - that was too close for comfort. Thanks to the team for being on top of the problem even at the weekend. Stats site - http://www.teamocuk.co.uk - still alive and (just about) kicking. |
zoom3+1=4 Send message Joined: 30 Nov 03 Posts: 66349 Credit: 55,293,173 RAC: 49 |
Thanks for the News Jeff. Savoir-Faire is everywhere! The T1 Trust, T1 Class 4-4-4-4 #5550, America's First HST |
Jeff Cobb Send message Joined: 1 Mar 99 Posts: 122 Credit: 40,367 RAC: 0 |
We're taking the project off line for the night, partly (mostly) for temperature concerns, but also to let the back end queues drain and give more I/O to the thumper root mirror re-sync that became necessary. |
Gary Charpentier Send message Joined: 25 Dec 00 Posts: 31012 Credit: 53,134,872 RAC: 32 |
We're taking the project off line for the night, partly (mostly) for temperature concerns, but also to let the back end queues drain and give more I/O to the thumper root mirror re-sync that became necessary. Thanks for the heads up. |
kittyman Send message Joined: 9 Jul 00 Posts: 51478 Credit: 1,018,363,574 RAC: 1,004 |
Thanks for the updated information Jeff. I do find it hard to believe that the condenser fan is hard to get..... Most AC equipment uses fairly standardized motors, unless this unit is either very old or a one-off odd duck. Grainger has about every motor used in the refrigeration field in their online catalog. HVAC motors... "Time is simply the mechanism that keeps everything from happening all at once." |
Jeff Cobb Send message Joined: 1 Mar 99 Posts: 122 Credit: 40,367 RAC: 0 |
OK then, I think we may be up. The condenser fan motor was replaced today and we brought the projects up. It's good that we can test under load while we're still at the lab. Assuming that there are no problems, we'll just call this the start of the server run for this week. |
Gary Charpentier Send message Joined: 25 Dec 00 Posts: 31012 Credit: 53,134,872 RAC: 32 |
Thanks for the update. Hope the bailing wire and duct tape holds! :) |
TheFreshPrince a.k.a. BlueTooth76 Send message Joined: 4 Jun 99 Posts: 210 Credit: 10,315,944 RAC: 0 |
My queue's were empty and I have about 3500 finished WU's to upload... I hope I don't crash the servers :P Rig name: "x6Crunchy" OS: Win 7 x64 MB: Asus M4N98TD EVO CPU: AMD X6 1055T 2.8(1,2v) GPU: 2x Asus GTX560ti Member of: Dutch Power Cows |
Dirk Sadowski Send message Joined: 6 Apr 07 Posts: 7105 Credit: 147,663,825 RAC: 5 |
Jeff, thanks for the news! It's nice to see everything again in green at the 'server status page'.. ;-) |
B-Man Send message Joined: 11 Feb 01 Posts: 253 Credit: 147,366 RAC: 0 |
Any news on when the Beta project will come online? I can't get any of it to go. |
Claggy Send message Joined: 5 Jul 99 Posts: 4654 Credit: 47,537,079 RAC: 4 |
Any news on when the Beta project will come online? I can't get any of it to go. I can't get any Seti Beta uploads to go though, i can connect to the scheduler, but that doesn't help when uploads won't go through. Claggy |
B-Man Send message Joined: 11 Feb 01 Posts: 253 Credit: 147,366 RAC: 0 |
Any news on when the Beta project will come online? I can't get any of it to go. I managed to pull down one WU. I have one still stuck in upload mode. |
Dirk Sadowski Send message Joined: 6 Apr 07 Posts: 7105 Credit: 147,663,825 RAC: 5 |
How are the currently 'WU limit in progress'? It's look like much less than here mentioned: 'server run, September 3-6 2010'. OTOH, nearly 'no new WUs' available for DL. |
Donald L. Johnson Send message Joined: 5 Aug 02 Posts: 8240 Credit: 14,654,533 RAC: 20 |
How are the currently 'WU limit in progress'? I think, OTOH, you answered your own question. Donald Infernal Optimist / Submariner, retired |
Dirk Sadowski Send message Joined: 6 Apr 07 Posts: 7105 Credit: 147,663,825 RAC: 5 |
How are the currently 'WU limit in progress'? No.. this are 'two different pair of shoes'.. Message from server: No work sent Message from server: This computer has reached a limit on tasks in progress Scheduler request completed: got 0 new tasks Message from server: Project has no tasks available |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.