This has been a very difficult several days and we are still far from out of the woods.

This past Saturday morning the air conditioning in our server closet started acting up, apparently cycling on and off. Around noon that day, we deemed it bad enough to come to the lab. It's a good thing, because the AC was completely down when we got here. We shut most machines down and restarted the AC. It seemed to hold. But later that day our monitors showed the temperature increasing again, even with a small number of machines running. We came back to the lab and shut down everything except the web servers. That small load is OK even with no AC.

That's the way it has been, off and on, since. The physical plant people have been here several times. They have been doing a good job, even though low staffing levels have cut into the time that they can give us. The current diagnosis is that the AC has a bad condenser fan. Now it is a mater of getting the part - not trivial, unfortunately. In the meantime, they rigged up a piggyback fan, which did help some. Just not enough to run the project.

We're hoping that the new fan gets here soon.

---

Jeff, thanks for the news.

A pity..

..I hope also the fan will be sent and installed soon. ;-)

---

Hello Jeff, thanks for the update, glad no real damage has occurred!
Hat of for coming in the weekend to the lab, to see everything is OK :)

Hope that the UPLoad Servers will be online soon.
Regards
Fred.

---

Thanks for the update Jeff. Maybe you need an ambient temp to the server stats page! :)

---

Fingers crossed for you here.

Back when I ran VAX's in an airconditioned computer room (I know, I'm showing my age) we had temperature sensing attached to the UPS which could be set to signal the dependent servers to shut down if a temperature emergency arose (too high for too long).

I'd be surprised if there was not something available relatively cheaply nowadays to give a warning to signal the servers to shut down gracefully if a temperature emergency arises. Save you having to dash into work, and would be a safeguard for your data integrity too, removing the need for you guys to have to jump when the rude mechanics says "frog"

---

I'd be surprised if there was not something available relatively cheaply nowadays to give a warning to signal the servers to shut down gracefully if a temperature emergency arises. Save you having to dash into work, and would be a safeguard for your data integrity too, removing the need for you guys to have to jump when the rude mechanics says "frog"

How about free?

Most systems have temperature monitoring built in these days, and many Linux distributions have the tools either available or built in.

The problem of course is that it's an inexact science. Monitoring a CPU doesn't help you if a hard disk gets too hot (and they tend to be the first to 'die' now as modern CPUs usually have built-in thermal protection).

Then there's the lack of standards. There's a communication standard (I2C), but no standard as to what sensors should be implemented or what values represent a particular temperature. I got caught out by this - implemented thermal monitoring on a system which promptly shut itself down because the seemingly reasonable high temperature limit I'd set was misinterpreted as a value that was lower than the ambient temperature in the server room.

Anyway, glad to see the project survived the AC woes - that was too close for comfort. Thanks to the team for being on top of the problem even at the weekend.

---

Stats site - http://www.teamocuk.co.uk - still alive and (just about) kicking.

Thanks for the News Jeff.

---

Savoir-Faire is everywhere!
The T1 Trust, T1 Class 4-4-4-4 #5550, America's First HST

We're taking the project off line for the night, partly (mostly) for temperature concerns, but also to let the back end queues drain and give more I/O to the thumper root mirror re-sync that became necessary.

---

We're taking the project off line for the night, partly (mostly) for temperature concerns, but also to let the back end queues drain and give more I/O to the thumper root mirror re-sync that became necessary.

Thanks for the heads up.

---

Thanks for the updated information Jeff.

I do find it hard to believe that the condenser fan is hard to get.....
Most AC equipment uses fairly standardized motors, unless this unit is either very old or a one-off odd duck.
Grainger has about every motor used in the refrigeration field in their online catalog. HVAC motors...

---

"Time is simply the mechanism that keeps everything from happening all at once."

OK then, I think we may be up. The condenser fan motor was replaced today and we brought the projects up. It's good that we can test under load while we're still at the lab.

Assuming that there are no problems, we'll just call this the start of the server run for this week.

---

Thanks for the update. Hope the bailing wire and duct tape holds! :)

---

My queue's were empty and I have about 3500 finished WU's to upload...
I hope I don't crash the servers :P

---

Rig name: "x6Crunchy"
OS: Win 7 x64
MB: Asus M4N98TD EVO
CPU: AMD X6 1055T 2.8(1,2v)
GPU: 2x Asus GTX560ti
Member of: Dutch Power Cows

Jeff, thanks for the news!

It's nice to see everything again in green at the 'server status page'.. ;-)

---

Any news on when the Beta project will come online? I can't get any of it to go.

---

Any news on when the Beta project will come online? I can't get any of it to go.

I can't get any Seti Beta uploads to go though, i can connect to the scheduler, but that doesn't help when uploads won't go through.

Claggy

Any news on when the Beta project will come online? I can't get any of it to go.

I can't get any Seti Beta uploads to go though, i can connect to the scheduler, but that doesn't help when uploads won't go through.

Claggy

I managed to pull down one WU. I have one still stuck in upload mode.

---

How are the currently 'WU limit in progress'?

It's look like much less than here mentioned: 'server run, September 3-6 2010'.

OTOH, nearly 'no new WUs' available for DL.

---

How are the currently 'WU limit in progress'?

It's look like much less than here mentioned: 'server run, September 3-6 2010'.

OTOH, nearly 'no new WUs' available for DL.

I think, OTOH, you answered your own question.

---

Donald
Infernal Optimist / Submariner, retired

How are the currently 'WU limit in progress'?

It's look like much less than here mentioned: 'server run, September 3-6 2010'.

OTOH, nearly 'no new WUs' available for DL.

I think, OTOH, you answered your own question.

No.. this are 'two different pair of shoes'..


Message from server: No work sent
Message from server: This computer has reached a limit on tasks in progress


Scheduler request completed: got 0 new tasks
Message from server: Project has no tasks available

---