Happy Lupercalia! (Feb 14 2011)

Slow, steady progress... We're hoping to have everything copied from gowron onto thumper by tomorrow. Yeah, I know it's going slowly, but there's lots of bottlenecks (degraded RAID, NFS, tons of small files as opposed to a few big ones). After the usual outage we might actually have thumper ready to be the temporary workunit storage server so we can get back to business while doing the necessary upgrades on gowron (which make take as much as a week, unobtrusively running in the background).

That new-ish server synergy rebooted itself on Sunday. This concerned me as this has happened a couple times already. However, I discovered the three reboots thus far all happened on Sunday at 3pm, and two weeks apart from each other. There are no smoking-gun cronjobs, but it is plugged into an old UPS of unknown quality, so we're going to remove that from the equation and watch what happens. The reboots have all been harmless thus far.

Somebody somewhere on these forums asked what our server makeup was. It certainly isn't limited to what's on the server status page. If you just count the unix-based machines, there are currently 26 systems all told. Combining all the stuff inside, we have roughly 100 CPUs, 500GB RAM, and 150 TB raw storage. There are also several appliances (routers, switches, UPSes, kvms, remote controlled power strips, etc. etc.). Usually in these threads I'm griping about public facing servers, or ones causing the BOINC back end to jam up for one reason or another. I rarely mention the mundane, day-to-day, garden variety IT stuff.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update there Matt.

Actually it was me that asked about all the non-public facing servers that Seti had :-) Sometime in the future when you are all not so busy, it would be nice to see a list of them and what they do. It will give people a real feel for the scope of the admin tasks that you all are responsible for.



____________
Damsel Rescuer, Kitty Patron, Raccoon Friend, Uli Fan,
Julie Supporter, ES99 Admirer, PETA Member, 1st Childhood

It would be nice to see a list of them and what they do.

If you just count the unix-based machines, there are currently 26 systems all told. Combining all the stuff inside, we have roughly 100 CPUs, 500GB RAM, and 150 TB raw storage. There are also several appliances (routers, switches, UPSes, kvms, remote controlled power strips, etc. etc.).

Matt, thanks for the news!

____________


>Das Deutsche Cafe. The German Cafe.<

Maybe you should implement the slow ramp-up that was the rule back during the three-day outage era...

Oh, and don't forget to bring up beta sometime soon, once production gets running nicely...
____________
.

Just an FYI. I was up in Oakland during the last big one during the World Series. Anyway, all our equipment went down even though we had UPS's on everything. What caused it? Seems that the batteries in the UPS's are not eternal and have to be replaced like every 3 - 5 years! Could there be any relation to what is causing your issues?

Just an FYI. I was up in Oakland during the last big one during the World Series. Anyway, all our equipment went down even though we had UPS's on everything. What caused it? Seems that the batteries in the UPS's are not eternal and have to be replaced like every 3 - 5 years! Could there be any relation to what is causing your issues?

Are you talking the Earthquake series in '89 or last year?

The routinely spaced hits at the same wall-clock time two weeks apart that Matt is talking about don't seem to be battery-related to me... (assuming, of course, that Matt has checked the possibility that someone [who comes in alternate weeks...] is routinely turning off the circuit that the UPS in question is plugged into...)

The routinely spaced hits at the same wall-clock time two weeks apart that Matt is talking about don't seem to be battery-related to me... (assuming, of course, that Matt has checked the possibility that someone [who comes in alternate weeks...] is routinely turning off the circuit that the UPS in question is plugged into...)

The cleaning crew unplugging something to crank up their vacuum cleaners?