Fill 'er up (Jun 23 2010)

Since last I wrote a lot has happened. Looking at the traffic graphs it's like feast or famine - either we are unable to create/send out workunits, or we're sending out as many as we can fit through the pipe. Mostly it's been the usual gremlins.

However regarding the past 24 hours it was a new problem: the result space on the upload server filled up unexpectedly, which would have been fine except this (perhaps) inspired some RAID freakout on the system. We couldn't really sort it out until this morning. From the looks of it we had something like a six drive simultaneous failure. Jeff and I beat on it for a while - we eventually assumed this was just a hardware blip, and the data was more or less intact on the drives, but the RAID metadata got a little screwed up. Long story short we were able to carefully bring down the RAID and recreate the meta devices from scratch with the data intact, and all was well. Phew. For the record we do have a virtually-up-to-date result storage backup at all times in case of catastrophic failure on this system.

In any case, the main culprit was our disks filling up, so as I write this we're keeping the project down until major queues drain and the constituent workunit/result files can be deleted.

On a more happy (perhaps) note, yesterday the core group of us were in the same place at the same time (which is rare) and we had an ad hoc meeting about our current project status/plans, especially in light of many recent server problems, increasingly random schedules, and embarrassingly low funding. We're all kind of tired and beaten up and wanting some results already - so I like to think this paved the way for several large and ultimately positive changes in the future.

Also Jeff has been working on this nagging mysterious problem where some of our raw data files are only getting partially processed (which vastly increases our "burn rate" and leads to unexpected workunit shortages). He found some major clues today, and we brainstormed why this is happening and what the exact effect is. At least there's a smoking gun on that front.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update Matt,

Claggy

Thanks for the excellent news item. Most of us are a patient lot and understand you are reduced to rock soup. Don't let the others get you down.
____________

I know you guys have your hands full-sometimes those sudden meetings are the best to have occurred.
____________

Thank you Matt.
David A, Eric, Dan, Jeff, Josh Von Korff, Robert Bankay, Kevin Douglas.
You folks show outstanding dedication to SETI@home.
Best wishes and peace.
from Vancouver, Canada.
Byron.


____________

and we had an ad hoc meeting about our current project status/plans, especially in light of many recent server problems, increasingly random schedules, and embarrassingly low funding. We're all kind of tired and beaten up and wanting some results already - so I like to think this paved the way for several large and ultimately positive changes in the future.

He found some major clues today, and we brainstormed why this is happening and what the exact effect is. At least there's a smoking gun on that front.

Matt, thank you very much for the update, it really shed's a lot of light on the recent problems. Here's hoping you guys get it running smooth again, or as smooth as you can for a bit. LOL Have a great rest of the week!

Thanks Matt! I'm interested in donating to the cause for new hardware.
Please let us know.
Todd
____________

Hi Matt,

Thanks for the update and the great job you guys do to keep things running.

Since Blurf decided to reduce his involvement we don't seem to have anyone to coordinate hardware (or other) funding drives. Is is possible to get the list of required equipment updated so we have an idea of what is currently needed please.

Cheers
____________
BOINC blog

Hi Guys,

On behalf of the SETI@Netherlands team I would like to thank you for all your hard work you are putting in.
No budget, no manpower and no luck. It's a bitch.
But please, don't get discouraged and keep up the good work.

When you've got the flow running again, we will again do our part in the search :)
____________
Treasurer of team SETI@Netherlands

Hey folks. We need to do something about this funding issue. SETI has more crunchers than just about any other project. This is a huge resource, and it feels like the operation at Berkeley is slowly deteriorating. The guys there can only do so much to keep things running and eventually, critical hardware will fail that they cannot afford to replace.

Given the broad spectrum of people who contribute to this project, there have to be some profs who can help pursue grants to reinvigorate the project. Any ideas?

Shouldn't the thread title be "Fill 'er up (Jun 23 2010)" ?

It ain't July yet.


____________
/The grumpy old Swede.

"I'm so old, that 98% of all trees in the forest, are younger than I am"

Roll 'im back.