Advance (Jun 25 2009)

Fallout continues from the outage on Tuesday. Turns out the minor corruption in various MyISAM tables is messing up replication. Every so often a duplicate entry appears on the replica queue which is easy to remove but requires human intervention. This is causing the replica to fall further and futher behind. I'm loathe to give up on it, though, as that means being forced to point all queries, including non-essential ones, at the master. And that'll break everything.

We also had to fall back to using two download servers, but we did so using simple DNS round-robin load balancing. Obviously this wasn't working out so well. DNS rollout/caching is never balanced (we saw this several times before, especially during the feeder mod polarity issues a year or two ago). So this morning we fell all the way back to using "pound" - which forces exactly 50% of all incoming connections to go to the first server, and the rest to the second one. This immediate broke the current download log jam, though of course we're still maxed out bandwidth-wise as I write this paragraph.

Seems like there are a lot of frustrated people on these threads. There's no right or wrong way to feel about these outages. We're kind of a special case. At the core we're an academic project with no deadlines - normally nobody gets hurt if science is delayed a day or a month or a decade. On the other hand, we're forced to be "professional" since we're asking for various forms of support from many thousands of people, and you can't have that large a number of people involved without some sort of professional grade management and public relations. It's a daily puzzle marrying the two completely separate worlds.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Seems like there are a lot of frustrated people on these threads. There's no right or wrong way to feel about these outages. We're kind of a special case. At the core we're an academic project with no deadlines - normally nobody gets hurt if science is delayed a day or a month or a decade. On the other hand, we're forced to be "professional" since we're asking for various forms of support from many thousands of people, and you can't have that large a number of people involved without some sort of professional grade management and public relations. It's a daily puzzle marrying the two completely separate worlds.

- Matt

hey, chill, take a breather... and relax a little.... i've leanred, being a tv station, to relax even when your to the point of shooting a piece of crap machine thats a p3 doing a dual-core job....
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

Thanks for the updates Matt.

I'm one of those that subscribes to the view this is science and there is no point getting upset if something breaks. Boinc will sort itself out.

I had a CPDN wu that uploaded all except one file. It took almost 2 weeks to get it uploaded due to issues on the server (apparently ran out of disk space). No point stressing.

Einstein's file server has crashed. That might be why Seti is getting more traffic than normal, apart from the outage recovery.

Cheers,
MarkJ
____________
BOINC blog

Thanks for letting us know about all the issues, it does help to prevent people getting annoyed when they know what is wrong. Computers break, simple as, no point in getting annoyed at all in my opinion. We have been with seti for long enough to know what the norm is, just set a larger cache than you would for other projects (I use 7 days) and then you are not effected by these small problems. My computers will be happily crunching into the middle of next week =)

Ignore those who complain, as long as you guys are doing your best, which we all know you are, then people will just have to accept these things happen.
____________

Seems like there are a lot of frustrated people on these threads. There's no right or wrong way to feel about these outages. We're kind of a special case. At the core we're an academic project with no deadlines - normally nobody gets hurt if science is delayed a day or a month or a decade. On the other hand, we're forced to be "professional" since we're asking for various forms of support from many thousands of people, and you can't have that large a number of people involved without some sort of professional grade management and public relations. It's a daily puzzle marrying the two completely separate worlds.

- Matt

These wise words should be placed at the top of the NC board permanently.. This is a serious proposal. It could save a large proportion of the unhappiness.

Shouldn't the weekly outage be changed to 'weeklong outage'? Tell us again how the project needs more people crunching, please. Oh, and be sure to ask for $$$.

I, too, fully agree with Matt's statement quoted below. Yet, I believe there is always something to learn from reasoned criticism. Hopefully, Matt and others bite the lemon, lick the salt, down the tequilla, and read the comments from time to time.

Looking at your past post history, I really don't think you're just trying to be cute, Elgar. In fact, it seems your comment was downright "trollish" because of everyone else offering their support for Matt which is the exact opposite of your agenda you've been on for quite some time now. And I see through browsing your past comments, you often like to "mask" your trollishness in the same way every other troll I've ever met does by trying to create an aura of legitimacy around your very forked-tongue questions.
____________