The End of All Things (Oct 30 2008)

Okay. So the assimilator memory leak wasn't a problem so much as an effect. It's consumption of resources still needs to be addressed, but it was only affecting itself, and being aggravated by the other problems around it.

Poring through logs I confirmed that the network bursts were indeed due to Astropulse downloads - during the "baseline" 2 out of 100 workunit downloads are Astropulse, but during the "burst" 40 out of 100 are Astropulse. The Astropulse workunits are much larger in size than SETI@home workunits, hence the bandwidth consumption. I also confirmed it wasn't a single (or few) clients hitting us at once - connections were randomly distributed over many IP addresses.

It finally dawned on me, and now like most things is painfully obvious on hindsight. The SETI@home and Astropulse splitters have separate high water marks. For SETI@home, if we get above 50000 results ready to send, we temporarily halt splitting. For Astropulse, it is still set pretty low at 2500. Every so often a splitter process checks to see the size of the queue and if it should stop. Since there are many SETI@home splitters running at a time, and there is always a delay in transitioning state, thousands of workunits may be generated before the splitters actually realize they are above the high water mark. And then they go to sleep for a while - like an hour or so - until the queue drains enough and they wake up again and get back to work.

The thing is, during SETI@home's "sleep until we're needed again" phase the Astropulse splitters continue to run since they haven't reached their high water mark even though it's much lower - those splitters are fewer in number and run slower. Now remember when workunits are created, the transitioners also create respecitve results to "send." New results are id'ed serially - i.e. they are tagged with a number in the database which increases automatically. So during these periods you'll get an area in database id space rich in Astropulse results.

Moving on to the feeder. Since it's stupid regarding application types, it fills its own send queue with the oldest results ready to send regardless of application, and the way mysql works this tends to mean in database id order. Of course with the ready-to-send queue at 50000 or so, we have to send out 50000 results before we finally see the effects of what happened above - many hours, usually. Then suddenly - bam! - 20 times more Astropulse workunits than normal. That arbitrary time delay really confused matters.

Anyway, one easy solution is to make the feeder smarter. It does have an "-allapps" flag to send to all applications equally. We were hesitant to use this before due to fear this will give too many shared memory slots to Astropulse - and it may very well cause periods of low work during peak periods as the feeder has half the memory for SETI@home workunits than it did. Nevertheless we turned this on today and it had an immediate, positive smoothing effect. Sweet.

Other than that today... some data pipeline scripting, and continuing discussions amongst the gang regarding changing redundancy to zero - trying to wrap our brains around all the current bottlenecks and what will suffer depending on what we do. As it stands now, our servers most likely will not be able to support reducing redundancy all the way to zero *and* keeping up with current workunit demand. So we have to either improve our server i/o or figure out what other knobs to turn.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update Jeff.

Thanks for the update Jeff.

The problems last night/this morning were random apache failure on the scheduler, which happens for no good reason from time to time. Jeff restarted it when he got in this morning.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Is this still an issue ? I keep getting http errors.

Is this still an issue ? I keep getting http errors.

Sure is, and I´m running out of work.........
But at least I can do a bit for Milkyway !

The server is up (shows the apche-test-page at least).
Might be just the 90 mbit/s wall we're knocking on for couple of hrs now.
____________
mic.

The server is up (shows the apche-test-page at least).
Might be just the 90 mbit/s wall we're knocking on for couple of hrs now.

I'm getting an occasional upload complete, but absolutely zero downloads. I mean nothing. When the backlog is this bad they usually start dl'ing and time out part way through, but now it's not a single byte making it through.

I'm getting an occasional upload complete, but absolutely zero downloads. I mean nothing. When the backlog is this bad they usually start dl'ing and time out part way through, but now it's not a single byte making it through.

I referred to the upload server mentioned in the post before mine.

This one is still up and I get all my uploads through after some trys (due to network load I think).
On the download side at least bane is up.
Vader is not responding...again...

____________
mic.

I'm getting an occasional upload complete, but absolutely zero downloads. I mean nothing. When the backlog is this bad they usually start dl'ing and time out part way through, but now it's not a single byte making it through.

Looka like a long weekend........LOL......crunch 'em if you got 'em.........