Working as Expected (Jul 13 2009)

The data pipeline over the weekend seemed to be more or less okay, thanks to running out of Astropulse workunits and not having any raw data to split to create new ones. Of course, I shovelled some more raw data to the pile this morning, and our bandwidth shot right back up again. This pretty much proves that our recent headaches have been largely due to the disparity of workunit sizes/compute times between multibeam/Astropulse, but that's all academic at this point as Eric is close to implementing a configuration change which will increase the resolution of chirp rates (thus increasing analysis/sensitivity) and also slowing clients down so they don't contact our servers as often. We should be back to lower levels of traffic soon enough.

We are running fairly low on data from our archives, which is a bit scary. We're burning through it rather quickly. Luckily, Andrew is down at Arecibo now, with one of our new drive bays - he'll plug it in perhaps today and we'll hopefully be collecting data later tonight...?

To be clear, we actually have hundreds of raw data files in our archives, but most of them suffer from (a) lack of embedded hardware radar signals (therefore making it currently impossible to analyse without being blitzed by RFI), or (b) accidental extra coordinate precession, or (c) both of the above. Software is in the works (mostly waiting on me) to solve all the above.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update Matt,

Claggy

I hope that the deadlines also increased with the increased work.
____________


BOINC WIKI

I hope that the deadlines also increased with the increased work.

Send me the largest WU's you got, Matt! :D

Send me the largest WU's you got, Matt! :D

I have a lot of results ready for UL, but the UL server is offline?

Longer crunching time for MB.. I hope the credits will increase also.. ;-)
The applications don't need to be adjusted?

But.. BTW.. if AP overload the internet traffic to the server.. why not 'disable' AP to the time faster traffic is possible ('bigger cable')?

____________

...
This pretty much proves that our recent headaches have been largely due to the disparity of workunit sizes/compute times between multibeam/Astropulse, but that's all academic at this point as Eric is close to implementing a configuration change which will increase the resolution of chirp rates (thus increasing analysis/sensitivity) and also slowing clients down so they don't contact our servers as often. We should be back to lower levels of traffic soon enough.
...

Indeed so... Working exactly as expected.

For the link limits and congestion... Note:

In a email that was sent to Seti Staff. At a point in time the 100Megabit link was Full Duplex. Meaning Uploads should not interfere with Downloads and vice versa (each is in its own channel).

We forget that TCP is a sliding window protocol. If the 100 megabit line is saturated inbound, part of that inbound traffic are the ACKs for the outbound traffic.

When the ACKs are delayed or lost, at some point the sender stops sending new data, and waits. When the ACKs don't arrive (because they were lost) data is resent.

In either direction, when the load is very high, data in the other direction will suffer too.

That's a very 'subdued' way of describing the situation.

Lose the TCP control packets in either direction and the link is DOSed with an exponentially increasing stack of resend attempts that DOS for further attempts that then DOS for... Until the link disgracefully degrades to being totally blocked. Max link utilisation but no useful information gets through.

The only limiting factors are the TCP timeouts and the rate of new connection attempts.


And I thought the smooth 71Mb/s was due to some cool traffic management. OK, so restricting the available WUs is also a clumsy way to "traffic manage"!


In short, keep the link at never anything more than 89Mb/s MAX and everyone is happy!

Happy smooth crunchin',
Martin

BTW.

~ 400 results in the BOINC UL overview and every few minutes increasing..

____________

Matt,

Please turn uploads back on during maintenance, while we have nice clean bandwidth to use.

Reading Matts Tech News the other day and the Cricket Graphs when the Feeder was putting out Just MultiBeam... You could see that Uploads (Uploads and Scheduler Requests) were getting through and eating about 17.5% of the bandwidth. Downloads due to the feeder settings were keeping up and eating ~63%
In that cirucumstance the balance was very visible.

Now as Uploads have been turned Off, I presume that Matt is tunning the "Feeder" to see what settings might be better. In order to accurately set this Uploads would interfere with the calibrations

Regards






____________
Please consider a Donation to the Seti Project.

Reading Matts Tech News the other day and the Cricket Graphs when the Feeder was putting out Just MultiBeam... You could see that Uploads (Uploads and Scheduler Requests) were getting through and eating about 17.5% of the bandwidth. Downloads due to the feeder settings were keeping up and eating ~63%
In that cirucumstance the balance was very visible.

Now as Uploads have been turned Off, I presume that Matt is tunning the "Feeder" to see what settings might be better. In order to accurately set this Uploads would interfere with the calibrations

Regards