I wasn't selected for jury duty! Hooray! I fulfilled my civic duty without having to miss work!

So we're in the middle of a slight server malaise - the data we're currently splitting/sending out is of the sort that it gets processed quickly and returned much faster than average. That's one big difference with our current multi-beam processing: the variance of data processing time per workunit is far greater than before, so we get into these unpredictable heavy periods and have no choice but to wait them out.

Well... that's not entirely true. Jeff actually moved the rest of the raw data from this day out of the way so we can move to other days which are potentially more friendly towards our servers. Also we could predict, with very coarse resolution, what days might be "rough" before sending them through the pipeline. But we're going to split the data anyway at some point, so why not get it over with? At any rate we started more splitters to keep from running out of work, and we'll keep an eye on this as we progress into the holiday weekend.

Happy Thanksgiving! Or if you're not in the U.S. - Happy Thursday!

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

But we're going to split the data anyway at some point, so why not get it over with?

Please understand the spirit in which I'm going to say what I'm about to say:

Was it wise to attempt such a feat after back-to-back daily outages and going into a long weekend when you probably would rather be away and not have to worry about this?

Was it wise to attempt such a feat after back-to-back daily outages and going into a long weekend when you probably would rather be away and not have to worry about this?

Understood. The point is that without significant effort up front there's no way to tell the quality of the data before it enters the splitter queue. So once it enters the queue and we find it to be "ugly," we might as well just let it finish. In today's case, the file was already half split by the time we determined the issue and its cause, so we're just gonna let it wrap up naturally, which it looks like it will in a few hours.

In any case, there's no worry: SETI@home is such a dynamic system with so many variables mixed with the regularly flexible schedules of our staff (not just during a holiday weekend). So anything can happen at any time with only a small subset of people on hand - and many times that subset contains zero elements. After many years of this you become an expert on unexpected minor/major disaster recovery and simply stop worrying.

And the current state is by no means a disaster. A temporary uncomfortable period that is showing all signs of passing.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the updates Matt - Have an Excellent Holiday too . . .

> that goes for the Rest of You @ Berkeley - Enjoy Your Holidays also . . .

---

BOINC Wiki . . .

Science Status Page . . .

@ Matt - Eclecticity - and Diversity - 1998 through 2004 . . . too bad these bands don't do something again eh . . . i like what they did though

U TOTEM in concert at New Music Montreal - November 5, 1990 "One Nail Draws Another" - Part 1

U TOTEM in concert at New Music Montreal - November 5, 1990 "One Nail Draws Another" - Part 2



1. One Nail Draws Another
2. Two Looks At One End
3. Dance Of The Awkward
4. Both Your Houses
5. Yellow Umbrella Gallery
6. The Judas Goat
7. Vagabonds Home

. . . i sure miss Montreal these days ;)

---

BOINC Wiki . . .

Science Status Page . . .

Thanks for the info Matt.

Also. @ Dr C.E.T.I., thanks for the music links.

Regards

PJ

---

You are Most Welcome PJ . . . i would wish You a Happy Thanksgiving - but you already had yours on October 8th . . . so enjoy your Weekend . . .

---

BOINC Wiki . . .

Science Status Page . . .

...without significant effort up front there's no way to tell the quality of the data before it enters the splitter queue. So once it enters the queue and we find it to be "ugly," we might as well just let it finish...
- Matt

What we're dealing with here is the slew rate of the telescope rather than a data quality issue. I'd think that while dividing the data into 50 GB chunks it might be fairly easy to check start and end times, then query the Arecibo database for telescope position periodically between those times and derive the slew rates.

If the slew rate is consistently above 0.0105 degrees per second the generated WUs will be the quick ones which the system cannot yet handle gracefully. The splitter estimates they will take about 1/6 the time of WUs produced from typical lower slew rates, so the Scheduler assigns 6 times as many WUs per host.

Because we look at those WUs for a shorter time, I suppose "ugly" is a fair description in that sense. In any case, if the GALFACTS project gets into full swing after Arecibo is recommissioned there will be a lot of high slew rate work to deal with, and some planning for that seems in order.

                                                                Joe

...without significant effort up front there's no way to tell the quality of the data before it enters the splitter queue. So once it enters the queue and we find it to be "ugly," we might as well just let it finish...
- Matt

What we're dealing with here is the slew rate of the telescope rather than a data quality issue. I'd think that while dividing the data into 50 GB chunks it might be fairly easy to check start and end times, then query the Arecibo database for telescope position periodically between those times and derive the slew rates.

If the slew rate is consistently above 0.0105 degrees per second the generated WUs will be the quick ones which the system cannot yet handle gracefully. The splitter estimates they will take about 1/6 the time of WUs produced from typical lower slew rates, so the Scheduler assigns 6 times as many WUs per host.

Because we look at those WUs for a shorter time, I suppose "ugly" is a fair description in that sense. In any case, if the GALFACTS project gets into full swing after Arecibo is recommissioned there will be a lot of high slew rate work to deal with, and some planning for that seems in order.

                                                                Joe

And the slew rate of the telescope should be known, or at least in principle knowable, from the 'tape' name (== recording date) and the telescope observing schedule.

At the moment, the splitter algorithm seems to call for all active splitters - typically six - to work on the same 'tape' at the same time: so when the high slew rate recordings are encountered, they affect all the WUs split around that time.

Would it be feasible to arrange for each splitter to work on its own 'tape', so that the six splitter instances are working on data recorded on different days, or at least different times of the same day? Then there might be more of a mix of WU durations in the tasks available for download, and we wouldn't get the 6::1 ratio - just average things out a bit.

I assume it is possible to find out how much downloads the servers and the bandwith can handle without becoming overloaded.

If the number of active splitters is limited to producing this maximum number of results and the ready to send queue is set to a much lower maximum (less then 10.000) then it is possible to avoid the download overload completely.

When "normal" runtime results are send out there is enough work for everybody because the splitters can make more results then are needed.
Only when "short" runtime results are produced there will be moments that there is not enough work because it is send out as fast as the splitters can create it.

I think however that it is more acceptable to recieve the "No work" message then to have results sitting on the transfer tab doing download retrys because the servers are overloaded.

---

Also we could predict, with very coarse resolution, what days might be "rough" before sending them through the pipeline.

Now, if you could find a way to "tell" the splitters (or even just the feeders) which batches will be processed fast or slow, then theoretically, you would send out a "mix" of both fast and slow processing tasks. I suppose the feeders would have to be much smarter to do this properly (i.e., give the scheduler varied task sizes). But how brilliant of an advancement to BOINC would that be?

At the moment, the splitter algorithm seems to call for all active splitters - typically six - to work on the same 'tape' at the same time: so when the high slew rate recordings are encountered, they affect all the WUs split around that time.

Would it be feasible to arrange for each splitter to work on its own 'tape', so that the six splitter instances are working on data recorded on different days, or at least different times of the same day? Then there might be more of a mix of WU durations in the tasks available for download, and we wouldn't get the 6::1 ratio - just average things out a bit.

This is probably the smartest and simplest solution. Except that one splitter working on one "file" would take forever + 1 day to complete it. Even with 4 to 6 splitters working on the same "file" it takes a long time to finish it.

At the moment, the splitter algorithm seems to call for all active splitters - typically six - to work on the same 'tape' at the same time: so when the high slew rate recordings are encountered, they affect all the WUs split around that time.

Would it be feasible to arrange for each splitter to work on its own 'tape', so that the six splitter instances are working on data recorded on different days, or at least different times of the same day? Then there might be more of a mix of WU durations in the tasks available for download, and we wouldn't get the 6::1 ratio - just average things out a bit.

This is probably the smartest and simplest solution. Except that one splitter working on one "file" would take forever + 1 day to complete it. Even with 4 to 6 splitters working on the same "file" it takes a long time to finish it.

That would be a much simplier solution. However, you'll need 6x the disk space for pre-split data, as those files will spend that much longer in the cycle.

0.0105 degree per second times 107 seconds is about 1.12 degrees. So any angle range above about 1.12 degrees doesn't require as much computation, probably because it's more difficult to study a possible radio source that's zipping through rather than one that's moving more slowly through the telescope field-of-view, right?

---

...without significant effort up front there's no way to tell the quality of the data before it enters the splitter queue. So once it enters the queue and we find it to be "ugly," we might as well just let it finish...
- Matt

What we're dealing with here is the slew rate of the telescope rather than a data quality issue. I'd think that while dividing the data into 50 GB chunks it might be fairly easy to check start and end times, then query the Arecibo database for telescope position periodically between those times and derive the slew rates.

If the slew rate is consistently above 0.0105 degrees per second the generated WUs will be the quick ones which the system cannot yet handle gracefully. The splitter estimates they will take about 1/6 the time of WUs produced from typical lower slew rates, so the Scheduler assigns 6 times as many WUs per host.

Because we look at those WUs for a shorter time, I suppose "ugly" is a fair description in that sense. In any case, if the GALFACTS project gets into full swing after Arecibo is recommissioned there will be a lot of high slew rate work to deal with, and some planning for that seems in order.

                                                                Joe

And the slew rate of the telescope should be known, or at least in principle knowable, from the 'tape' name (== recording date) and the telescope observing schedule.

At the moment, the splitter algorithm seems to call for all active splitters - typically six - to work on the same 'tape' at the same time: so when the high slew rate recordings are encountered, they affect all the WUs split around that time.

Would it be feasible to arrange for each splitter to work on its own 'tape', so that the six splitter instances are working on data recorded on different days, or at least different times of the same day? Then there might be more of a mix of WU durations in the tasks available for download, and we wouldn't get the 6::1 ratio - just average things out a bit.

Interesting new pattern on the Server Status page - looks as if this idea from way back might be getting a trial run. It'll be interesting to see how it goes.

In the meantime, I've found a way of automating recovery of the actual telescope recording time (in Arecibo's AST time-zone) from my logs of WUs past. So I can tell exactly when the telescope was observing each of the last 525 'shorty' WUs that passed through my E5320 box. Before I ruin my remaining eyesight poring through the Arecibo schedule PDFs we found before Christmas, does anyone have any idea if the telescope project allocation data could be downloaded anywhere in a structured, database-able format?

...
Before I ruin my remaining eyesight poring through the Arecibo schedule PDFs we found before Christmas, does anyone have any idea if the telescope project allocation data could be downloaded anywhere in a structured, database-able format?

The files in http://www.naic.edu/vscience/schedule/Scheds/ look like they'd be fairly easy to parse once the relationships to the schedule GIFs is figured out. They even give times in textual form so you don't have to estimate from the side scales.

                                                                Joe

...
Before I ruin my remaining eyesight poring through the Arecibo schedule PDFs we found before Christmas, does anyone have any idea if the telescope project allocation data could be downloaded anywhere in a structured, database-able format?

The files in http://www.naic.edu/vscience/schedule/Scheds/ look like they'd be fairly easy to parse once the relationships to the schedule GIFs is figured out. They even give times in textual form so you don't have to estimate from the side scales.

                                                                Joe

Excellent - exactly what I was looking for - thanks, Joe.

Now I just have to quantize my data in 15-minute chunks (I guessed at 10 minutes the first time round, because it's easier to do the string manipulation!), and I'll try and post something. It'll be tomorrow, though.

...
Before I ruin my remaining eyesight poring through the Arecibo schedule PDFs we found before Christmas, does anyone have any idea if the telescope project allocation data could be downloaded anywhere in a structured, database-able format?

The files in http://www.naic.edu/vscience/schedule/Scheds/ look like they'd be fairly easy to parse once the relationships to the schedule GIFs is figured out. They even give times in textual form so you don't have to estimate from the side scales.

                                                                Joe

Excellent - exactly what I was looking for - thanks, Joe.

Now I just have to quantize my data in 15-minute chunks (I guessed at 10 minutes the first time round, because it's easier to do the string manipulation!), and I'll try and post something. It'll be tomorrow, though.

Bit of a late 'tomorrow', but the first (raw, unchecked) data are coming out of the cruncher.

Candidates for the VVHAR storm over the holidays (AR>2, issued in late December):

Arecibo Project	CountOfWU
A2172 .............. 81
A2174 .............. 43
A2222 .............. 12

VHAR work in general, last three months:

Arecibo Project	CountOfWU
A2060	.............	189
A2172	..............	95
A2222	..............	75
P2030	..............	43
A2174	..............	43
T1892	..............	28
MAINT	..............	22
R2182	..............	18
A2119	..............	12

(top candidates only listed, both tables). So I think we have a clear conclusion: watch out for days when A2172 or A2174 are observing.

[Edit - and guess where A2172 is managed from! Matt, next time your download servers are maxxed out, go and have a chat with Carl Heiles across the campus.]