One Nail Draws Another (Nov 21 2007)


log in

Advanced search

Message boards : Technical News : One Nail Draws Another (Nov 21 2007)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 682181 - Posted: 21 Nov 2007, 21:41:44 UTC

I wasn't selected for jury duty! Hooray! I fulfilled my civic duty without having to miss work!

So we're in the middle of a slight server malaise - the data we're currently splitting/sending out is of the sort that it gets processed quickly and returned much faster than average. That's one big difference with our current multi-beam processing: the variance of data processing time per workunit is far greater than before, so we get into these unpredictable heavy periods and have no choice but to wait them out.

Well... that's not entirely true. Jeff actually moved the rest of the raw data from this day out of the way so we can move to other days which are potentially more friendly towards our servers. Also we could predict, with very coarse resolution, what days might be "rough" before sending them through the pipeline. But we're going to split the data anyway at some point, so why not get it over with? At any rate we started more splitters to keep from running out of work, and we'll keep an eye on this as we progress into the holiday weekend.

Happy Thanksgiving! Or if you're not in the U.S. - Happy Thursday!

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Brian Silvers
Send message
Joined: 11 Jun 99
Posts: 1681
Credit: 492,052
RAC: 0
United States
Message 682201 - Posted: 21 Nov 2007, 22:41:13 UTC - in response to Message 682181.

But we're going to split the data anyway at some point, so why not get it over with?


Please understand the spirit in which I'm going to say what I'm about to say:

Was it wise to attempt such a feat after back-to-back daily outages and going into a long weekend when you probably would rather be away and not have to worry about this?

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 682208 - Posted: 21 Nov 2007, 22:58:08 UTC - in response to Message 682201.

Was it wise to attempt such a feat after back-to-back daily outages and going into a long weekend when you probably would rather be away and not have to worry about this?


Understood. The point is that without significant effort up front there's no way to tell the quality of the data before it enters the splitter queue. So once it enters the queue and we find it to be "ugly," we might as well just let it finish. In today's case, the file was already half split by the time we determined the issue and its cause, so we're just gonna let it wrap up naturally, which it looks like it will in a few hours.

In any case, there's no worry: SETI@home is such a dynamic system with so many variables mixed with the regularly flexible schedules of our staff (not just during a holiday weekend). So anything can happen at any time with only a small subset of people on hand - and many times that subset contains zero elements. After many years of this you become an expert on unexpected minor/major disaster recovery and simply stop worrying.

And the current state is by no means a disaster. A temporary uncomfortable period that is showing all signs of passing.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 682210 - Posted: 21 Nov 2007, 23:06:16 UTC

Thanks for the updates Matt - Have an Excellent Holiday too . . .

> that goes for the Rest of You @ Berkeley - Enjoy Your Holidays also . . .

____________
BOINC Wiki . . .

Science Status Page . . .

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 682519 - Posted: 22 Nov 2007, 12:19:50 UTC



@ Matt - Eclecticity - and Diversity - 1998 through 2004 . . . too bad these bands don't do something again eh . . . i like what they did though

U TOTEM in concert at New Music Montreal - November 5, 1990 "One Nail Draws Another" - Part 1

U TOTEM in concert at New Music Montreal - November 5, 1990 "One Nail Draws Another" - Part 2



1. One Nail Draws Another
2. Two Looks At One End
3. Dance Of The Awkward
4. Both Your Houses
5. Yellow Umbrella Gallery
6. The Judas Goat
7. Vagabonds Home

. . . i sure miss Montreal these days ;)

____________
BOINC Wiki . . .

Science Status Page . . .

Sirius B
Volunteer tester
Avatar
Send message
Joined: 26 Dec 00
Posts: 11491
Credit: 1,721,363
RAC: 1,910
Israel
Message 682524 - Posted: 22 Nov 2007, 12:53:46 UTC - in response to Message 682519.

Thanks for the info Matt.

Also. @ Dr C.E.T.I., thanks for the music links.

Regards

PJ
____________

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 682562 - Posted: 22 Nov 2007, 15:14:12 UTC


You are Most Welcome PJ . . . i would wish You a Happy Thanksgiving - but you already had yours on October 8th . . . so enjoy your Weekend . . .

____________
BOINC Wiki . . .

Science Status Page . . .

Josef W. SegurProject donor
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4296
Credit: 1,065,441
RAC: 951
United States
Message 682719 - Posted: 22 Nov 2007, 20:00:06 UTC - in response to Message 682208.

...without significant effort up front there's no way to tell the quality of the data before it enters the splitter queue. So once it enters the queue and we find it to be "ugly," we might as well just let it finish...
- Matt

What we're dealing with here is the slew rate of the telescope rather than a data quality issue. I'd think that while dividing the data into 50 GB chunks it might be fairly easy to check start and end times, then query the Arecibo database for telescope position periodically between those times and derive the slew rates.

If the slew rate is consistently above 0.0105 degrees per second the generated WUs will be the quick ones which the system cannot yet handle gracefully. The splitter estimates they will take about 1/6 the time of WUs produced from typical lower slew rates, so the Scheduler assigns 6 times as many WUs per host.

Because we look at those WUs for a shorter time, I suppose "ugly" is a fair description in that sense. In any case, if the GALFACTS project gets into full swing after Arecibo is recommissioned there will be a lot of high slew rate work to deal with, and some planning for that seems in order.
Joe

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8629
Credit: 51,368,165
RAC: 50,228
United Kingdom
Message 682725 - Posted: 22 Nov 2007, 20:12:55 UTC - in response to Message 682719.

...without significant effort up front there's no way to tell the quality of the data before it enters the splitter queue. So once it enters the queue and we find it to be "ugly," we might as well just let it finish...
- Matt

What we're dealing with here is the slew rate of the telescope rather than a data quality issue. I'd think that while dividing the data into 50 GB chunks it might be fairly easy to check start and end times, then query the Arecibo database for telescope position periodically between those times and derive the slew rates.

If the slew rate is consistently above 0.0105 degrees per second the generated WUs will be the quick ones which the system cannot yet handle gracefully. The splitter estimates they will take about 1/6 the time of WUs produced from typical lower slew rates, so the Scheduler assigns 6 times as many WUs per host.

Because we look at those WUs for a shorter time, I suppose "ugly" is a fair description in that sense. In any case, if the GALFACTS project gets into full swing after Arecibo is recommissioned there will be a lot of high slew rate work to deal with, and some planning for that seems in order.
Joe

And the slew rate of the telescope should be known, or at least in principle knowable, from the 'tape' name (== recording date) and the telescope observing schedule.

At the moment, the splitter algorithm seems to call for all active splitters - typically six - to work on the same 'tape' at the same time: so when the high slew rate recordings are encountered, they affect all the WUs split around that time.

Would it be feasible to arrange for each splitter to work on its own 'tape', so that the six splitter instances are working on data recorded on different days, or at least different times of the same day? Then there might be more of a mix of WU durations in the tasks available for download, and we wouldn't get the 6::1 ratio - just average things out a bit.

Henk Haneveld
Send message
Joined: 16 May 99
Posts: 154
Credit: 1,493,205
RAC: 67
Netherlands
Message 682776 - Posted: 22 Nov 2007, 22:04:40 UTC

I assume it is possible to find out how much downloads the servers and the bandwith can handle without becoming overloaded.

If the number of active splitters is limited to producing this maximum number of results and the ready to send queue is set to a much lower maximum (less then 10.000) then it is possible to avoid the download overload completely.

When "normal" runtime results are send out there is enough work for everybody because the splitters can make more results then are needed.
Only when "short" runtime results are produced there will be moments that there is not enough work because it is send out as fast as the splitters can create it.

I think however that it is more acceptable to recieve the "No work" message then to have results sitting on the transfer tab doing download retrys because the servers are overloaded.
____________

DJStarfox
Send message
Joined: 23 May 01
Posts: 1044
Credit: 559,666
RAC: 601
United States
Message 682882 - Posted: 23 Nov 2007, 0:55:04 UTC - in response to Message 682181.

Also we could predict, with very coarse resolution, what days might be "rough" before sending them through the pipeline.


Now, if you could find a way to "tell" the splitters (or even just the feeders) which batches will be processed fast or slow, then theoretically, you would send out a "mix" of both fast and slow processing tasks. I suppose the feeders would have to be much smarter to do this properly (i.e., give the scheduler varied task sizes). But how brilliant of an advancement to BOINC would that be?

Profile Geek@PlayProject donor
Volunteer tester
Avatar
Send message
Joined: 31 Jul 01
Posts: 2467
Credit: 86,143,646
RAC: 1,271
United States
Message 682914 - Posted: 23 Nov 2007, 3:49:33 UTC - in response to Message 682725.

At the moment, the splitter algorithm seems to call for all active splitters - typically six - to work on the same 'tape' at the same time: so when the high slew rate recordings are encountered, they affect all the WUs split around that time.

Would it be feasible to arrange for each splitter to work on its own 'tape', so that the six splitter instances are working on data recorded on different days, or at least different times of the same day? Then there might be more of a mix of WU durations in the tasks available for download, and we wouldn't get the 6::1 ratio - just average things out a bit.


This is probably the smartest and simplest solution. Except that one splitter working on one "file" would take forever + 1 day to complete it. Even with 4 to 6 splitters working on the same "file" it takes a long time to finish it.

DJStarfox
Send message
Joined: 23 May 01
Posts: 1044
Credit: 559,666
RAC: 601
United States
Message 683085 - Posted: 23 Nov 2007, 14:51:59 UTC - in response to Message 682914.

At the moment, the splitter algorithm seems to call for all active splitters - typically six - to work on the same 'tape' at the same time: so when the high slew rate recordings are encountered, they affect all the WUs split around that time.

Would it be feasible to arrange for each splitter to work on its own 'tape', so that the six splitter instances are working on data recorded on different days, or at least different times of the same day? Then there might be more of a mix of WU durations in the tasks available for download, and we wouldn't get the 6::1 ratio - just average things out a bit.


This is probably the smartest and simplest solution. Except that one splitter working on one "file" would take forever + 1 day to complete it. Even with 4 to 6 splitters working on the same "file" it takes a long time to finish it.


That would be a much simplier solution. However, you'll need 6x the disk space for pre-split data, as those files will spend that much longer in the cycle.

Profile Clyde C. Phillips, III
Send message
Joined: 2 Aug 00
Posts: 1851
Credit: 5,955,047
RAC: 0
United States
Message 683127 - Posted: 23 Nov 2007, 17:29:15 UTC

0.0105 degree per second times 107 seconds is about 1.12 degrees. So any angle range above about 1.12 degrees doesn't require as much computation, probably because it's more difficult to study a possible radio source that's zipping through rather than one that's moving more slowly through the telescope field-of-view, right?
____________

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8629
Credit: 51,368,165
RAC: 50,228
United Kingdom
Message 696813 - Posted: 2 Jan 2008, 22:30:23 UTC - in response to Message 682725.

...without significant effort up front there's no way to tell the quality of the data before it enters the splitter queue. So once it enters the queue and we find it to be "ugly," we might as well just let it finish...
- Matt

What we're dealing with here is the slew rate of the telescope rather than a data quality issue. I'd think that while dividing the data into 50 GB chunks it might be fairly easy to check start and end times, then query the Arecibo database for telescope position periodically between those times and derive the slew rates.

If the slew rate is consistently above 0.0105 degrees per second the generated WUs will be the quick ones which the system cannot yet handle gracefully. The splitter estimates they will take about 1/6 the time of WUs produced from typical lower slew rates, so the Scheduler assigns 6 times as many WUs per host.

Because we look at those WUs for a shorter time, I suppose "ugly" is a fair description in that sense. In any case, if the GALFACTS project gets into full swing after Arecibo is recommissioned there will be a lot of high slew rate work to deal with, and some planning for that seems in order.
Joe

And the slew rate of the telescope should be known, or at least in principle knowable, from the 'tape' name (== recording date) and the telescope observing schedule.

At the moment, the splitter algorithm seems to call for all active splitters - typically six - to work on the same 'tape' at the same time: so when the high slew rate recordings are encountered, they affect all the WUs split around that time.

Would it be feasible to arrange for each splitter to work on its own 'tape', so that the six splitter instances are working on data recorded on different days, or at least different times of the same day? Then there might be more of a mix of WU durations in the tasks available for download, and we wouldn't get the 6::1 ratio - just average things out a bit.

Interesting new pattern on the Server Status page - looks as if this idea from way back might be getting a trial run. It'll be interesting to see how it goes.

In the meantime, I've found a way of automating recovery of the actual telescope recording time (in Arecibo's AST time-zone) from my logs of WUs past. So I can tell exactly when the telescope was observing each of the last 525 'shorty' WUs that passed through my E5320 box. Before I ruin my remaining eyesight poring through the Arecibo schedule PDFs we found before Christmas, does anyone have any idea if the telescope project allocation data could be downloaded anywhere in a structured, database-able format?

Josef W. SegurProject donor
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4296
Credit: 1,065,441
RAC: 951
United States
Message 696823 - Posted: 2 Jan 2008, 23:09:41 UTC - in response to Message 696813.

...
Before I ruin my remaining eyesight poring through the Arecibo schedule PDFs we found before Christmas, does anyone have any idea if the telescope project allocation data could be downloaded anywhere in a structured, database-able format?

The files in http://www.naic.edu/vscience/schedule/Scheds/ look like they'd be fairly easy to parse once the relationships to the schedule GIFs is figured out. They even give times in textual form so you don't have to estimate from the side scales.
Joe

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8629
Credit: 51,368,165
RAC: 50,228
United Kingdom
Message 696832 - Posted: 2 Jan 2008, 23:42:44 UTC - in response to Message 696823.

...
Before I ruin my remaining eyesight poring through the Arecibo schedule PDFs we found before Christmas, does anyone have any idea if the telescope project allocation data could be downloaded anywhere in a structured, database-able format?

The files in http://www.naic.edu/vscience/schedule/Scheds/ look like they'd be fairly easy to parse once the relationships to the schedule GIFs is figured out. They even give times in textual form so you don't have to estimate from the side scales.
Joe

Excellent - exactly what I was looking for - thanks, Joe.

Now I just have to quantize my data in 15-minute chunks (I guessed at 10 minutes the first time round, because it's easier to do the string manipulation!), and I'll try and post something. It'll be tomorrow, though.

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8629
Credit: 51,368,165
RAC: 50,228
United Kingdom
Message 697976 - Posted: 6 Jan 2008, 21:08:04 UTC - in response to Message 696832.
Last modified: 6 Jan 2008, 21:44:42 UTC

...
Before I ruin my remaining eyesight poring through the Arecibo schedule PDFs we found before Christmas, does anyone have any idea if the telescope project allocation data could be downloaded anywhere in a structured, database-able format?

The files in http://www.naic.edu/vscience/schedule/Scheds/ look like they'd be fairly easy to parse once the relationships to the schedule GIFs is figured out. They even give times in textual form so you don't have to estimate from the side scales.
Joe

Excellent - exactly what I was looking for - thanks, Joe.

Now I just have to quantize my data in 15-minute chunks (I guessed at 10 minutes the first time round, because it's easier to do the string manipulation!), and I'll try and post something. It'll be tomorrow, though.

Bit of a late 'tomorrow', but the first (raw, unchecked) data are coming out of the cruncher.

Candidates for the VVHAR storm over the holidays (AR>2, issued in late December):

Arecibo Project CountOfWU A2172 .............. 81 A2174 .............. 43 A2222 .............. 12


VHAR work in general, last three months:

Arecibo Project CountOfWU A2060 ............. 189 A2172 .............. 95 A2222 .............. 75 P2030 .............. 43 A2174 .............. 43 T1892 .............. 28 MAINT .............. 22 R2182 .............. 18 A2119 .............. 12


(top candidates only listed, both tables). So I think we have a clear conclusion: watch out for days when A2172 or A2174 are observing.

[Edit - and guess where A2172 is managed from! Matt, next time your download servers are maxxed out, go and have a chat with Carl Heiles across the campus.]

Message boards : Technical News : One Nail Draws Another (Nov 21 2007)

Copyright © 2014 University of California