Splitsville (Nov 02 2009)


log in

Advanced search

Message boards : Technical News : Splitsville (Nov 02 2009)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1390
Credit: 74,079
RAC: 0
United States
Message 944723 - Posted: 2 Nov 2009, 22:57:50 UTC

In case you haven't noticed, we've been low on workunits. As warned in several previous tech news items (and now on the front page) we're still in the process of converting our data pipeline to use the new radar blanking suite (to vastly reduce noise/interference). This conversion process has been slowed by several factors, including these two: it takes a long time to bring up old data from our archives (approximately 4 hours per 50 GB file), and it turns out a lot of these files contain garbage that make it impossible to process (which we can only discover after spending the time to bring the files up here). We are also low of current data because ALFA has been offline for a month due to maintenance.

In better news, ALFA is back up and we're collecting new data again. As well I moved the "testing phase" version of the data pipeline onto the main production data file server, which should generally help as we'll at least speed up disk i/o. Also our assimilator queue finally drained to zero again. I see that people are complaining about lack of work on various threads. We don't guarantee a steady stream of work, but do understand that such a steady stream is important for maintaining public interest. We're doing what we can. I'm getting another file on line as I type this - should be splittable (I hope) sometime this evening.

Our science database server (thumper) lost another disk over the weekend. No big deal, and the RAID recovered with a spare just fine - but nevertheless this is just another reminder that we really need to reconfigure the disk arrays on that system - they are unwieldy and inefficient.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4213
Credit: 34,475,835
RAC: 15,444
United Kingdom
Message 944724 - Posted: 2 Nov 2009, 22:59:30 UTC - in response to Message 944723.

Thanks for the update Matt.

Claggy

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 16,211,560
RAC: 5,231
United States
Message 944725 - Posted: 2 Nov 2009, 23:09:34 UTC - in response to Message 944723.

As usual you are on top of everything. Good to hear ALFA is back up and running and especially that you're getting another tape up and (hopefully) splitting. I'm crunching my last two right now on my CPUs. My GPU ran out about an hour ago.

Somehow I picked up some WUs last night or early this morning that lasted me through now. I'm not worried or complaining, the work will come when it comes. No hurry. A little down time will probably be a good change for my poor little old machine.
____________


PROUD MEMBER OF Team Starfire World BOINC

Profile BigDaddyDave
Avatar
Send message
Joined: 8 Oct 00
Posts: 65
Credit: 6,764,947
RAC: 1,309
United States
Message 944729 - Posted: 2 Nov 2009, 23:34:39 UTC - in response to Message 944723.

Hi Matt,

Thanks for all the hard work and I think it's a great idea to post to the front page. I've been with SETI for 9 years and I just started reading the threads last February! I know... I know.


BDDave

____________

Profile LiliKrist
Volunteer tester
Avatar
Send message
Joined: 12 Aug 09
Posts: 333
Credit: 143,167
RAC: 0
Indonesia
Message 944751 - Posted: 3 Nov 2009, 1:52:23 UTC

Just take your time to process of transitioning parts of your data pipeline Master Matt. We will wait with patient =)

Thanks for the update...
____________


N = R x fp x ne x fl x fi x fc x L

DJStarfox
Send message
Joined: 23 May 01
Posts: 1045
Credit: 568,361
RAC: 192
United States
Message 944845 - Posted: 3 Nov 2009, 14:29:53 UTC - in response to Message 944723.

The silver lining on the situation is that having a scarcity of workunits means the servers can catch up. E.g., the assimilator queue draining as you said, etc.

It might even be a good idea to have a "start-up" period where the splitters are running but there is no work being handed out (disable the feeder?). The purpose would be to generate a small queue to handle the spike in demand (at first). If you get data to split today, then I'd say the maintenance window would be the perfect time for that.

At least I will be patient. :)

PS I have one more thought. It is possible or feasible to split a source signal file for MB & AP simultaneously? I'm thinking that would save disk I/O.

Message boards : Technical News : Splitsville (Nov 02 2009)

Copyright © 2014 University of California