Splitsville (Nov 02 2009)

Message boards : Technical News : Splitsville (Nov 02 2009)

Author	Message
Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 944723 - Posted: 2 Nov 2009, 22:57:50 UTC
	In case you haven't noticed, we've been low on workunits. As warned in several previous tech news items (and now on the front page) we're still in the process of converting our data pipeline to use the new radar blanking suite (to vastly reduce noise/interference). This conversion process has been slowed by several factors, including these two: it takes a long time to bring up old data from our archives (approximately 4 hours per 50 GB file), and it turns out a lot of these files contain garbage that make it impossible to process (which we can only discover after spending the time to bring the files up here). We are also low of current data because ALFA has been offline for a month due to maintenance. In better news, ALFA is back up and we're collecting new data again. As well I moved the "testing phase" version of the data pipeline onto the main production data file server, which should generally help as we'll at least speed up disk i/o. Also our assimilator queue finally drained to zero again. I see that people are complaining about lack of work on various threads. We don't guarantee a steady stream of work, but do understand that such a steady stream is important for maintaining public interest. We're doing what we can. I'm getting another file on line as I type this - should be splittable (I hope) sometime this evening. Our science database server (thumper) lost another disk over the weekend. No big deal, and the RAID recovered with a spare just fine - but nevertheless this is just another reminder that we really need to reconfigure the disk arrays on that system - they are unwieldy and inefficient. - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 944723 ·

Claggy Volunteer tester Send message Joined: 5 Jul 99 Posts: 3363 Credit: 25,950,039 RAC: 1,120	Message 944724 - Posted: 2 Nov 2009, 22:59:30 UTC - in response to Message 944723.
	Thanks for the update Matt. Claggy
	ID: 944724 ·

perryjay Send message Joined: 20 Aug 02 Posts: 3359 Credit: 9,646,195 RAC: 10,378	Message 944725 - Posted: 2 Nov 2009, 23:09:34 UTC - in response to Message 944723.
	As usual you are on top of everything. Good to hear ALFA is back up and running and especially that you're getting another tape up and (hopefully) splitting. I'm crunching my last two right now on my CPUs. My GPU ran out about an hour ago. Somehow I picked up some WUs last night or early this morning that lasted me through now. I'm not worried or complaining, the work will come when it comes. No hurry. A little down time will probably be a good change for my poor little old machine. ____________ PROUD MEMBER OF Team Starfire World BOINC
	ID: 944725 ·

BigDaddyDave Send message Joined: 8 Oct 00 Posts: 64 Credit: 4,890,676 RAC: 3,595	Message 944729 - Posted: 2 Nov 2009, 23:34:39 UTC - in response to Message 944723.
	Hi Matt, Thanks for all the hard work and I think it's a great idea to post to the front page. I've been with SETI for 9 years and I just started reading the threads last February! I know... I know. BDDave ____________
	ID: 944729 ·

LiliKrist Volunteer tester Send message Joined: 12 Aug 09 Posts: 333 Credit: 143,167 RAC: 0	Message 944751 - Posted: 3 Nov 2009, 1:52:23 UTC
	Just take your time to process of transitioning parts of your data pipeline Master Matt. We will wait with patient =) Thanks for the update... ____________ N = R x fp x ne x fl x fi x fc x L
	ID: 944751 ·

DJStarfox Send message Joined: 23 May 01 Posts: 1040 Credit: 485,117 RAC: 71	Message 944845 - Posted: 3 Nov 2009, 14:29:53 UTC - in response to Message 944723.
	The silver lining on the situation is that having a scarcity of workunits means the servers can catch up. E.g., the assimilator queue draining as you said, etc. It might even be a good idea to have a "start-up" period where the splitters are running but there is no work being handed out (disable the feeder?). The purpose would be to generate a small queue to handle the spike in demand (at first). If you get data to split today, then I'd say the maintenance window would be the perfect time for that. At least I will be patient. :) PS I have one more thought. It is possible or feasible to split a source signal file for MB & AP simultaneously? I'm thinking that would save disk I/O.
	ID: 944845 ·

Message boards : Technical News : Splitsville (Nov 02 2009)