Splitsville (Nov 02 2009)

Message boards : Technical News : Splitsville (Nov 02 2009)
Message board moderation

To post messages, you must log in.

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 944723 - Posted: 2 Nov 2009, 22:57:50 UTC

In case you haven't noticed, we've been low on workunits. As warned in several previous tech news items (and now on the front page) we're still in the process of converting our data pipeline to use the new radar blanking suite (to vastly reduce noise/interference). This conversion process has been slowed by several factors, including these two: it takes a long time to bring up old data from our archives (approximately 4 hours per 50 GB file), and it turns out a lot of these files contain garbage that make it impossible to process (which we can only discover after spending the time to bring the files up here). We are also low of current data because ALFA has been offline for a month due to maintenance.

In better news, ALFA is back up and we're collecting new data again. As well I moved the "testing phase" version of the data pipeline onto the main production data file server, which should generally help as we'll at least speed up disk i/o. Also our assimilator queue finally drained to zero again. I see that people are complaining about lack of work on various threads. We don't guarantee a steady stream of work, but do understand that such a steady stream is important for maintaining public interest. We're doing what we can. I'm getting another file on line as I type this - should be splittable (I hope) sometime this evening.

Our science database server (thumper) lost another disk over the weekend. No big deal, and the RAID recovered with a spare just fine - but nevertheless this is just another reminder that we really need to reconfigure the disk arrays on that system - they are unwieldy and inefficient.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 944723 · Report as offensive
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 944724 - Posted: 2 Nov 2009, 22:59:30 UTC - in response to Message 944723.  

Thanks for the update Matt.

ID: 944724 · Report as offensive
Profile perryjay
Volunteer tester

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 944725 - Posted: 2 Nov 2009, 23:09:34 UTC - in response to Message 944723.  

As usual you are on top of everything. Good to hear ALFA is back up and running and especially that you're getting another tape up and (hopefully) splitting. I'm crunching my last two right now on my CPUs. My GPU ran out about an hour ago.

Somehow I picked up some WUs last night or early this morning that lasted me through now. I'm not worried or complaining, the work will come when it comes. No hurry. A little down time will probably be a good change for my poor little old machine.

ID: 944725 · Report as offensive
Profile BigDaddyDave

Send message
Joined: 8 Oct 00
Posts: 67
Credit: 20,337,995
RAC: 15
United States
Message 944729 - Posted: 2 Nov 2009, 23:34:39 UTC - in response to Message 944723.  

Hi Matt,

Thanks for all the hard work and I think it's a great idea to post to the front page. I've been with SETI for 9 years and I just started reading the threads last February! I know... I know.


ID: 944729 · Report as offensive
Profile LiliKrist
Volunteer tester

Send message
Joined: 12 Aug 09
Posts: 333
Credit: 143,167
RAC: 0
Message 944751 - Posted: 3 Nov 2009, 1:52:23 UTC

Just take your time to process of transitioning parts of your data pipeline Master Matt. We will wait with patient =)

Thanks for the update...

N = R x fp x ne x fl x fi x fc x L
ID: 944751 · Report as offensive

Send message
Joined: 23 May 01
Posts: 1066
Credit: 1,226,053
RAC: 2
United States
Message 944845 - Posted: 3 Nov 2009, 14:29:53 UTC - in response to Message 944723.  

The silver lining on the situation is that having a scarcity of workunits means the servers can catch up. E.g., the assimilator queue draining as you said, etc.

It might even be a good idea to have a "start-up" period where the splitters are running but there is no work being handed out (disable the feeder?). The purpose would be to generate a small queue to handle the spike in demand (at first). If you get data to split today, then I'd say the maintenance window would be the perfect time for that.

At least I will be patient. :)

PS I have one more thought. It is possible or feasible to split a source signal file for MB & AP simultaneously? I'm thinking that would save disk I/O.
ID: 944845 · Report as offensive

Message boards : Technical News : Splitsville (Nov 02 2009)

©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.