Dam Breaks (Aug 17 2009)


log in

Advanced search

Message boards : Technical News : Dam Breaks (Aug 17 2009)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 926844 - Posted: 17 Aug 2009, 21:22:19 UTC

Okay things haven't been running so well the past couple of days. First, there were some mount problems in the middle of last week which caused our assimilator queue to clog up. This inflates our result table causing all kinds of table fragmentation which never helps the general pipeline. Later in the week I noticed the spike table in the science table was running out of space, so Bob added a few more database chunks. That process eats up a bunch of disk i/o, causing splitters/assimilators to slow down temporarily. But then we hit some major chokepoint causing work production to grind to a halt.

Actually it was worse than that - things were working normally, but only really slowly. This makes it hard to find an obvious smoking gun. Usually this is a symptom of heavy disk/database i/o on thumper. We were testing all that this morning by turning processes off but to no avail.

So.. remember how I mentioned in my last note how we just got new raw data from Arecibo? Well, the script copying it over to the raw data storage server failed to register the file system was full, and packed it up tight. Turns out this caused the storage server some distress, and when I finally checked into it this morning the load was high and all the nfsd's were in disk wait. I deleted one excess file, the nfsd's sprung to life and the whole dam broke, the splitters charged full steam ahead, and the network bandwidth is now tapped out trying to catch up on demand. Fair enough.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile S@NL - Eesger - www.knoop.nl
Avatar
Send message
Joined: 7 Oct 01
Posts: 384
Credit: 37,186,031
RAC: 14,299
Netherlands
Message 926853 - Posted: 17 Aug 2009, 22:17:06 UTC

As always, great work! Keep them wu's comming ;)
____________
The SETI@Home Gauntlet 2012 april 16 - 30| info / chat | STATS

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 678
Credit: 5,926,770
RAC: 3,988
New Zealand
Message 926937 - Posted: 18 Aug 2009, 6:27:58 UTC - in response to Message 926844.
Last modified: 18 Aug 2009, 6:35:21 UTC


and when I finally checked into it this morning the load was high and all the nfsd's were in disk wait. I deleted one excess file, the nfsd's sprung to life and the whole dam broke, the splitters charged full steam ahead, and the network bandwidth is now tapped out trying to catch up on demand. Fair enough.

Thank you for getting work flowing again. How long dose it take to fill the local drive(s) with the new data & was it a 750GB File that you deleted to get the work flowing? It looks as I'm one of the lucky one I have a couple of AP units and as I type this this are none ready to send.
____________

Live in NZ y not join Smile City?

Message boards : Technical News : Dam Breaks (Aug 17 2009)

Copyright © 2014 University of California