Now it's 2014 (Jan 07 2014)

Message boards : Technical News : Now it's 2014 (Jan 07 2014)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1461596 - Posted: 7 Jan 2014, 23:34:02 UTC

The nightmare that was my 2013 schedule is behind us. Actually 2014 may be better, or may be worse. Time will tell.

Ugh, it's been a while. Once again I'm finding myself in major-catchup-mode when trying to figure out what to mention. On the surface, it doesn't seem that much has been happening (maybe), but we've been quite busy on various projects. Here's some of them off the top of my head.

First, neither the perfect solution nor the money ever appeared to magically speed up our science databases, so were are working on plan B: vastly reduce the size of these databases so we can actually work with them! We've been coming up with clever ways to quickly bring our final science results down to less than 10% its original size without a minimal, perhaps zero, compromise in sensitivity. That's one project. Maybe we'll create a new client which can do this reduction for us (which will require much larger workunits).

Second, we recently obtained a generous donation of a lustre file server from Xyratex. There was some effort to install this system and ramp up on managing lustre, but now we have a 120TB sandbox to play with. Currently we're using it to house all kinds of data from various other SETI projects, or as a backup/SETI@home data buffer. As we get more comfortable with the system we'll push on it with more i/o intensive projects.

Third, as SETI@home chugs along we are also dividing our efforts working on various other SETI projects. This will all be clearer after we launch (no ETA as of yet) our new web site.

Meanwhile we had a few crashes recently. As we now have faster network inside the colocation facility and larger disk arrays, we are hitting some linux i/o limits causing CPUs to randomly lock up (requiring hard power cycles). Anybody have any tips on this front? I'm messing with /sys/block/device/queue/scheduler to see if that helps.

Also on of Eric's systems had a RAID6 on it which suffered 3 drive failures within a week over the holiday. Such cruel timing. We're recovering from that (it's backed up regularly) but until it's back on line other co-dependent systems are getting headaches.

Same old, same old. I'm looking forward to the newer web site, which will have more contributors and more information - one of our general problems over the years.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1461596 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1461603 - Posted: 7 Jan 2014, 23:46:03 UTC - in response to Message 1461596.  

Thanks for the update Matt, welcome back.

Claggy
ID: 1461603 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1461606 - Posted: 7 Jan 2014, 23:51:14 UTC - in response to Message 1461603.  

Thanks for the update Matt, welcome back.

Claggy

+1

Cheers.
ID: 1461606 · Report as offensive
Sleepy
Volunteer tester
Avatar

Send message
Joined: 21 May 99
Posts: 219
Credit: 98,947,784
RAC: 28,360
Italy
Message 1461612 - Posted: 8 Jan 2014, 0:00:15 UTC - in response to Message 1461603.  

Thanks for the update Matt, welcome back.

Claggy


+1 = >>>> 2
ID: 1461612 · Report as offensive
Profile ivan
Volunteer tester
Avatar

Send message
Joined: 5 Mar 01
Posts: 783
Credit: 348,560,338
RAC: 223
United Kingdom
Message 1461629 - Posted: 8 Jan 2014, 1:35:53 UTC - in response to Message 1461603.  

Thanks for the update Matt, welcome back.

Indeed!
ID: 1461629 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 1461636 - Posted: 8 Jan 2014, 2:23:44 UTC

Matt, thanks for the news!

I wish all a Happy New Year 2014!

* Best regards! :-) * Philip J. Fry, team seti.international founder. * Optimize your PC for higher RAC. * SETI@home needs your help. *
ID: 1461636 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 1461644 - Posted: 8 Jan 2014, 2:45:33 UTC - in response to Message 1461596.  

That's terrific news Matt. Thanks for keeping us up to date!
ID: 1461644 · Report as offensive
Thomas
Volunteer tester

Send message
Joined: 9 Dec 11
Posts: 1499
Credit: 1,345,576
RAC: 0
France
Message 1461704 - Posted: 8 Jan 2014, 6:57:53 UTC - in response to Message 1461612.  

Thanks for the update Matt, welcome back.

Claggy


+1 = >>>> 2

+3 and HNY Matt ! :)
ID: 1461704 · Report as offensive
David S
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 18352
Credit: 27,761,924
RAC: 12
United States
Message 1461909 - Posted: 8 Jan 2014, 20:02:30 UTC

Good to hear from you, Matt.
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

ID: 1461909 · Report as offensive
Profile Uli
Volunteer tester
Avatar

Send message
Joined: 6 Feb 00
Posts: 10923
Credit: 5,996,015
RAC: 1
Germany
Message 1462064 - Posted: 9 Jan 2014, 7:42:01 UTC

Thank you Matt for the update.

Maybe some of the chatter (not that has it been to loud lately) would die down, if you made your posts in the News Section.
Hearing from you is always good News to me, plus it would show on the front page.

Just a thought.
Pluto will always be a planet to me.

Seti Ambassador
Not to late to order an Anni Shirt
ID: 1462064 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1462610 - Posted: 10 Jan 2014, 16:59:29 UTC

Glad to see you back in the lab, Matt. And thank you for the background info.
Upward and onward.

Do you have the drives you need to get Eric's RAID rig back in action?
Or do we need an emergency fundraiser to help get it back online quickly?
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1462610 · Report as offensive
Hugo A. Durantini Luca

Send message
Joined: 24 Sep 09
Posts: 14
Credit: 7,253,842
RAC: 10
Argentina
Message 1464963 - Posted: 16 Jan 2014, 11:10:33 UTC
Last modified: 16 Jan 2014, 11:13:02 UTC

Thanks for the update, is wonderful to have some news and can't wait for the new developments! :)

ID: 1464963 · Report as offensive
Profile Julie
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 28 Oct 09
Posts: 34041
Credit: 18,883,157
RAC: 18
Belgium
Message 1465155 - Posted: 16 Jan 2014, 19:54:11 UTC

Thanx for the update Matt:)
rOZZ
Music
Pictures
ID: 1465155 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24870
Credit: 3,081,182
RAC: 7
Ireland
Message 1465228 - Posted: 16 Jan 2014, 22:43:37 UTC

Thanks for the update Matt. You mention a server donation. When you've finally got that sorted, any chance of calling it: Curmudgeon
ID: 1465228 · Report as offensive
Profile Andrew Clayton
Avatar

Send message
Joined: 12 Apr 99
Posts: 7
Credit: 907,810
RAC: 0
United Kingdom
Message 1470555 - Posted: 30 Jan 2014, 0:44:46 UTC

Regarding your Linux IO problems.

I wonder if you really are seeing lock ups or just non-responsive behaviour.

Not sure the specs on that machine. But I imagine it has quite a few GBs of RAM.
In which case I wonder if your hitting the VM writeback problem.

The kernel allows a certain amount of dirty data (data that needs to be written
out to backing store) to build up in memory. In current kernels, this can be as
much as 20% of your RAM that it will need to write out and can easily block
other IO causing long stall periods where it may seem that the machine has hung.

(These defaults where made in the heady days of having a machine with as much
as 64MB of RAM!).

It's also recognised that today even having a percentage at all isn't really
suitable and thus byte limits have also been introduced (but not used by
default).

So you may want to have a look at tuning

/proc/sys/vm/dirty_background_bytes and
/proc/sys/vm/dirty_bytes

Which ever of bytes or ratio is set last is what's used.

As to what to set them, that may take a bit of trial and error to see what works
best for you.

You could start out with say (8 & 24MB), you may well need to go quite a bit
higher, but you probably want to avoid having GBs of dirty data to be writing
out.

# echo 8388608 > /proc/sys/vm/dirty_background_bytes
# echo 25165824 > /proc/sys/vm/dirty_bytes

(Stick in /etc/sysctl.conf to make it permanent)

You can watch this in action with something like

$ watch -n 5 grep -e ^Dirty: -e ^Writeback: /proc/meminfo

You'll see the dirty value build then when it reaches the limit you'll see the
writeback value ramp up.

See https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Cheers,
Andrew
ID: 1470555 · Report as offensive
Filipe

Send message
Joined: 12 Aug 00
Posts: 218
Credit: 21,281,677
RAC: 20
Portugal
Message 1470837 - Posted: 30 Jan 2014, 19:03:05 UTC

We've been coming up with clever ways to quickly bring our final science results down to less than 10% its original size without a minimal, perhaps zero, compromise in sensitivity.


More news on that front?

A smaller science database will open the doors to a manageable Near time persistency checker?
ID: 1470837 · Report as offensive
Profile edjcox
Avatar

Send message
Joined: 20 May 99
Posts: 96
Credit: 5,878,353
RAC: 0
United States
Message 1470977 - Posted: 31 Jan 2014, 3:03:21 UTC

Currently we're using it to house all kinds of data from various other SETI projects, or as a backup/SETI@home data buffer.

Why?
Never engage stupid people at their level, they then have the home court advantage.....
ID: 1470977 · Report as offensive
Profile Donald L. Johnson
Avatar

Send message
Joined: 5 Aug 02
Posts: 8240
Credit: 14,654,533
RAC: 20
United States
Message 1471848 - Posted: 2 Feb 2014, 5:10:45 UTC - in response to Message 1470977.  

Currently we're using it to house all kinds of data from various other SETI projects, or as a backup/SETI@home data buffer.

Why?

Matt answered that question in the next sentence, ed:

As we get more comfortable with the system we'll push on it with more i/o intensive projects.

Donald
Infernal Optimist / Submariner, retired
ID: 1471848 · Report as offensive
Gene Todd

Send message
Joined: 10 Feb 01
Posts: 16
Credit: 20,603,027
RAC: 245
United States
Message 1507039 - Posted: 21 Apr 2014, 21:40:25 UTC

Strange, very strange. I have downloaded the latest version and now all of my work downloads fail. What is happening?
ID: 1507039 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 1507047 - Posted: 21 Apr 2014, 22:00:01 UTC - in response to Message 1507039.  

It is best to ask for assistance with issues in the Questions and Answers forum or in Number Crunching. There's a lot of very knowledgeable people who can help you if you ask.
ID: 1507047 · Report as offensive

Message boards : Technical News : Now it's 2014 (Jan 07 2014)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.