Now it's 2014 (Jan 07 2014)


log in

Advanced search

Message boards : Technical News : Now it's 2014 (Jan 07 2014)

1 · 2 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1461596 - Posted: 7 Jan 2014, 23:34:02 UTC

The nightmare that was my 2013 schedule is behind us. Actually 2014 may be better, or may be worse. Time will tell.

Ugh, it's been a while. Once again I'm finding myself in major-catchup-mode when trying to figure out what to mention. On the surface, it doesn't seem that much has been happening (maybe), but we've been quite busy on various projects. Here's some of them off the top of my head.

First, neither the perfect solution nor the money ever appeared to magically speed up our science databases, so were are working on plan B: vastly reduce the size of these databases so we can actually work with them! We've been coming up with clever ways to quickly bring our final science results down to less than 10% its original size without a minimal, perhaps zero, compromise in sensitivity. That's one project. Maybe we'll create a new client which can do this reduction for us (which will require much larger workunits).

Second, we recently obtained a generous donation of a lustre file server from Xyratex. There was some effort to install this system and ramp up on managing lustre, but now we have a 120TB sandbox to play with. Currently we're using it to house all kinds of data from various other SETI projects, or as a backup/SETI@home data buffer. As we get more comfortable with the system we'll push on it with more i/o intensive projects.

Third, as SETI@home chugs along we are also dividing our efforts working on various other SETI projects. This will all be clearer after we launch (no ETA as of yet) our new web site.

Meanwhile we had a few crashes recently. As we now have faster network inside the colocation facility and larger disk arrays, we are hitting some linux i/o limits causing CPUs to randomly lock up (requiring hard power cycles). Anybody have any tips on this front? I'm messing with /sys/block/device/queue/scheduler to see if that helps.

Also on of Eric's systems had a RAID6 on it which suffered 3 drive failures within a week over the holiday. Such cruel timing. We're recovering from that (it's backed up regularly) but until it's back on line other co-dependent systems are getting headaches.

Same old, same old. I'm looking forward to the newer web site, which will have more contributors and more information - one of our general problems over the years.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Claggy
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4048
Credit: 32,693,420
RAC: 508
United Kingdom
Message 1461603 - Posted: 7 Jan 2014, 23:46:03 UTC - in response to Message 1461596.

Thanks for the update Matt, welcome back.

Claggy

Profile Wiggo
Avatar
Send message
Joined: 24 Jan 00
Posts: 6530
Credit: 90,634,498
RAC: 75,081
Australia
Message 1461606 - Posted: 7 Jan 2014, 23:51:14 UTC - in response to Message 1461603.

Thanks for the update Matt, welcome back.

Claggy

+1

Cheers.

sleepy
Avatar
Send message
Joined: 21 May 99
Posts: 76
Credit: 21,524,137
RAC: 20,847
Italy
Message 1461612 - Posted: 8 Jan 2014, 0:00:15 UTC - in response to Message 1461603.

Thanks for the update Matt, welcome back.

Claggy


+1 = >>>> 2

Profile ivan
Volunteer tester
Avatar
Send message
Joined: 5 Mar 01
Posts: 579
Credit: 131,577,100
RAC: 114,011
United Kingdom
Message 1461629 - Posted: 8 Jan 2014, 1:35:53 UTC - in response to Message 1461603.

Thanks for the update Matt, welcome back.

Indeed!
____________

Profile [seti.international] Dirk Sadowski
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7028
Credit: 59,291,462
RAC: 20,778
Germany
Message 1461636 - Posted: 8 Jan 2014, 2:23:44 UTC

Matt, thanks for the news!

I wish all a Happy New Year 2014!

* Best regards! :-) * Philip J. Fry, team seti.international founder. * Optimize your PC for higher RAC. * SETI@home needs your help. *
____________
BR



>Das Deutsche Cafe. The German Cafe.<

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13542
Credit: 29,406,652
RAC: 15,955
United States
Message 1461644 - Posted: 8 Jan 2014, 2:45:33 UTC - in response to Message 1461596.

That's terrific news Matt. Thanks for keeping us up to date!

Profile {BDC} Thomas Dupont
Volunteer tester
Avatar
Send message
Joined: 9 Dec 11
Posts: 3642
Credit: 1,288,029
RAC: 366
France
Message 1461704 - Posted: 8 Jan 2014, 6:57:53 UTC - in response to Message 1461612.

Thanks for the update Matt, welcome back.

Claggy


+1 = >>>> 2

+3 and HNY Matt ! :)
____________
Team Founder BRIGADE DU COSMOS




BRIGADE DU COSMOS is proudly sponsored by Zenovia Digital Exchange

N9JFE David S
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 10760
Credit: 13,484,676
RAC: 14,618
United States
Message 1461909 - Posted: 8 Jan 2014, 20:02:30 UTC

Good to hear from you, Matt.
____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


Profile Uli
Volunteer tester
Avatar
Send message
Joined: 6 Feb 00
Posts: 9588
Credit: 5,311,550
RAC: 3,864
Germany
Message 1462064 - Posted: 9 Jan 2014, 7:42:01 UTC

Thank you Matt for the update.

Maybe some of the chatter (not that has it been to loud lately) would die down, if you made your posts in the News Section.
Hearing from you is always good News to me, plus it would show on the front page.

Just a thought.
____________
Pluto will always be a planet to me.
Order your 15th Seti Anniversary Shirt today. Just PM me for details.
Cash Donation Specialist

Seti Ambassador

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38327
Credit: 560,996,578
RAC: 653,877
United States
Message 1462610 - Posted: 10 Jan 2014, 16:59:29 UTC

Glad to see you back in the lab, Matt. And thank you for the background info.
Upward and onward.

Do you have the drives you need to get Eric's RAID rig back in action?
Or do we need an emergency fundraiser to help get it back online quickly?
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Hugo A. Durantini Luca
Send message
Joined: 24 Sep 09
Posts: 6
Credit: 1,137,143
RAC: 1,371
Argentina
Message 1464963 - Posted: 16 Jan 2014, 11:10:33 UTC
Last modified: 16 Jan 2014, 11:13:02 UTC

Thanks for the update, is wonderful to have some news and can't wait for the new developments! :)

Profile Julie
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 28 Oct 09
Posts: 18735
Credit: 3,388,817
RAC: 7,316
Belgium
Message 1465155 - Posted: 16 Jan 2014, 19:54:11 UTC

Thanx for the update Matt:)
____________


rOZZ

Profile Sirius B
Volunteer tester
Avatar
Send message
Joined: 26 Dec 00
Posts: 10257
Credit: 1,530,132
RAC: 264
United Kingdom
Message 1465228 - Posted: 16 Jan 2014, 22:43:37 UTC

Thanks for the update Matt. You mention a server donation. When you've finally got that sorted, any chance of calling it: Curmudgeon
____________

Profile Chris S
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 31146
Credit: 11,335,717
RAC: 20,580
United Kingdom
Message 1466211 - Posted: 19 Jan 2014, 14:02:08 UTC

Very many thanks for your post and return to the fold Matt, as always very much appreciated. This new website you refer to sounds very interesting indeed, and I very much look forward to seeing it.

I am pleased to second Sirius's suggestion that a server could be called Curmudgeon to honour the long time Setizen John Clark, who recently passed away at the end of last year. I attended his funeral, and I was privileged to be allowed to speak on behalf of his many Seti friends and colleagues around the world. It would indeed be a lovely and lasting tribute to him.

Profile Andrew Clayton
Avatar
Send message
Joined: 12 Apr 99
Posts: 7
Credit: 897,519
RAC: 80
United Kingdom
Message 1470555 - Posted: 30 Jan 2014, 0:44:46 UTC

Regarding your Linux IO problems.

I wonder if you really are seeing lock ups or just non-responsive behaviour.

Not sure the specs on that machine. But I imagine it has quite a few GBs of RAM.
In which case I wonder if your hitting the VM writeback problem.

The kernel allows a certain amount of dirty data (data that needs to be written
out to backing store) to build up in memory. In current kernels, this can be as
much as 20% of your RAM that it will need to write out and can easily block
other IO causing long stall periods where it may seem that the machine has hung.

(These defaults where made in the heady days of having a machine with as much
as 64MB of RAM!).

It's also recognised that today even having a percentage at all isn't really
suitable and thus byte limits have also been introduced (but not used by
default).

So you may want to have a look at tuning

/proc/sys/vm/dirty_background_bytes and
/proc/sys/vm/dirty_bytes

Which ever of bytes or ratio is set last is what's used.

As to what to set them, that may take a bit of trial and error to see what works
best for you.

You could start out with say (8 & 24MB), you may well need to go quite a bit
higher, but you probably want to avoid having GBs of dirty data to be writing
out.

# echo 8388608 > /proc/sys/vm/dirty_background_bytes
# echo 25165824 > /proc/sys/vm/dirty_bytes

(Stick in /etc/sysctl.conf to make it permanent)

You can watch this in action with something like

$ watch -n 5 grep -e ^Dirty: -e ^Writeback: /proc/meminfo

You'll see the dirty value build then when it reaches the limit you'll see the
writeback value ramp up.

See https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Cheers,
Andrew

Filipe
Send message
Joined: 12 Aug 00
Posts: 110
Credit: 3,796,784
RAC: 8,735
Portugal
Message 1470837 - Posted: 30 Jan 2014, 19:03:05 UTC

We've been coming up with clever ways to quickly bring our final science results down to less than 10% its original size without a minimal, perhaps zero, compromise in sensitivity.


More news on that front?

A smaller science database will open the doors to a manageable Near time persistency checker?
____________

Profile edjcox
Avatar
Send message
Joined: 20 May 99
Posts: 63
Credit: 3,970,276
RAC: 877
United States
Message 1470977 - Posted: 31 Jan 2014, 3:03:21 UTC

Currently we're using it to house all kinds of data from various other SETI projects, or as a backup/SETI@home data buffer.

Why?
____________
Never engage stupid people at their level, they then have the home court advantage.....

Profile Donald L. Johnson
Avatar
Send message
Joined: 5 Aug 02
Posts: 6015
Credit: 635,868
RAC: 1,043
United States
Message 1471848 - Posted: 2 Feb 2014, 5:10:45 UTC - in response to Message 1470977.

Currently we're using it to house all kinds of data from various other SETI projects, or as a backup/SETI@home data buffer.

Why?

Matt answered that question in the next sentence, ed:

As we get more comfortable with the system we'll push on it with more i/o intensive projects.

____________
Donald
Infernal Optimist / Submariner, retired

Tamarin Tracker
Send message
Joined: 10 Feb 01
Posts: 12
Credit: 3,465,715
RAC: 3,218
United States
Message 1507039 - Posted: 21 Apr 2014, 21:40:25 UTC

Strange, very strange. I have downloaded the latest version and now all of my work downloads fail. What is happening?
____________

1 · 2 · Next

Message boards : Technical News : Now it's 2014 (Jan 07 2014)

Copyright © 2014 University of California