Spindles (Jan 06 2011)

Message boards : Technical News : Spindles (Jan 06 2011)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1064066 - Posted: 6 Jan 2011, 22:15:44 UTC

The informix tweak planned yesterday was postponed and completed today. Why was it postponed? Because the weekly science backup (which happens in the background - doesn't require an outage like the mysql database) wasn't done yet. Normally it takes a few hours. But during major activity it looks like it'll take 10 days! Jeff stopped the ntpckr/rfi processes and that sped things up.

This clearly points out oscar's inability to handle the crazy random i/o's we desire, though to be fair oscar is indeed operating better in its current state than the old science database. There's still MANY knobs to turn in informix-land before we need to add more disk spindles. For example, we still haven't given all the memory available in the system over to informix. The tweak we made today added an additional 20GB to the buffers. Note that it takes a bout a week to fill these buffers, so we won't notice any improvement, if any, until then.

Meanwhile I've been back to working on my various ntpckr and data testing projects. It's hard to page these pieces of code back into my RAM once they've been flushed to disk - know what I mean?

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1064066 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1064067 - Posted: 6 Jan 2011, 22:24:55 UTC - in response to Message 1064066.  

Thanks for the update Matt,

Claggy
ID: 1064067 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 1064070 - Posted: 6 Jan 2011, 22:33:54 UTC - in response to Message 1064066.  

Matt, thanks for the news!

ID: 1064070 · Report as offensive
Bad Spartan
Volunteer tester
Avatar

Send message
Joined: 14 Mar 03
Posts: 6
Credit: 47,663,261
RAC: 117
Puerto Rico
Message 1064096 - Posted: 6 Jan 2011, 23:45:05 UTC

Matt, I do know the new servers are HP Proliants. Which Smart Array controllers does it have and how many? How are the disk arrays and logical drive defined? I have worked with Proliants for many years and could be of help setting up yours!!!!
ID: 1064096 · Report as offensive
musicplayer

Send message
Joined: 17 May 10
Posts: 2430
Credit: 926,046
RAC: 0
Message 1065051 - Posted: 9 Jan 2011, 21:12:11 UTC

I was wondering about the disc space capacities on the new servers.

Is it enough? I was able to read that the new specs went for a RAID of
12 1 TB SATA-discs.

In earlier days magnetic tape was used to store large amounts of data which
was meant for processing. Data went to / from the tapes to discs. Today 1 TB as well as 2 TB SATA-discs are available.

If returned data is not stored on magnetic tape, you may end up having numerous discs which will have to be swapped in and out depending on have many discs or drives the RAID is able to handle simultaneously.

Is this correct?
ID: 1065051 · Report as offensive
Profile Andy Lee Robinson
Avatar

Send message
Joined: 8 Dec 05
Posts: 630
Credit: 59,973,836
RAC: 0
Hungary
Message 1065243 - Posted: 10 Jan 2011, 15:48:44 UTC - in response to Message 1065051.  

If returned data is not stored on magnetic tape, you may end up having numerous discs which will have to be swapped in and out depending on have many discs or drives the RAID is able to handle simultaneously.

Is this correct?


As I understand it, no disks are swapped in and out of the RAID set unless they fail. Drive syncing is expensive on system resources! It just looks like 1 big disk to the file system, that can be scaled to fit the requirements.

Drives returning with data from Arecibo are treated exactly in the same way as tapes - as a data source, but cheaper and faster.
They are mounted, read and split by the machines responsible for processing the raw data.
This working data and project tracking end up on the RAID until purged, with only the results remaining, until those too are archived.
ID: 1065243 · Report as offensive
Profile soft^spirit
Avatar

Send message
Joined: 18 May 99
Posts: 6497
Credit: 34,134,168
RAC: 0
United States
Message 1065256 - Posted: 10 Jan 2011, 16:17:49 UTC - in response to Message 1064066.  

The informix tweak planned yesterday was postponed and completed today. Why was it postponed? Because the weekly science backup (which happens in the background - doesn't require an outage like the mysql database) wasn't done yet. Normally it takes a few hours. But during major activity it looks like it'll take 10 days! Jeff stopped the ntpckr/rfi processes and that sped things up.

This clearly points out oscar's inability to handle the crazy random i/o's we desire, though to be fair oscar is indeed operating better in its current state than the old science database. There's still MANY knobs to turn in informix-land before we need to add more disk spindles. For example, we still haven't given all the memory available in the system over to informix. The tweak we made today added an additional 20GB to the buffers. Note that it takes a bout a week to fill these buffers, so we won't notice any improvement, if any, until then.

Meanwhile I've been back to working on my various ntpckr and data testing projects. It's hard to page these pieces of code back into my RAM once they've been flushed to disk - know what I mean?

- Matt


I am curious Matt, perhaps you or someone could help me understand...

While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access?

Yet again, just a thought.
Janice
ID: 1065256 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 3187
Credit: 57,163,290
RAC: 0
United States
Message 1065279 - Posted: 10 Jan 2011, 17:31:41 UTC - in response to Message 1065256.  



I am curious Matt, perhaps you or someone could help me understand...

While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access?

Yet again, just a thought.


As I understand the prices, SSD drives of sufficient size for the database are cost-prohibitive at this time... plus someone would have to donate the SSD drives, or their purchase price!
.

Hello, from Albany, CA!...
ID: 1065279 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1065297 - Posted: 10 Jan 2011, 18:05:36 UTC

A quick trawl says prices, about four times the money for one quarter the space.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1065297 · Report as offensive
Profile soft^spirit
Avatar

Send message
Joined: 18 May 99
Posts: 6497
Credit: 34,134,168
RAC: 0
United States
Message 1065368 - Posted: 10 Jan 2011, 22:18:25 UTC - in response to Message 1065279.  



I am curious Matt, perhaps you or someone could help me understand...

While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access?

Yet again, just a thought.


As I understand the prices, SSD drives of sufficient size for the database are cost-prohibitive at this time... plus someone would have to donate the SSD drives, or their purchase price!


Of course I had that thought in mind. But the questions remain, would it work and would it be a good idea?

I would cringe at the thought of the base science database being ONLY on such drives. So.. could they be mirrored?
Janice
ID: 1065368 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1065392 - Posted: 10 Jan 2011, 23:41:21 UTC - in response to Message 1064096.  

Matt, I do know the new servers are HP Proliants. Which Smart Array controllers does it have and how many? How are the disk arrays and logical drive defined? I have worked with Proliants for many years and could be of help setting up yours!!!!


They are Smart Array P212's - one in each server controlling 12 drives. Both systems are set up the same way: A 2-drive RAID1 mirror (for root), and an 8-drive RAID10 (for data), and 2 global hot spares.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1065392 · Report as offensive
Bad Spartan
Volunteer tester
Avatar

Send message
Joined: 14 Mar 03
Posts: 6
Credit: 47,663,261
RAC: 117
Puerto Rico
Message 1065426 - Posted: 11 Jan 2011, 1:18:25 UTC

I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O?
ID: 1065426 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1065944 - Posted: 12 Jan 2011, 17:54:03 UTC - in response to Message 1065426.  

I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O?


random I/O - that's what the killer is. When it's sequential we get 200-300MB/sec reads/writes without breaking a sweat. When it's random, it's more like maxed at at 5-10MB/sec.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1065944 · Report as offensive
Profile Todd Hebert
Volunteer tester
Avatar

Send message
Joined: 16 Jun 00
Posts: 648
Credit: 228,292,957
RAC: 0
United States
Message 1065976 - Posted: 12 Jan 2011, 19:23:48 UTC

Does it have the cache module on the board? From my experience the SmartArrary cards leave something to be desired especially when you load up the number of drives on the card and dealing with DB operations that are typically random in nature.

A few of my clients have moved to LSI/Intel based cards that can use SSD's as cache with the addition of a feature key. http://www.intelraid.com/uploads_demo/Intel_RAID_SSD_Cache_with_FastPath.pdf

Adaptec has a similar device too - http://www.adaptec.com/en-us/products/cloudcomputing/maxcache/ssd-cache-performance/

Here is a test example using the adaptec solution and MySQL
http://www.adaptec.com/nr/rdonlyres/2767c960-2905-4695-9523-bc1e394943f4/0/applabsmysql.pdf

Todd
ID: 1065976 · Report as offensive
Profile soft^spirit
Avatar

Send message
Joined: 18 May 99
Posts: 6497
Credit: 34,134,168
RAC: 0
United States
Message 1065978 - Posted: 12 Jan 2011, 19:32:28 UTC - in response to Message 1065976.  

Does it have the cache module on the board? From my experience the SmartArrary cards leave something to be desired especially when you load up the number of drives on the card and dealing with DB operations that are typically random in nature.

A few of my clients have moved to LSI/Intel based cards that can use SSD's as cache with the addition of a feature key. http://www.intelraid.com/uploads_demo/Intel_RAID_SSD_Cache_with_FastPath.pdf

Adaptec has a similar device too - http://www.adaptec.com/en-us/products/cloudcomputing/maxcache/ssd-cache-performance/

Here is a test example using the adaptec solution and MySQL
http://www.adaptec.com/nr/rdonlyres/2767c960-2905-4695-9523-bc1e394943f4/0/applabsmysql.pdf

Todd


Matt: and yes we can shake the collection jar again if things like that will help. It might be close to tax time by the time we get it, but we can do it.

Janice
ID: 1065978 · Report as offensive
Profile RottenMutt
Avatar

Send message
Joined: 15 Mar 01
Posts: 1011
Credit: 230,314,058
RAC: 0
United States
Message 1068853 - Posted: 21 Jan 2011, 3:01:32 UTC - in response to Message 1065944.  

I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O?


random I/O - that's what the killer is. When it's sequential we get 200-300MB/sec reads/writes without breaking a sweat. When it's random, it's more like maxed at at 5-10MB/sec.

- Matt


can't you enable write cache on raid controllers with a battery backup, which would dramatically improve write performance.

but i suppose we don't have real raid controller. i wouldn't enable write cache with only a ups, a server crash can lead to data loss.
ID: 1068853 · Report as offensive

Message boards : Technical News : Spindles (Jan 06 2011)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.