Spindles (Jan 06 2011)


log in

Advanced search

Message boards : Technical News : Spindles (Jan 06 2011)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1064066 - Posted: 6 Jan 2011, 22:15:44 UTC

The informix tweak planned yesterday was postponed and completed today. Why was it postponed? Because the weekly science backup (which happens in the background - doesn't require an outage like the mysql database) wasn't done yet. Normally it takes a few hours. But during major activity it looks like it'll take 10 days! Jeff stopped the ntpckr/rfi processes and that sped things up.

This clearly points out oscar's inability to handle the crazy random i/o's we desire, though to be fair oscar is indeed operating better in its current state than the old science database. There's still MANY knobs to turn in informix-land before we need to add more disk spindles. For example, we still haven't given all the memory available in the system over to informix. The tweak we made today added an additional 20GB to the buffers. Note that it takes a bout a week to fill these buffers, so we won't notice any improvement, if any, until then.

Meanwhile I've been back to working on my various ntpckr and data testing projects. It's hard to page these pieces of code back into my RAM once they've been flushed to disk - know what I mean?

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4139
Credit: 33,482,302
RAC: 21,999
United Kingdom
Message 1064067 - Posted: 6 Jan 2011, 22:24:55 UTC - in response to Message 1064066.

Thanks for the update Matt,

Claggy

Profile [seti.international] Dirk SadowskiProject donor
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7101
Credit: 60,875,418
RAC: 16,815
Germany
Message 1064070 - Posted: 6 Jan 2011, 22:33:54 UTC - in response to Message 1064066.

Matt, thanks for the news!

____________
BR

SETI@home Needs your Help ... $10 & U get a Star!

Team seti.international

Das Deutsche Cafe. The German Cafe.

Morpheus57
Volunteer tester
Avatar
Send message
Joined: 14 Mar 03
Posts: 6
Credit: 11,082,326
RAC: 5,106
Puerto Rico
Message 1064096 - Posted: 6 Jan 2011, 23:45:05 UTC

Matt, I do know the new servers are HP Proliants. Which Smart Array controllers does it have and how many? How are the disk arrays and logical drive defined? I have worked with Proliants for many years and could be of help setting up yours!!!!
____________

musicplayer
Send message
Joined: 17 May 10
Posts: 1457
Credit: 707,032
RAC: 635
Message 1065051 - Posted: 9 Jan 2011, 21:12:11 UTC

I was wondering about the disc space capacities on the new servers.

Is it enough? I was able to read that the new specs went for a RAID of
12 1 TB SATA-discs.

In earlier days magnetic tape was used to store large amounts of data which
was meant for processing. Data went to / from the tapes to discs. Today 1 TB as well as 2 TB SATA-discs are available.

If returned data is not stored on magnetic tape, you may end up having numerous discs which will have to be swapped in and out depending on have many discs or drives the RAID is able to handle simultaneously.

Is this correct?

Profile Andy Lee Robinson
Avatar
Send message
Joined: 8 Dec 05
Posts: 615
Credit: 42,581,501
RAC: 23,270
Hungary
Message 1065243 - Posted: 10 Jan 2011, 15:48:44 UTC - in response to Message 1065051.

If returned data is not stored on magnetic tape, you may end up having numerous discs which will have to be swapped in and out depending on have many discs or drives the RAID is able to handle simultaneously.

Is this correct?


As I understand it, no disks are swapped in and out of the RAID set unless they fail. Drive syncing is expensive on system resources! It just looks like 1 big disk to the file system, that can be scaled to fit the requirements.

Drives returning with data from Arecibo are treated exactly in the same way as tapes - as a data source, but cheaper and faster.
They are mounted, read and split by the machines responsible for processing the raw data.
This working data and project tracking end up on the RAID until purged, with only the results remaining, until those too are archived.

Profile soft^spirit
Avatar
Send message
Joined: 18 May 99
Posts: 6374
Credit: 28,631,148
RAC: 2
United States
Message 1065256 - Posted: 10 Jan 2011, 16:17:49 UTC - in response to Message 1064066.

The informix tweak planned yesterday was postponed and completed today. Why was it postponed? Because the weekly science backup (which happens in the background - doesn't require an outage like the mysql database) wasn't done yet. Normally it takes a few hours. But during major activity it looks like it'll take 10 days! Jeff stopped the ntpckr/rfi processes and that sped things up.

This clearly points out oscar's inability to handle the crazy random i/o's we desire, though to be fair oscar is indeed operating better in its current state than the old science database. There's still MANY knobs to turn in informix-land before we need to add more disk spindles. For example, we still haven't given all the memory available in the system over to informix. The tweak we made today added an additional 20GB to the buffers. Note that it takes a bout a week to fill these buffers, so we won't notice any improvement, if any, until then.

Meanwhile I've been back to working on my various ntpckr and data testing projects. It's hard to page these pieces of code back into my RAM once they've been flushed to disk - know what I mean?

- Matt


I am curious Matt, perhaps you or someone could help me understand...

While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access?

Yet again, just a thought.
____________

Janice

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1958
Credit: 10,431,289
RAC: 7,902
United States
Message 1065279 - Posted: 10 Jan 2011, 17:31:41 UTC - in response to Message 1065256.



I am curious Matt, perhaps you or someone could help me understand...

While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access?

Yet again, just a thought.


As I understand the prices, SSD drives of sufficient size for the database are cost-prohibitive at this time... plus someone would have to donate the SSD drives, or their purchase price!
____________
.

rob smithProject donor
Volunteer tester
Send message
Joined: 7 Mar 03
Posts: 8526
Credit: 59,116,555
RAC: 78,849
United Kingdom
Message 1065297 - Posted: 10 Jan 2011, 18:05:36 UTC

A quick trawl says prices, about four times the money for one quarter the space.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Profile soft^spirit
Avatar
Send message
Joined: 18 May 99
Posts: 6374
Credit: 28,631,148
RAC: 2
United States
Message 1065368 - Posted: 10 Jan 2011, 22:18:25 UTC - in response to Message 1065279.



I am curious Matt, perhaps you or someone could help me understand...

While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access?

Yet again, just a thought.


As I understand the prices, SSD drives of sufficient size for the database are cost-prohibitive at this time... plus someone would have to donate the SSD drives, or their purchase price!


Of course I had that thought in mind. But the questions remain, would it work and would it be a good idea?

I would cringe at the thought of the base science database being ONLY on such drives. So.. could they be mirrored?
____________

Janice

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1065392 - Posted: 10 Jan 2011, 23:41:21 UTC - in response to Message 1064096.

Matt, I do know the new servers are HP Proliants. Which Smart Array controllers does it have and how many? How are the disk arrays and logical drive defined? I have worked with Proliants for many years and could be of help setting up yours!!!!


They are Smart Array P212's - one in each server controlling 12 drives. Both systems are set up the same way: A 2-drive RAID1 mirror (for root), and an 8-drive RAID10 (for data), and 2 global hot spares.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Morpheus57
Volunteer tester
Avatar
Send message
Joined: 14 Mar 03
Posts: 6
Credit: 11,082,326
RAC: 5,106
Puerto Rico
Message 1065426 - Posted: 11 Jan 2011, 1:18:25 UTC

I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O?
____________

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1065944 - Posted: 12 Jan 2011, 17:54:03 UTC - in response to Message 1065426.

I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O?


random I/O - that's what the killer is. When it's sequential we get 200-300MB/sec reads/writes without breaking a sweat. When it's random, it's more like maxed at at 5-10MB/sec.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Todd Hebert
Volunteer tester
Avatar
Send message
Joined: 16 Jun 00
Posts: 647
Credit: 217,127,962
RAC: 0
United States
Message 1065976 - Posted: 12 Jan 2011, 19:23:48 UTC

Does it have the cache module on the board? From my experience the SmartArrary cards leave something to be desired especially when you load up the number of drives on the card and dealing with DB operations that are typically random in nature.

A few of my clients have moved to LSI/Intel based cards that can use SSD's as cache with the addition of a feature key. http://www.intelraid.com/uploads_demo/Intel_RAID_SSD_Cache_with_FastPath.pdf

Adaptec has a similar device too - http://www.adaptec.com/en-us/products/cloudcomputing/maxcache/ssd-cache-performance/

Here is a test example using the adaptec solution and MySQL
http://www.adaptec.com/nr/rdonlyres/2767c960-2905-4695-9523-bc1e394943f4/0/applabsmysql.pdf

Todd
____________

Profile soft^spirit
Avatar
Send message
Joined: 18 May 99
Posts: 6374
Credit: 28,631,148
RAC: 2
United States
Message 1065978 - Posted: 12 Jan 2011, 19:32:28 UTC - in response to Message 1065976.

Does it have the cache module on the board? From my experience the SmartArrary cards leave something to be desired especially when you load up the number of drives on the card and dealing with DB operations that are typically random in nature.

A few of my clients have moved to LSI/Intel based cards that can use SSD's as cache with the addition of a feature key. http://www.intelraid.com/uploads_demo/Intel_RAID_SSD_Cache_with_FastPath.pdf

Adaptec has a similar device too - http://www.adaptec.com/en-us/products/cloudcomputing/maxcache/ssd-cache-performance/

Here is a test example using the adaptec solution and MySQL
http://www.adaptec.com/nr/rdonlyres/2767c960-2905-4695-9523-bc1e394943f4/0/applabsmysql.pdf

Todd


Matt: and yes we can shake the collection jar again if things like that will help. It might be close to tax time by the time we get it, but we can do it.

____________

Janice

Profile RottenMutt
Avatar
Send message
Joined: 15 Mar 01
Posts: 992
Credit: 207,654,737
RAC: 0
United States
Message 1068853 - Posted: 21 Jan 2011, 3:01:32 UTC - in response to Message 1065944.

I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O?


random I/O - that's what the killer is. When it's sequential we get 200-300MB/sec reads/writes without breaking a sweat. When it's random, it's more like maxed at at 5-10MB/sec.

- Matt


can't you enable write cache on raid controllers with a battery backup, which would dramatically improve write performance.

but i suppose we don't have real raid controller. i wouldn't enable write cache with only a ups, a server crash can lead to data loss.
____________

Message boards : Technical News : Spindles (Jan 06 2011)

Copyright © 2014 University of California