Spindles (Jan 06 2011)

Message boards : Technical News : Spindles (Jan 06 2011)

Author	Message
Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 1064066 - Posted: 6 Jan 2011, 22:15:44 UTC
	The informix tweak planned yesterday was postponed and completed today. Why was it postponed? Because the weekly science backup (which happens in the background - doesn't require an outage like the mysql database) wasn't done yet. Normally it takes a few hours. But during major activity it looks like it'll take 10 days! Jeff stopped the ntpckr/rfi processes and that sped things up. This clearly points out oscar's inability to handle the crazy random i/o's we desire, though to be fair oscar is indeed operating better in its current state than the old science database. There's still MANY knobs to turn in informix-land before we need to add more disk spindles. For example, we still haven't given all the memory available in the system over to informix. The tweak we made today added an additional 20GB to the buffers. Note that it takes a bout a week to fill these buffers, so we won't notice any improvement, if any, until then. Meanwhile I've been back to working on my various ntpckr and data testing projects. It's hard to page these pieces of code back into my RAM once they've been flushed to disk - know what I mean? - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 1064066 ·

Claggy Volunteer tester Send message Joined: 5 Jul 99 Posts: 3363 Credit: 25,950,039 RAC: 1,120	Message 1064067 - Posted: 6 Jan 2011, 22:24:55 UTC - in response to Message 1064066.
	Thanks for the update Matt, Claggy
	ID: 1064067 ·

[seti.international] Philip J. Fry Volunteer tester Send message Joined: 6 Apr 07 Posts: 6568 Credit: 50,763,006 RAC: 20,309	Message 1064070 - Posted: 6 Jan 2011, 22:33:54 UTC - in response to Message 1064066.
	Matt, thanks for the news! ____________ >Das Deutsche Cafe. The German Cafe.<
	ID: 1064070 ·

Morpheus57 Volunteer tester Send message Joined: 14 Mar 03 Posts: 6 Credit: 8,747,149 RAC: 4,395	Message 1064096 - Posted: 6 Jan 2011, 23:45:05 UTC
	Matt, I do know the new servers are HP Proliants. Which Smart Array controllers does it have and how many? How are the disk arrays and logical drive defined? I have worked with Proliants for many years and could be of help setting up yours!!!! ____________
	ID: 1064096 ·

Andy Lee Robinson Send message Joined: 8 Dec 05 Posts: 615 Credit: 36,747,225 RAC: 1,996	Message 1065243 - Posted: 10 Jan 2011, 15:48:44 UTC - in response to Message 1065051.
	If returned data is not stored on magnetic tape, you may end up having numerous discs which will have to be swapped in and out depending on have many discs or drives the RAID is able to handle simultaneously. Is this correct? As I understand it, no disks are swapped in and out of the RAID set unless they fail. Drive syncing is expensive on system resources! It just looks like 1 big disk to the file system, that can be scaled to fit the requirements. Drives returning with data from Arecibo are treated exactly in the same way as tapes - as a data source, but cheaper and faster. They are mounted, read and split by the machines responsible for processing the raw data. This working data and project tracking end up on the RAID until purged, with only the results remaining, until those too are archived.
	ID: 1065243 ·

soft^spirit Send message Joined: 18 May 99 Posts: 6231 Credit: 27,513,239 RAC: 9,551	Message 1065256 - Posted: 10 Jan 2011, 16:17:49 UTC - in response to Message 1064066.
	The informix tweak planned yesterday was postponed and completed today. Why was it postponed? Because the weekly science backup (which happens in the background - doesn't require an outage like the mysql database) wasn't done yet. Normally it takes a few hours. But during major activity it looks like it'll take 10 days! Jeff stopped the ntpckr/rfi processes and that sped things up. This clearly points out oscar's inability to handle the crazy random i/o's we desire, though to be fair oscar is indeed operating better in its current state than the old science database. There's still MANY knobs to turn in informix-land before we need to add more disk spindles. For example, we still haven't given all the memory available in the system over to informix. The tweak we made today added an additional 20GB to the buffers. Note that it takes a bout a week to fill these buffers, so we won't notice any improvement, if any, until then. Meanwhile I've been back to working on my various ntpckr and data testing projects. It's hard to page these pieces of code back into my RAM once they've been flushed to disk - know what I mean? - Matt I am curious Matt, perhaps you or someone could help me understand... While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access? Yet again, just a thought. ____________ Janice
	ID: 1065256 ·

KWSN THE Holy Hand Grenade! Volunteer tester Send message Joined: 20 Dec 05 Posts: 1594 Credit: 4,424,103 RAC: 4,183	Message 1065279 - Posted: 10 Jan 2011, 17:31:41 UTC - in response to Message 1065256.
	I am curious Matt, perhaps you or someone could help me understand... While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access? Yet again, just a thought. As I understand the prices, SSD drives of sufficient size for the database are cost-prohibitive at this time... plus someone would have to donate the SSD drives, or their purchase price! ____________ .
	ID: 1065279 ·

rob smith Volunteer moderator Send message Joined: 7 Mar 03 Posts: 5449 Credit: 20,663,351 RAC: 126,800	Message 1065297 - Posted: 10 Jan 2011, 18:05:36 UTC
	A quick trawl says prices, about four times the money for one quarter the space. ____________ Bob Smith Member of Seti PIPPS (Pluto is a Planet Protest Society) Somewhere in the (un)known Universe?
	ID: 1065297 ·

soft^spirit Send message Joined: 18 May 99 Posts: 6231 Credit: 27,513,239 RAC: 9,551	Message 1065368 - Posted: 10 Jan 2011, 22:18:25 UTC - in response to Message 1065279.
	I am curious Matt, perhaps you or someone could help me understand... While the Science database is too precious to trust to SSD drives, are the access needs mostly "read" or interactive read/write.. and if mostly "read" could the data be mirrored to SSD to speed up the access? Yet again, just a thought. As I understand the prices, SSD drives of sufficient size for the database are cost-prohibitive at this time... plus someone would have to donate the SSD drives, or their purchase price! Of course I had that thought in mind. But the questions remain, would it work and would it be a good idea? I would cringe at the thought of the base science database being ONLY on such drives. So.. could they be mirrored? ____________ Janice
	ID: 1065368 ·

Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 1065392 - Posted: 10 Jan 2011, 23:41:21 UTC - in response to Message 1064096.
	Matt, I do know the new servers are HP Proliants. Which Smart Array controllers does it have and how many? How are the disk arrays and logical drive defined? I have worked with Proliants for many years and could be of help setting up yours!!!! They are Smart Array P212's - one in each server controlling 12 drives. Both systems are set up the same way: A 2-drive RAID1 mirror (for root), and an 8-drive RAID10 (for data), and 2 global hot spares. - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 1065392 ·

Morpheus57 Volunteer tester Send message Joined: 14 Mar 03 Posts: 6 Credit: 8,747,149 RAC: 4,395	Message 1065426 - Posted: 11 Jan 2011, 1:18:25 UTC
	I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O? ____________
	ID: 1065426 ·

Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 1065944 - Posted: 12 Jan 2011, 17:54:03 UTC - in response to Message 1065426.
	I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O? random I/O - that's what the killer is. When it's sequential we get 200-300MB/sec reads/writes without breaking a sweat. When it's random, it's more like maxed at at 5-10MB/sec. - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 1065944 ·

Todd Hebert Volunteer tester Send message Joined: 16 Jun 00 Posts: 647 Credit: 217,127,962 RAC: 0	Message 1065976 - Posted: 12 Jan 2011, 19:23:48 UTC
	Does it have the cache module on the board? From my experience the SmartArrary cards leave something to be desired especially when you load up the number of drives on the card and dealing with DB operations that are typically random in nature. A few of my clients have moved to LSI/Intel based cards that can use SSD's as cache with the addition of a feature key. http://www.intelraid.com/uploads_demo/Intel_RAID_SSD_Cache_with_FastPath.pdf Adaptec has a similar device too - http://www.adaptec.com/en-us/products/cloudcomputing/maxcache/ssd-cache-performance/ Here is a test example using the adaptec solution and MySQL http://www.adaptec.com/nr/rdonlyres/2767c960-2905-4695-9523-bc1e394943f4/0/applabsmysql.pdf Todd ____________
	ID: 1065976 ·

soft^spirit Send message Joined: 18 May 99 Posts: 6231 Credit: 27,513,239 RAC: 9,551	Message 1065978 - Posted: 12 Jan 2011, 19:32:28 UTC - in response to Message 1065976.
	Does it have the cache module on the board? From my experience the SmartArrary cards leave something to be desired especially when you load up the number of drives on the card and dealing with DB operations that are typically random in nature. A few of my clients have moved to LSI/Intel based cards that can use SSD's as cache with the addition of a feature key. http://www.intelraid.com/uploads_demo/Intel_RAID_SSD_Cache_with_FastPath.pdf Adaptec has a similar device too - http://www.adaptec.com/en-us/products/cloudcomputing/maxcache/ssd-cache-performance/ Here is a test example using the adaptec solution and MySQL http://www.adaptec.com/nr/rdonlyres/2767c960-2905-4695-9523-bc1e394943f4/0/applabsmysql.pdf Todd Matt: and yes we can shake the collection jar again if things like that will help. It might be close to tax time by the time we get it, but we can do it. ____________ Janice
	ID: 1065978 ·

RottenMutt Send message Joined: 15 Mar 01 Posts: 929 Credit: 182,333,794 RAC: 200,060	Message 1068853 - Posted: 21 Jan 2011, 3:01:32 UTC - in response to Message 1065944.
	I think your configuration is based on the standard 256MB onboard cache with 3.5" 1TB 7.2K rpm Hard Drives. As for DB operations are they mostly inserts, selects, or random I/O? random I/O - that's what the killer is. When it's sequential we get 200-300MB/sec reads/writes without breaking a sweat. When it's random, it's more like maxed at at 5-10MB/sec. - Matt can't you enable write cache on raid controllers with a battery backup, which would dramatically improve write performance. but i suppose we don't have real raid controller. i wouldn't enable write cache with only a ups, a server crash can lead to data loss. ____________
	ID: 1068853 ·

Message boards : Technical News : Spindles (Jan 06 2011)