Moving Right Along (Jul 30 2009)

Message boards : Technical News : Moving Right Along (Jul 30 2009)

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1441
Credit: 213,689
RAC: 0
United States
Message 922497 - Posted: 30 Jul 2009, 19:46:03 UTC

So we seem to have gotten over the hump with this new replica server. I should point out working on this server has had zero effect on the rest of the normal project operations, except for perhaps eating up all my time. Anyway, my script got around the dump/restore bug, and after some configuration headaches this morning we are successfully replicating on mork! Of course, sidious continues to be the replica we are using for production, while mork is considered "beta test."

It is catching up on the backlog far slowly than we hoped, especially given the power of the machine. Of course, power is measured in network, disk, memory and cpu. This system certainly has cpu (24 processors!) but word on the street is that mysql actually *drops* in performance after n processors. What "n" is, and what the penalty is remains unclear. Also, this system has fewer disk spindles than sidious (8 compared to 10), and they are slower disks, I think. So we may be seeing a disk i/o hit, but iostat doesn't really show anything amiss. The system is also in our lab and not in the closet, so there may be an extra network hop or two slowing things down. Anyway, as it progresses we'll gauge its performance and act accordingly.

As for changing linux flavors, the current issue here is mysql versions, and not so much linux distributions. As mentioned elsewhere we're trying to adhere to a homogenous setup, and we have less than zero time to mess around with anything experimental like trying new OSes on for size. In any case, Fedora works well enough, and while I generally swear by open source software for both philosophical and practical reasons, I do understand that you get what you pay for.

- Matt


-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ID: 922497 · Report as offensive
DJStarfox

Send message
Joined: 23 May 01
Posts: 1057
Credit: 802,388
RAC: 176
United States
Message 922499 - Posted: 30 Jul 2009, 20:11:06 UTC - in response to Message 922497.
Last modified: 30 Jul 2009, 21:04:58 UTC

Indeed, I've also heard that MySQL doesn't scale well to processor counts beyond 4. Perhaps Mork could serve a different purpose? Seems to me you'd want a 4 or 8 core system with as many disk spindles as you can get for MySQL.

Also, is Mork setup to use RAID 10, 5, or a set of RAID 1 mirrors? I read a very interesting article on how using RAID 1 mirrors (with an RDBMS that can partition tables evenly itself) gives better concurrency/performance than a big RAID 10 with the same number of spindles.

Edit: Monk would be a better name, anyway. :)

ID: 922499 · Report as offensive
Invisible Man

Send message
Joined: 24 Jun 01
Posts: 22
Credit: 1,129,336
RAC: 0
United Kingdom
Message 922506 - Posted: 30 Jul 2009, 20:52:05 UTC - in response to Message 922499.

DJStarfox: If you read other posts, it looks as though your references to "Monk" should read "Mork".


ID: 922506 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 922577 - Posted: 31 Jul 2009, 3:15:42 UTC - in response to Message 922506.

on another note; i see that yall have added some more data disk to the list....



I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

ID: 922577 · Report as offensive
Profile Byron Leigh Hatch @ team Carl Sagan
Volunteer tester
Avatar

Send message
Joined: 5 Jul 99
Posts: 3637
Credit: 14,025,554
RAC: 8,613
Canada
Message 922678 - Posted: 31 Jul 2009, 15:31:11 UTC



July 30, 2009

Open source Database Breakthrough 10 - 80 times Faster

A groud breaking database kernel is now being combined with the leading open source relational database from Ingres

The figure below shows the architecture of the new VectorWise engine. The left part shows the system architecture (“X100” execution engine and ColumnBM buffer manager) and how it maps on the computer resources (CPU cache, RAM and disk). The right part shows a query in action, having been decomposed into so-called relational operators (Aggregate, Project, Select and Scan) and execution primitives (such as summation – aggr_sum_flt_col).





A groud breaking database kernel is now being combined with the leading open source relational database from Ingres



The Ingres VectorWise project team has worked with Intel to evaluate database performance on the new Intel Xeon processor 5500 series based platform. To date, the results of the project have demonstrated dramatic cost and performance capabilities as evidenced by nearly 80 fold speed up on a query modelled after the Q1 query of TPC-H3 suite on the Intel Xeon processor.

VectorWise next-generation database technology is based on a novel query processing architecture that allows modern microprocessors and memory architectures to reach their full potential. This is a unique achievement: in detail studies that compare common computing tasks such as scientific calculation, multi-media authoring, games, and databases have consistently shown that typical database engines do not benefit from new processor performance features such as SSE, out of order execution, chip multithreading, and increasingly larger L2/L3 caches due to their large, complex legacy code structure.

The computational power that database systems provide is known to be lower than the performance realized by hand-coding the same task in a (e.g. C++) program. However, the actual performance difference can be surprisingly large: a factor 100. VectorWise has created the first database system to revert that situation, with dramatic efficiency improvements as a result


http://www.vectorwise.com/index_js.php?page=mission_technology

http://www.vectorwise.com/index_js.php?page=mission_technology


http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html

http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html




ID: 922678 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7066
Credit: 101,006,262
RAC: 65,138
Germany
Message 922725 - Posted: 31 Jul 2009, 19:01:47 UTC
Last modified: 31 Jul 2009, 19:04:16 UTC


I hope one guy from the Berkeley crew noticed that the UL server is offline..

Maybe one have time before the WE for to 'kick'..? ;-)


Maybe for the future Matt, if it's possible you could note it ~ a day before on the main site and/or in the TNews that one/some (UL/DL/scheduler) server will be diabled soon? It would help to hold down/would reduce angry posts in the 'NC panic thread'.


ID: 922725 · Report as offensive
Soggy

Send message
Joined: 28 Jan 03
Posts: 2
Credit: 4,631,709
RAC: 0
United States
Message 922768 - Posted: 31 Jul 2009, 21:02:32 UTC - in response to Message 922725.

Dito on the UL Server. Got work that won't upload since AM Eastern. Still not in the PM Eastern. Whats up with that?


ID: 922768 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 922769 - Posted: 31 Jul 2009, 21:09:54 UTC - in response to Message 922768.

perhaps we should all chill, I'm pretty sure they're working on the upload server and doing something to help increase to the performance of the network, data on the upload(Bruno)

ID: 922769 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 2578
Credit: 34,671,579
RAC: 20,437
United States
Message 922779 - Posted: 31 Jul 2009, 21:39:49 UTC
Last modified: 31 Jul 2009, 21:40:25 UTC

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?


.

ID: 922779 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 922782 - Posted: 31 Jul 2009, 21:56:02 UTC - in response to Message 922779.

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?

... and you're sure that this was simply a case of someone not noticing, and couldn't possibly have been the entire staff working all day to fix broken hardware?????

ID: 922782 · Report as offensive
Soggy

Send message
Joined: 28 Jan 03
Posts: 2
Credit: 4,631,709
RAC: 0
United States
Message 922785 - Posted: 31 Jul 2009, 22:00:36 UTC - in response to Message 922779.

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?


ID: 922785 · Report as offensive
Profile Andy Lee Robinson
Avatar

Send message
Joined: 8 Dec 05
Posts: 630
Credit: 55,952,216
RAC: 10,376
Hungary
Message 922798 - Posted: 31 Jul 2009, 23:16:26 UTC - in response to Message 922497.

Matt, if you want to use mysql to get the benefit of the 24 processors, then you'll have to install mysql 5.4 - this has a lot of enhancements written by google programmers.
With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible. Core and thread usage is less than optimal.
In Fedora my.cnf, also try setting tmpdir=/dev/shm
This greatly increases throughput of queries that have to use temp tables.

ID: 922798 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7475
Credit: 90,910,123
RAC: 45,357
Australia
Message 922800 - Posted: 31 Jul 2009, 23:35:45 UTC - in response to Message 922798.

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.
Grant
Darwin NT

ID: 922800 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7475
Credit: 90,910,123
RAC: 45,357
Australia
Message 922802 - Posted: 31 Jul 2009, 23:37:11 UTC - in response to Message 922800.

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.
Grant
Darwin NT

ID: 922802 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 922819 - Posted: 1 Aug 2009, 0:53:52 UTC - in response to Message 922802.

upload is up...

ID: 922819 · Report as offensive
Profile MarkJProject Donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 978
Credit: 33,082,562
RAC: 9,676
Australia
Message 922821 - Posted: 1 Aug 2009, 0:57:52 UTC - in response to Message 922802.

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.


I was looking at the price of them. They are too expensive compared to the M version. You pay almost double the price of the M and get less than half the capacity.
BOINC blog

ID: 922821 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7475
Credit: 90,910,123
RAC: 45,357
Australia
Message 922836 - Posted: 1 Aug 2009, 2:02:42 UTC - in response to Message 922821.

I was looking at the price of them. They are too expensive compared to the M version. You pay almost double the price of the M and get less than half the capacity.

That's the disadvanatge of MLC (Multi Level Cell) compared to SLC (Single Level Cell) SSDs. But under enterprise workloads the MLC drives would have a pretty short lifespan, hence the need for SLC in such instances.
Grant
Darwin NT

ID: 922836 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 900
Credit: 7,846,588
RAC: 1,027
New Zealand
Message 923058 - Posted: 1 Aug 2009, 23:23:34 UTC - in response to Message 922577.

on another note; i see that yall have added some more data disk to the list....

I noticed this as well, nice work. Maybe Matt got his software up & running, so we can crunch tapes from ages ago? Thanks team for all the hard work.

ID: 923058 · Report as offensive
Chris Campbell

Send message
Joined: 14 May 99
Posts: 17
Credit: 3,267,358
RAC: 1,732
United States
Message 923104 - Posted: 2 Aug 2009, 4:18:19 UTC - in response to Message 922497.

Matt -

I see that you (or whoever) have now linked to the 10th anniversary videos from the home page.

It would be nice to have Eric's presentation, because the camera work during his segment was ... regrettable.

Please beg Eric for his presentation file (exported to PDF?) and post it alongside his video on the 10yr page. I've stated the full case for this here.

Thank you and everyone at S@H for your efforts!

- Chris


Running S@H since Day 1 -- May 14th, 1999!

ID: 923104 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 13305
Credit: 154,259,675
RAC: 114,015
United Kingdom
Message 923117 - Posted: 2 Aug 2009, 8:07:25 UTC - in response to Message 922785.

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?





Errr- What's "hair"???


Unknown concept at this location
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

ID: 923117 · Report as offensive

Message boards : Technical News : Moving Right Along (Jul 30 2009)


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.