Moving Right Along (Jul 30 2009)


log in

Advanced search

Message boards : Technical News : Moving Right Along (Jul 30 2009)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 922497 - Posted: 30 Jul 2009, 19:46:03 UTC

So we seem to have gotten over the hump with this new replica server. I should point out working on this server has had zero effect on the rest of the normal project operations, except for perhaps eating up all my time. Anyway, my script got around the dump/restore bug, and after some configuration headaches this morning we are successfully replicating on mork! Of course, sidious continues to be the replica we are using for production, while mork is considered "beta test."

It is catching up on the backlog far slowly than we hoped, especially given the power of the machine. Of course, power is measured in network, disk, memory and cpu. This system certainly has cpu (24 processors!) but word on the street is that mysql actually *drops* in performance after n processors. What "n" is, and what the penalty is remains unclear. Also, this system has fewer disk spindles than sidious (8 compared to 10), and they are slower disks, I think. So we may be seeing a disk i/o hit, but iostat doesn't really show anything amiss. The system is also in our lab and not in the closet, so there may be an extra network hop or two slowing things down. Anyway, as it progresses we'll gauge its performance and act accordingly.

As for changing linux flavors, the current issue here is mysql versions, and not so much linux distributions. As mentioned elsewhere we're trying to adhere to a homogenous setup, and we have less than zero time to mess around with anything experimental like trying new OSes on for size. In any case, Fedora works well enough, and while I generally swear by open source software for both philosophical and practical reasons, I do understand that you get what you pay for.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

DJStarfox
Send message
Joined: 23 May 01
Posts: 1045
Credit: 561,441
RAC: 371
United States
Message 922499 - Posted: 30 Jul 2009, 20:11:06 UTC - in response to Message 922497.
Last modified: 30 Jul 2009, 21:04:58 UTC

Indeed, I've also heard that MySQL doesn't scale well to processor counts beyond 4. Perhaps Mork could serve a different purpose? Seems to me you'd want a 4 or 8 core system with as many disk spindles as you can get for MySQL.

Also, is Mork setup to use RAID 10, 5, or a set of RAID 1 mirrors? I read a very interesting article on how using RAID 1 mirrors (with an RDBMS that can partition tables evenly itself) gives better concurrency/performance than a big RAID 10 with the same number of spindles.

Edit: Monk would be a better name, anyway. :)

Invisible Man
Send message
Joined: 24 Jun 01
Posts: 22
Credit: 1,129,336
RAC: 0
United Kingdom
Message 922506 - Posted: 30 Jul 2009, 20:52:05 UTC - in response to Message 922499.

DJStarfox: If you read other posts, it looks as though your references to "Monk" should read "Mork".
____________

zpm
Volunteer tester
Avatar
Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,602,404
RAC: 162
United States
Message 922577 - Posted: 31 Jul 2009, 3:15:42 UTC - in response to Message 922506.

on another note; i see that yall have added some more data disk to the list....
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

Profile Byron Leigh Hatch @ team Carl SaganProject donor
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 3619
Credit: 11,916,794
RAC: 1,068
Canada
Message 922678 - Posted: 31 Jul 2009, 15:31:11 UTC



July 30, 2009

Open source Database Breakthrough 10 - 80 times Faster

A groud breaking database kernel is now being combined with the leading open source relational database from Ingres

The figure below shows the architecture of the new VectorWise engine. The left part shows the system architecture (“X100” execution engine and ColumnBM buffer manager) and how it maps on the computer resources (CPU cache, RAM and disk). The right part shows a query in action, having been decomposed into so-called relational operators (Aggregate, Project, Select and Scan) and execution primitives (such as summation – aggr_sum_flt_col).





A groud breaking database kernel is now being combined with the leading open source relational database from Ingres



The Ingres VectorWise project team has worked with Intel to evaluate database performance on the new Intel Xeon processor 5500 series based platform. To date, the results of the project have demonstrated dramatic cost and performance capabilities as evidenced by nearly 80 fold speed up on a query modelled after the Q1 query of TPC-H3 suite on the Intel Xeon processor.

VectorWise next-generation database technology is based on a novel query processing architecture that allows modern microprocessors and memory architectures to reach their full potential. This is a unique achievement: in detail studies that compare common computing tasks such as scientific calculation, multi-media authoring, games, and databases have consistently shown that typical database engines do not benefit from new processor performance features such as SSE, out of order execution, chip multithreading, and increasingly larger L2/L3 caches due to their large, complex legacy code structure.

The computational power that database systems provide is known to be lower than the performance realized by hand-coding the same task in a (e.g. C++) program. However, the actual performance difference can be surprisingly large: a factor 100. VectorWise has created the first database system to revert that situation, with dramatic efficiency improvements as a result


http://www.vectorwise.com/index_js.php?page=mission_technology

http://www.vectorwise.com/index_js.php?page=mission_technology


http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html

http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html




____________

Profile [seti.international] Dirk SadowskiProject donor
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 61,046,833
RAC: 18,269
Germany
Message 922725 - Posted: 31 Jul 2009, 19:01:47 UTC
Last modified: 31 Jul 2009, 19:04:16 UTC


I hope one guy from the Berkeley crew noticed that the UL server is offline..

Maybe one have time before the WE for to 'kick'..? ;-)


Maybe for the future Matt, if it's possible you could note it ~ a day before on the main site and/or in the TNews that one/some (UL/DL/scheduler) server will be diabled soon? It would help to hold down/would reduce angry posts in the 'NC panic thread'.

____________
BR

SETI@home Needs your Help ... $10 & U get a Star!

Team seti.international

Das Deutsche Cafe. The German Cafe.

Soggy
Send message
Joined: 28 Jan 03
Posts: 2
Credit: 4,614,619
RAC: 4,393
United States
Message 922768 - Posted: 31 Jul 2009, 21:02:32 UTC - in response to Message 922725.

Dito on the UL Server. Got work that won't upload since AM Eastern. Still not in the PM Eastern. Whats up with that?
____________

zpm
Volunteer tester
Avatar
Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,602,404
RAC: 162
United States
Message 922769 - Posted: 31 Jul 2009, 21:09:54 UTC - in response to Message 922768.

perhaps we should all chill, I'm pretty sure they're working on the upload server and doing something to help increase to the performance of the network, data on the upload(Bruno)

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1965
Credit: 10,602,789
RAC: 14,215
United States
Message 922779 - Posted: 31 Jul 2009, 21:39:49 UTC
Last modified: 31 Jul 2009, 21:40:25 UTC

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?
____________
.

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 922782 - Posted: 31 Jul 2009, 21:56:02 UTC - in response to Message 922779.

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?

... and you're sure that this was simply a case of someone not noticing, and couldn't possibly have been the entire staff working all day to fix broken hardware?????
____________

Soggy
Send message
Joined: 28 Jan 03
Posts: 2
Credit: 4,614,619
RAC: 4,393
United States
Message 922785 - Posted: 31 Jul 2009, 22:00:36 UTC - in response to Message 922779.

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?
____________

Profile Andy Lee Robinson
Avatar
Send message
Joined: 8 Dec 05
Posts: 615
Credit: 42,867,858
RAC: 27,082
Hungary
Message 922798 - Posted: 31 Jul 2009, 23:16:26 UTC - in response to Message 922497.

Matt, if you want to use mysql to get the benefit of the 24 processors, then you'll have to install mysql 5.4 - this has a lot of enhancements written by google programmers.
With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible. Core and thread usage is less than optimal.
In Fedora my.cnf, also try setting tmpdir=/dev/shm
This greatly increases throughput of queries that have to use temp tables.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5872
Credit: 60,877,761
RAC: 47,409
Australia
Message 922800 - Posted: 31 Jul 2009, 23:35:45 UTC - in response to Message 922798.

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.
____________
Grant
Darwin NT.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5872
Credit: 60,877,761
RAC: 47,409
Australia
Message 922802 - Posted: 31 Jul 2009, 23:37:11 UTC - in response to Message 922800.

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.
____________
Grant
Darwin NT.

zpm
Volunteer tester
Avatar
Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,602,404
RAC: 162
United States
Message 922819 - Posted: 1 Aug 2009, 0:53:52 UTC - in response to Message 922802.

upload is up...

Profile MarkJProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 08
Posts: 942
Credit: 25,131,488
RAC: 15,176
Australia
Message 922821 - Posted: 1 Aug 2009, 0:57:52 UTC - in response to Message 922802.

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.


I was looking at the price of them. They are too expensive compared to the M version. You pay almost double the price of the M and get less than half the capacity.
____________
BOINC blog

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5872
Credit: 60,877,761
RAC: 47,409
Australia
Message 922836 - Posted: 1 Aug 2009, 2:02:42 UTC - in response to Message 922821.

I was looking at the price of them. They are too expensive compared to the M version. You pay almost double the price of the M and get less than half the capacity.

That's the disadvanatge of MLC (Multi Level Cell) compared to SLC (Single Level Cell) SSDs. But under enterprise workloads the MLC drives would have a pretty short lifespan, hence the need for SLC in such instances.
____________
Grant
Darwin NT.

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 680
Credit: 5,936,843
RAC: 2,914
New Zealand
Message 923058 - Posted: 1 Aug 2009, 23:23:34 UTC - in response to Message 922577.

on another note; i see that yall have added some more data disk to the list....

I noticed this as well, nice work. Maybe Matt got his software up & running, so we can crunch tapes from ages ago? Thanks team for all the hard work.
____________

Live in NZ y not join Smile City?

Chris Campbell
Send message
Joined: 14 May 99
Posts: 17
Credit: 2,339,543
RAC: 597
United States
Message 923104 - Posted: 2 Aug 2009, 4:18:19 UTC - in response to Message 922497.

Matt -

I see that you (or whoever) have now linked to the 10th anniversary videos from the home page.

It would be nice to have Eric's presentation, because the camera work during his segment was ... regrettable.

Please beg Eric for his presentation file (exported to PDF?) and post it alongside his video on the 10yr page. I've stated the full case for this here.

Thank you and everyone at S@H for your efforts!

- Chris

____________
Running S@H since Day 1 -- May 14th, 1999!

rob smithProject donor
Volunteer tester
Send message
Joined: 7 Mar 03
Posts: 8559
Credit: 60,061,248
RAC: 93,795
United Kingdom
Message 923117 - Posted: 2 Aug 2009, 8:07:25 UTC - in response to Message 922785.

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?





Errr- What's "hair"???


Unknown concept at this location
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Message boards : Technical News : Moving Right Along (Jul 30 2009)

Copyright © 2014 University of California