Moving Right Along (Jul 30 2009)

Message boards : Technical News : Moving Right Along (Jul 30 2009)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 922497 - Posted: 30 Jul 2009, 19:46:03 UTC

So we seem to have gotten over the hump with this new replica server. I should point out working on this server has had zero effect on the rest of the normal project operations, except for perhaps eating up all my time. Anyway, my script got around the dump/restore bug, and after some configuration headaches this morning we are successfully replicating on mork! Of course, sidious continues to be the replica we are using for production, while mork is considered "beta test."

It is catching up on the backlog far slowly than we hoped, especially given the power of the machine. Of course, power is measured in network, disk, memory and cpu. This system certainly has cpu (24 processors!) but word on the street is that mysql actually *drops* in performance after n processors. What "n" is, and what the penalty is remains unclear. Also, this system has fewer disk spindles than sidious (8 compared to 10), and they are slower disks, I think. So we may be seeing a disk i/o hit, but iostat doesn't really show anything amiss. The system is also in our lab and not in the closet, so there may be an extra network hop or two slowing things down. Anyway, as it progresses we'll gauge its performance and act accordingly.

As for changing linux flavors, the current issue here is mysql versions, and not so much linux distributions. As mentioned elsewhere we're trying to adhere to a homogenous setup, and we have less than zero time to mess around with anything experimental like trying new OSes on for size. In any case, Fedora works well enough, and while I generally swear by open source software for both philosophical and practical reasons, I do understand that you get what you pay for.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 922497 · Report as offensive
DJStarfox

Send message
Joined: 23 May 01
Posts: 1066
Credit: 1,226,053
RAC: 2
United States
Message 922499 - Posted: 30 Jul 2009, 20:11:06 UTC - in response to Message 922497.  
Last modified: 30 Jul 2009, 21:04:58 UTC

Indeed, I've also heard that MySQL doesn't scale well to processor counts beyond 4. Perhaps Mork could serve a different purpose? Seems to me you'd want a 4 or 8 core system with as many disk spindles as you can get for MySQL.

Also, is Mork setup to use RAID 10, 5, or a set of RAID 1 mirrors? I read a very interesting article on how using RAID 1 mirrors (with an RDBMS that can partition tables evenly itself) gives better concurrency/performance than a big RAID 10 with the same number of spindles.

Edit: Monk would be a better name, anyway. :)
ID: 922499 · Report as offensive
Invisible Man

Send message
Joined: 24 Jun 01
Posts: 22
Credit: 1,129,336
RAC: 0
United Kingdom
Message 922506 - Posted: 30 Jul 2009, 20:52:05 UTC - in response to Message 922499.  

DJStarfox: If you read other posts, it looks as though your references to "Monk" should read "Mork".
ID: 922506 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 922577 - Posted: 31 Jul 2009, 3:15:42 UTC - in response to Message 922506.  

on another note; i see that yall have added some more data disk to the list....

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.
ID: 922577 · Report as offensive
Profile Byron Leigh Hatch @ team Carl Sagan
Volunteer tester
Avatar

Send message
Joined: 5 Jul 99
Posts: 4548
Credit: 35,667,570
RAC: 4
Canada
Message 922678 - Posted: 31 Jul 2009, 15:31:11 UTC



July 30, 2009

Open source Database Breakthrough 10 - 80 times Faster

A groud breaking database kernel is now being combined with the leading open source relational database from Ingres

The figure below shows the architecture of the new VectorWise engine. The left part shows the system architecture (“X100” execution engine and ColumnBM buffer manager) and how it maps on the computer resources (CPU cache, RAM and disk). The right part shows a query in action, having been decomposed into so-called relational operators (Aggregate, Project, Select and Scan) and execution primitives (such as summation – aggr_sum_flt_col).





A groud breaking database kernel is now being combined with the leading open source relational database from Ingres



The Ingres VectorWise project team has worked with Intel to evaluate database performance on the new Intel Xeon processor 5500 series based platform. To date, the results of the project have demonstrated dramatic cost and performance capabilities as evidenced by nearly 80 fold speed up on a query modelled after the Q1 query of TPC-H3 suite on the Intel Xeon processor.

VectorWise next-generation database technology is based on a novel query processing architecture that allows modern microprocessors and memory architectures to reach their full potential. This is a unique achievement: in detail studies that compare common computing tasks such as scientific calculation, multi-media authoring, games, and databases have consistently shown that typical database engines do not benefit from new processor performance features such as SSE, out of order execution, chip multithreading, and increasingly larger L2/L3 caches due to their large, complex legacy code structure.

The computational power that database systems provide is known to be lower than the performance realized by hand-coding the same task in a (e.g. C++) program. However, the actual performance difference can be surprisingly large: a factor 100. VectorWise has created the first database system to revert that situation, with dramatic efficiency improvements as a result


http://www.vectorwise.com/index_js.php?page=mission_technology

http://www.vectorwise.com/index_js.php?page=mission_technology


http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html

http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html




ID: 922678 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 922725 - Posted: 31 Jul 2009, 19:01:47 UTC
Last modified: 31 Jul 2009, 19:04:16 UTC


I hope one guy from the Berkeley crew noticed that the UL server is offline..

Maybe one have time before the WE for to 'kick'..? ;-)


Maybe for the future Matt, if it's possible you could note it ~ a day before on the main site and/or in the TNews that one/some (UL/DL/scheduler) server will be diabled soon? It would help to hold down/would reduce angry posts in the 'NC panic thread'.

ID: 922725 · Report as offensive
Soggy

Send message
Joined: 28 Jan 03
Posts: 2
Credit: 4,631,709
RAC: 0
United States
Message 922768 - Posted: 31 Jul 2009, 21:02:32 UTC - in response to Message 922725.  

Dito on the UL Server. Got work that won't upload since AM Eastern. Still not in the PM Eastern. Whats up with that?
ID: 922768 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 922769 - Posted: 31 Jul 2009, 21:09:54 UTC - in response to Message 922768.  

perhaps we should all chill, I'm pretty sure they're working on the upload server and doing something to help increase to the performance of the network, data on the upload(Bruno)
ID: 922769 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 3187
Credit: 57,163,290
RAC: 0
United States
Message 922779 - Posted: 31 Jul 2009, 21:39:49 UTC
Last modified: 31 Jul 2009, 21:40:25 UTC

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?
.

Hello, from Albany, CA!...
ID: 922779 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 922782 - Posted: 31 Jul 2009, 21:56:02 UTC - in response to Message 922779.  

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?

... and you're sure that this was simply a case of someone not noticing, and couldn't possibly have been the entire staff working all day to fix broken hardware?????
ID: 922782 · Report as offensive
Soggy

Send message
Joined: 28 Jan 03
Posts: 2
Credit: 4,631,709
RAC: 0
United States
Message 922785 - Posted: 31 Jul 2009, 22:00:36 UTC - in response to Message 922779.  

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?
ID: 922785 · Report as offensive
Profile Andy Lee Robinson
Avatar

Send message
Joined: 8 Dec 05
Posts: 630
Credit: 59,973,836
RAC: 0
Hungary
Message 922798 - Posted: 31 Jul 2009, 23:16:26 UTC - in response to Message 922497.  

Matt, if you want to use mysql to get the benefit of the 24 processors, then you'll have to install mysql 5.4 - this has a lot of enhancements written by google programmers.
With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible. Core and thread usage is less than optimal.
In Fedora my.cnf, also try setting tmpdir=/dev/shm
This greatly increases throughput of queries that have to use temp tables.
ID: 922798 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 922800 - Posted: 31 Jul 2009, 23:35:45 UTC - in response to Message 922798.  

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.
Grant
Darwin NT
ID: 922800 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 922802 - Posted: 31 Jul 2009, 23:37:11 UTC - in response to Message 922800.  

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.
Grant
Darwin NT
ID: 922802 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 922819 - Posted: 1 Aug 2009, 0:53:52 UTC - in response to Message 922802.  

upload is up...
ID: 922819 · Report as offensive
MarkJ Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 1139
Credit: 80,854,192
RAC: 5
Australia
Message 922821 - Posted: 1 Aug 2009, 0:57:52 UTC - in response to Message 922802.  

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.


I was looking at the price of them. They are too expensive compared to the M version. You pay almost double the price of the M and get less than half the capacity.
BOINC blog
ID: 922821 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13715
Credit: 208,696,464
RAC: 304
Australia
Message 922836 - Posted: 1 Aug 2009, 2:02:42 UTC - in response to Message 922821.  

I was looking at the price of them. They are too expensive compared to the M version. You pay almost double the price of the M and get less than half the capacity.

That's the disadvanatge of MLC (Multi Level Cell) compared to SLC (Single Level Cell) SSDs. But under enterprise workloads the MLC drives would have a pretty short lifespan, hence the need for SLC in such instances.
Grant
Darwin NT
ID: 922836 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 923058 - Posted: 1 Aug 2009, 23:23:34 UTC - in response to Message 922577.  

on another note; i see that yall have added some more data disk to the list....

I noticed this as well, nice work. Maybe Matt got his software up & running, so we can crunch tapes from ages ago? Thanks team for all the hard work.
ID: 923058 · Report as offensive
Chris Campbell

Send message
Joined: 14 May 99
Posts: 17
Credit: 4,677,968
RAC: 2
United States
Message 923104 - Posted: 2 Aug 2009, 4:18:19 UTC - in response to Message 922497.  

Matt -

I see that you (or whoever) have now linked to the 10th anniversary videos from the home page.

It would be nice to have Eric's presentation, because the camera work during his segment was ... regrettable.

Please beg Eric for his presentation file (exported to PDF?) and post it alongside his video on the 10yr page. I've stated the full case for this here.

Thank you and everyone at S@H for your efforts!

- Chris

Running S@H since Day 1 -- May 14th, 1999!
ID: 923104 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22149
Credit: 416,307,556
RAC: 380
United Kingdom
Message 923117 - Posted: 2 Aug 2009, 8:07:25 UTC - in response to Message 922785.  

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?





Errr- What's "hair"???


Unknown concept at this location
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 923117 · Report as offensive

Message boards : Technical News : Moving Right Along (Jul 30 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.