Moving Right Along (Jul 30 2009)

So we seem to have gotten over the hump with this new replica server. I should point out working on this server has had zero effect on the rest of the normal project operations, except for perhaps eating up all my time. Anyway, my script got around the dump/restore bug, and after some configuration headaches this morning we are successfully replicating on mork! Of course, sidious continues to be the replica we are using for production, while mork is considered "beta test."

It is catching up on the backlog far slowly than we hoped, especially given the power of the machine. Of course, power is measured in network, disk, memory and cpu. This system certainly has cpu (24 processors!) but word on the street is that mysql actually *drops* in performance after n processors. What "n" is, and what the penalty is remains unclear. Also, this system has fewer disk spindles than sidious (8 compared to 10), and they are slower disks, I think. So we may be seeing a disk i/o hit, but iostat doesn't really show anything amiss. The system is also in our lab and not in the closet, so there may be an extra network hop or two slowing things down. Anyway, as it progresses we'll gauge its performance and act accordingly.

As for changing linux flavors, the current issue here is mysql versions, and not so much linux distributions. As mentioned elsewhere we're trying to adhere to a homogenous setup, and we have less than zero time to mess around with anything experimental like trying new OSes on for size. In any case, Fedora works well enough, and while I generally swear by open source software for both philosophical and practical reasons, I do understand that you get what you pay for.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

on another note; i see that yall have added some more data disk to the list....
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

July 30, 2009

Open source Database Breakthrough 10 - 80 times Faster

A groud breaking database kernel is now being combined with the leading open source relational database from Ingres

The figure below shows the architecture of the new VectorWise engine. The left part shows the system architecture (“X100” execution engine and ColumnBM buffer manager) and how it maps on the computer resources (CPU cache, RAM and disk). The right part shows a query in action, having been decomposed into so-called relational operators (Aggregate, Project, Select and Scan) and execution primitives (such as summation – aggr_sum_flt_col).





A groud breaking database kernel is now being combined with the leading open source relational database from Ingres



The Ingres VectorWise project team has worked with Intel to evaluate database performance on the new Intel Xeon processor 5500 series based platform. To date, the results of the project have demonstrated dramatic cost and performance capabilities as evidenced by nearly 80 fold speed up on a query modelled after the Q1 query of TPC-H3 suite on the Intel Xeon processor.

VectorWise next-generation database technology is based on a novel query processing architecture that allows modern microprocessors and memory architectures to reach their full potential. This is a unique achievement: in detail studies that compare common computing tasks such as scientific calculation, multi-media authoring, games, and databases have consistently shown that typical database engines do not benefit from new processor performance features such as SSE, out of order execution, chip multithreading, and increasingly larger L2/L3 caches due to their large, complex legacy code structure.

The computational power that database systems provide is known to be lower than the performance realized by hand-coding the same task in a (e.g. C++) program. However, the actual performance difference can be surprisingly large: a factor 100. VectorWise has created the first database system to revert that situation, with dramatic efficiency improvements as a result


http://www.vectorwise.com/index_js.php?page=mission_technology

http://www.vectorwise.com/index_js.php?page=mission_technology


http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html

http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html




____________

I hope one guy from the Berkeley crew noticed that the UL server is offline..

Maybe one have time before the WE for to 'kick'..? ;-)


Maybe for the future Matt, if it's possible you could note it ~ a day before on the main site and/or in the TNews that one/some (UL/DL/scheduler) server will be diabled soon? It would help to hold down/would reduce angry posts in the 'NC panic thread'.

____________


>Das Deutsche Cafe. The German Cafe.<

perhaps we should all chill, I'm pretty sure they're working on the upload server and doing something to help increase to the performance of the network, data on the upload(Bruno)

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?
____________
.

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?

upload is up...

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.

on another note; i see that yall have added some more data disk to the list....

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?