So we seem to have gotten over the hump with this new replica server. I should point out working on this server has had zero effect on the rest of the normal project operations, except for perhaps eating up all my time. Anyway, my script got around the dump/restore bug, and after some configuration headaches this morning we are successfully replicating on mork! Of course, sidious continues to be the replica we are using for production, while mork is considered "beta test."

It is catching up on the backlog far slowly than we hoped, especially given the power of the machine. Of course, power is measured in network, disk, memory and cpu. This system certainly has cpu (24 processors!) but word on the street is that mysql actually *drops* in performance after n processors. What "n" is, and what the penalty is remains unclear. Also, this system has fewer disk spindles than sidious (8 compared to 10), and they are slower disks, I think. So we may be seeing a disk i/o hit, but iostat doesn't really show anything amiss. The system is also in our lab and not in the closet, so there may be an extra network hop or two slowing things down. Anyway, as it progresses we'll gauge its performance and act accordingly.

As for changing linux flavors, the current issue here is mysql versions, and not so much linux distributions. As mentioned elsewhere we're trying to adhere to a homogenous setup, and we have less than zero time to mess around with anything experimental like trying new OSes on for size. In any case, Fedora works well enough, and while I generally swear by open source software for both philosophical and practical reasons, I do understand that you get what you pay for.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Indeed, I've also heard that MySQL doesn't scale well to processor counts beyond 4. Perhaps Mork could serve a different purpose? Seems to me you'd want a 4 or 8 core system with as many disk spindles as you can get for MySQL.

Also, is Mork setup to use RAID 10, 5, or a set of RAID 1 mirrors? I read a very interesting article on how using RAID 1 mirrors (with an RDBMS that can partition tables evenly itself) gives better concurrency/performance than a big RAID 10 with the same number of spindles.

Edit: Monk would be a better name, anyway. :)

DJStarfox: If you read other posts, it looks as though your references to "Monk" should read "Mork".

---

on another note; i see that yall have added some more data disk to the list....

---

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

July 30, 2009

Open source Database Breakthrough 10 - 80 times Faster

A groud breaking database kernel is now being combined with the leading open source relational database from Ingres

The figure below shows the architecture of the new VectorWise engine. The left part shows the system architecture (“X100” execution engine and ColumnBM buffer manager) and how it maps on the computer resources (CPU cache, RAM and disk). The right part shows a query in action, having been decomposed into so-called relational operators (Aggregate, Project, Select and Scan) and execution primitives (such as summation – aggr_sum_flt_col).





A groud breaking database kernel is now being combined with the leading open source relational database from Ingres



The Ingres VectorWise project team has worked with Intel to evaluate database performance on the new Intel Xeon processor 5500 series based platform. To date, the results of the project have demonstrated dramatic cost and performance capabilities as evidenced by nearly 80 fold speed up on a query modelled after the Q1 query of TPC-H3 suite on the Intel Xeon processor.

VectorWise next-generation database technology is based on a novel query processing architecture that allows modern microprocessors and memory architectures to reach their full potential. This is a unique achievement: in detail studies that compare common computing tasks such as scientific calculation, multi-media authoring, games, and databases have consistently shown that typical database engines do not benefit from new processor performance features such as SSE, out of order execution, chip multithreading, and increasingly larger L2/L3 caches due to their large, complex legacy code structure.

The computational power that database systems provide is known to be lower than the performance realized by hand-coding the same task in a (e.g. C++) program. However, the actual performance difference can be surprisingly large: a factor 100. VectorWise has created the first database system to revert that situation, with dramatic efficiency improvements as a result


http://www.vectorwise.com/index_js.php?page=mission_technology

http://www.vectorwise.com/index_js.php?page=mission_technology


http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html

http://nextbigfuture.com/2009/07/open-source-database-breakthrough-10-80.html

---

I hope one guy from the Berkeley crew noticed that the UL server is offline..

Maybe one have time before the WE for to 'kick'..? ;-)


Maybe for the future Matt, if it's possible you could note it ~ a day before on the main site and/or in the TNews that one/some (UL/DL/scheduler) server will be diabled soon? It would help to hold down/would reduce angry posts in the 'NC panic thread'.

---

Dito on the UL Server. Got work that won't upload since AM Eastern. Still not in the PM Eastern. Whats up with that?

---

perhaps we should all chill, I'm pretty sure they're working on the upload server and doing something to help increase to the performance of the network, data on the upload(Bruno)

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?

---

.

Hello, from Albany, CA!...

Uploads have been on the fritz since about 0400 Berkeley time...It took until 1300 before someone noticed?

... and you're sure that this was simply a case of someone not noticing, and couldn't possibly have been the entire staff working all day to fix broken hardware?????

---

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?

---

Matt, if you want to use mysql to get the benefit of the 24 processors, then you'll have to install mysql 5.4 - this has a lot of enhancements written by google programmers.
With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible. Core and thread usage is less than optimal.
In Fedora my.cnf, also try setting tmpdir=/dev/shm
This greatly increases throughput of queries that have to use temp tables.

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

---

Grant
Darwin NT

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.

---

Grant
Darwin NT

upload is up...

With Mysql 4,5,5.1 the main performance impact is gained by fast disks and ram and as much IO capacity as possible.

If only Intel could donate a few dozen of their enterprise SSDs.
Behold, the X25-E.

Just look at those database transaction rates.

I was looking at the price of them. They are too expensive compared to the M version. You pay almost double the price of the M and get less than half the capacity.

---

BOINC blog

I was looking at the price of them. They are too expensive compared to the M version. You pay almost double the price of the M and get less than half the capacity.

That's the disadvanatge of MLC (Multi Level Cell) compared to SLC (Single Level Cell) SSDs. But under enterprise workloads the MLC drives would have a pretty short lifespan, hence the need for SLC in such instances.

---

Grant
Darwin NT

on another note; i see that yall have added some more data disk to the list....

I noticed this as well, nice work. Maybe Matt got his software up & running, so we can crunch tapes from ages ago? Thanks team for all the hard work.

---

Matt -

I see that you (or whoever) have now linked to the 10th anniversary videos from the home page.

It would be nice to have Eric's presentation, because the camera work during his segment was ... regrettable.

Please beg Eric for his presentation file (exported to PDF?) and post it alongside his video on the 10yr page. I've stated the full case for this here.

Thank you and everyone at S@H for your efforts!

- Chris

---

Running S@H since Day 1 -- May 14th, 1999!

Understandable over looked. New Boinc platform...new software.....new hardware.....constantly updated..........anyone out there still have hair?

Errr- What's "hair"???


Unknown concept at this location

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?