There Goes a Tenner (Jan 20 2011)

That was many many years ago - late 80's into the early 90's. Went back to school to get my masters from UW-Madison and then went to Microsoft as a 5th level enterprise networking tech. Been up and down the road a few times :)
Todd
____________

would bumping up the disk cache on synergy help? seems like 96 Gb of ram would give you a lot of options...

though it depends on what its trying to do, and how random, the read/writes are, along with how much data gets pushed through.

Nick
____________

we had two large storerooms of full height 5.25" 1GB Micropolis SCSI drives and one storeroom would be empty in a month.

Humm, the M-word might be key there. I was the first kid on my block to own a Gigabyte hard drive. This enabled me to do audio editing, as an entire concert would fit on the drive. The actual drive was a 1.6 Gbyte IDE Micropolis, but back then I think it was pretty common to sell the same drive hardware with IDE or SCSI interface, with maybe a $50 adder for the SCSI variant. I was extremely pleased to pay a little over $1000 for my drive.

The first one failed less than two days of installation--not subtle bad sectors, but hard failure. Warranty replacement went OK, but the second one lasted only a couple of years before also failing hard.

Now a single user experience on tiny sample is not the way to judge HD reliability, but I have heard that Micropolis did not lead the field in reliability even in their good days. How did Cray come to choose them?

Oh, yes, and my application was not hammering the thing at all. I probably did active audio work less than 1% of the time, and did not yet have an internet connection, so the drive was closer to excess idle than to excess activity.

Alex Hilleary wrote:

I'd swear that Micropolis was the company that got caught packaging bricks in order make it's quarterly numbers one time.

Close, but you are thinking of MiniScribe. I'm not sure the bricks story was proven, or the fact that employees believed it to be true was cited as one illustration of just how completely broken things were. It was a famous mess, either way.

I think I have a 40 MB Miniscribe disk on my AT&T UNIX PC (PC 7300 aka Safari) dating back from 1986. It is still working, after many reformatting and reloading the UNIX System V OS fro 5 1/4 inch disks. Memory is a hefty 512 KB.
Tullio

Also by the way, somebody asked if we should have two upload servers. We used to have the upload server split onto two systems but this wasn't helping - in fact it was making it worse. The problem is not the lack of bandwidth i/o, but disk i/o. The results have to live somewhere, and require lots of random read/writes. So it's best if the upload server saves the results on directly attached storage. If it is also serving them over NFS (or likewise equivalent) such that a second upload server can write to them, it's too much of an overhead drag. So the upload server has to be a singular server which also (1) holds the results and (2) as much of the backend processing on these result files as possible. I think right now the only backend processing on results which bruno does NOT do is assimilation, which vader handles. You might think "why not just have the upload server save the results IT gets on ITS own storage?" Then we end up with two piles of results, randomly split, and then the NFS/mounting bottleneck is simply pushed down the pike to the validators, who need to read both piles at once.

(First post, so please bear with me...)

My spontaneous thought when reading the above, is that since the entire S@H thingie is about distribution... shouldn't there be a way to distribute this too?

Just like the home computers might not always be so fast, but compensate this by their large numbers, wouldn't it be possible to have even more upload servers (not just two), and perhaps one dedicated "indexing server", to keep track of which upload server has which data...

And by increasing the number of ul servers to more than two, you also distribute the network traffic. So even if the total traffic gets higher, the load gets more spread out.
In other words, just two ul servers maybe isn't enought to distribute the network traffic, but with even more servers, the mere numbers will compensate for the network limitations.

I realise I'm probably just missing something more or less obvious here, just thought I'd mention it anyways... ;)

Sorry if I was unclear...

I didn't mean distributed as "distributed between users", but rather as "distributed between a larger number of local upload servers at Berkeley".

It won't solve the problem with the 100mbit link (but won't make it worse either), my thought was that it however might solve the problem with concentrating all data to one single upload server, which of course doesn't give much redundancy AND as far as I've understood is a bottleneck even when it works at its best?

Sorry if I was unclear...

I didn't mean distributed as "distributed between users", but rather as "distributed between a larger number of local upload servers at Berkeley".

It won't solve the problem with the 100mbit link (but won't make it worse either), my thought was that it however might solve the problem with concentrating all data to one single upload server, which of course doesn't give much redundancy AND as far as I've understood is a bottleneck even when it works at its best?

Sorry, but all the data HAS to go thru one server - as that server does all the validating, assimilation, and deletions. As it has been explained to me, any other possibility (and several have been tried, so I understand...) makes too much network thrashing as the WU/results get tossed back and forth between servers.