Posts by Andrew Clayton


log in
1) Message boards : Technical News : Now it's 2014 (Jan 07 2014) (Message 1470555)
Posted 236 days ago by Profile Andrew Clayton
Regarding your Linux IO problems.

I wonder if you really are seeing lock ups or just non-responsive behaviour.

Not sure the specs on that machine. But I imagine it has quite a few GBs of RAM.
In which case I wonder if your hitting the VM writeback problem.

The kernel allows a certain amount of dirty data (data that needs to be written
out to backing store) to build up in memory. In current kernels, this can be as
much as 20% of your RAM that it will need to write out and can easily block
other IO causing long stall periods where it may seem that the machine has hung.

(These defaults where made in the heady days of having a machine with as much
as 64MB of RAM!).

It's also recognised that today even having a percentage at all isn't really
suitable and thus byte limits have also been introduced (but not used by
default).

So you may want to have a look at tuning

/proc/sys/vm/dirty_background_bytes and
/proc/sys/vm/dirty_bytes

Which ever of bytes or ratio is set last is what's used.

As to what to set them, that may take a bit of trial and error to see what works
best for you.

You could start out with say (8 & 24MB), you may well need to go quite a bit
higher, but you probably want to avoid having GBs of dirty data to be writing
out.

# echo 8388608 > /proc/sys/vm/dirty_background_bytes
# echo 25165824 > /proc/sys/vm/dirty_bytes

(Stick in /etc/sysctl.conf to make it permanent)

You can watch this in action with something like

$ watch -n 5 grep -e ^Dirty: -e ^Writeback: /proc/meminfo

You'll see the dirty value build then when it reaches the limit you'll see the
writeback value ramp up.

See https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Cheers,
Andrew
2) Message boards : Technical News : Mordent (Aug 19 2010) (Message 1026764)
Posted 1494 days ago by Profile Andrew Clayton
Dunno if you looked at Processing.js at all? Or if it is even suitable for your needs.

http://processingjs.org/

From the home page.

"Processing.js is an open programming language for people who want to program images, animation, and interactions for the web without using Flash or Java applets. Processing.js uses Javascript to draw shapes and manipulate images on the HTML5 Canvas element. The code is light-weight, simple to learn and makes an ideal tool for visualizing data, creating user-interfaces and developing web-based games."


Andrew
3) Message boards : Technical News : Semi Solutions (Jun 08 2009) (Message 905318)
Posted 1932 days ago by Profile Andrew Clayton
I wonder if you use, or have thought about using memcached?

http://www.danga.com/memcached/

4) Message boards : Technical News : Stumpy (Apr 23 2009) (Message 887764)
Posted 1978 days ago by Profile Andrew Clayton
A quick

# iptables -I INPUT -s $IP -j DROP

Will sort that out!
5) Message boards : Technical News : Harvey (Mar 24 2009) (Message 878941)
Posted 2008 days ago by Profile Andrew Clayton
If your RAID resync is going slow (check /sys/block/mdX/md/sync_speed, speed in KB/sec). You could try increasing it by tweaking
/sys/block/mdX/md/sync_speed_{min,max}
6) Message boards : Technical News : Monday Memo (Jun 23 2008) (Message 772802)
Posted 2281 days ago by Profile Andrew Clayton
One other advantage of software RAID is that you aren't locked in to a particular vendor and can easily move your RAID array from one machine to another.

As for performance I'm not sure it's specifically true that hardware RAID outperfoms software RAID these days with the CPU power available. True, there may be areas where Linux's RAID implementation could be improved.

There is also the work from Intel to allow a lot of the work to be offloaded to Intel Xscale IOP I/O processors, see:
http://marc.info/?l=linux-raid&m=118434090621144&w=2

Andrew
7) Message boards : Technical News : Monday Memo (Jun 23 2008) (Message 772631)
Posted 2282 days ago by Profile Andrew Clayton
If thumper is one of your Linux box's, then I assume your using mdadm. Maybe posting to the linux-raid mailing list (if you haven't already done so) with whatever problems your having, They seem a pretty friendly bunch and you may even get help from the man himself (Neil Brown)

http://vger.kernel.org/vger-lists.html#linux-raid

Personally I'd stick to software RAID and work on fixing any issues you have. Been bitten by hardware RAID before, RAID controller died, hosting company didn't have any more of that kind that would read the ondisk format, had to restore from backup.

Got a couple of small software (mdadm) RAID 5 arrays in the office (466GB and 2.1TB).


Cheers and keep up the good work.

Andrew



Copyright © 2014 University of California