Back to Eric's Home Page
Memory mapped files for IDL
Sparse Files -
Shared Memory -
Shortly after beginning to use IDL, I became annoyed with a couple features
of IDL. First, when working with many large images, I would often run
out of virtual memory, despite having 127 MB available. Second, the ASSOC
feature, which associates a file with an IDL array, does not work as I
had hoped. Rather than easily allowing access to any element of any
array contained in a file, it requires that elements be copied into
temporary arrays, and then written back to the file array. Eventually
I got tired of it both of these problems and decided to do something about
it. I sat down and wrote VARRAY. It has the advantages of solving
both problems, and a couple I never thought of.
VARRAY is in external system routine in IDL. It is written in C and
has been tested on IDL 4.X and 5.X under SunOS 4.1.3, Solaris 2.X, and RedHat Linux 6.1. It should, however,
work with minor modifications under most UNIX systems that support the
and ftruncate() functions.
Making it work under Win32 is more of an effort, but should
be possible. I have no idea whether there is a mmap() equivalent
VARRAY is shipped with a Makefile that may require some modifications
depending upon the system used. Macros are provided for SunOS systems
with GCC and ACC compilers (GCC is recommended), Solaris, and Linux systems.
Once the Makefile has been modified,
VARRAY is compiled with the command:
The shared object file "varray.so" needs to be linked into the running IDL
process using the LINKIMAGE routine. I place the following command in my
Once the shared object is loaded, the VARRAY function becomes available.
The syntax of the VARRAY function is:
- "filename" is the name of the file you wish to associate with the array.
If the filename is omitted, a writable temporary file with a unique name
is created in the /tmp directory.
- "element" is a variable type of each element of the array. Currently only
numeric scalar types are allowed.
- "dim1..dim8" are the dimensions of the array. If the file is writable
and the dimensions are larger than the current file size, the file is
ftruncate()d to the appropriate size.
- The "/writable" keyword specified that the file is writable and that
the changes to array elements are shared and written to disk.
- The "/status" keyword causes VARRAY to print the number of open
files to the standard output. Currently VARRAY supports 32
simultaneously open files.
For example, the command
opens a file named "test.dat", creating it if it doesn't exist, and assigns
it to a 512 by 512 byte array. The elements of this array can be accessed
and written to. For example:
will write random values into the file. When the variable is deleted (i.e DELVAR,A) or reallocated (i.e. A=SOMETHING) all changes will be updated on disk.
VARRAY supports sparse files. In a sparse file, only those portions
of the file that contain non-zero data are written to disk. Try the following
Chances are, you just saw the message (unless you had 256 MB free):
% Unable to allocate memory: to make array.
Not enough memory
% Execution halted at: $MAIN$
Now, with VARRAY loaded try the following:
You should see...
A FLOAT = Array(8192, 8192)
Going to UNIX and doing "ls -l", we see that the file is 268435456 bytes long
and takes up 24k of disk space. Now convince yourself that the array is real
You'd better see 3.14159. Checking the file size again you'll see that it's
still 268435456 bytes long, but now it takes up 40k of disk space. Check
that things are repeatable by deleting the variable, and reloading it.
You should still see 3.14159. If you want to sit around for a long time you
can even "print,total(a)".
A happy circumstance of VARRAY is that it allows memory to be shared
between IDL processes. If you map a file with the /writable keyword, the
changes will be shared with any other process that maps the file. As an
example, start two idl processes and link "varray.so" to them. In each,
now, in one enter "a(0)=1" then in the other, enter "print,a(0)."
Presto, interprocess communication. Of course there's no protection
for simultaneous access, so for each variable I would recommend that
one process read and the other write.
Bugs and Stuff-To-Do
There's always another bug or feature. Here are a few you should note:
- Arrays that are not mapped as writable use up swap space, as the system
ensures that enough swap space is available to support changes to the array
values. Thus, in order to save swap space, files must be mapped "/writable."
- In the above shared memory example, if the array in the reader process
is not mapped "/writable" and is written to, the array looses it's mapping
to the file, and the interprocess connection disappears.
- A planned "OFFSET" keyword, which specifies an offset in the file for the
mapping to begin, has not yet been implemented. Therefore, only headerless files
are supported for now.
This software is
Copyright 1998 UC Regents, All Rights Reserved.
Permission to modify and distribute this program is granted, provided
the copyright notice in the source code remains unchanged.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. The authors
and copyright holders are not liable for any damages resulting from
the use or misuse of this software.
Download Source Code
Download SPARC SunOS 4.1.3 Shared Object
Download SPARC Solaris 2.6 Shared Object
Download RedHat Linux 6.1 (glibc 2.1) Shared Object
- Version 1.00 First release
- Version 1.01 Fixed several function definitions
- Version 1.02 Background syncronization
- Version 1.03 Bug Fix, removed third parameter from the
call. Added Solaris section to Makefile.
- Version 1.03b Added some support for Digital Unix, improved Unix 97 compatibility.
- Version 1.04 Bug Fix, stupid error in line 202.
My thanks to Angelos Vourlidas at the Naval Research Lab, for providing
the Solaris binary and for pointing out the bug in the
Thanks to Nick Bower for providing the RedHat binary and reminding me to fix
the printf bug.
You are visitor
Counter provided by
Sparse Files -
Shared Memory -
Copyright © 1998 Eric Korpela