[daip] Large File Support on 32bit Linux systems (fwd)

[Luca Moscadelli]

Hello,

I would like to ask info about the "problem" of reading > 2Gbytes files by AIPS
on Operatin System Linux. Is that possible now ?  I asked our system manager to write down "as short as he could" what has been the upgrade of the Linux Kernel via the LFS (here below). After that, AIPS (version APR99) was recompiled , but 
still it was not possible to read > 2 Gbytes files. May you help us ?

I would also to ask if you have any info of the performance of AIPS on HP 
running with Operatin System Linux.

Thanks for your attention,

Luca

---------- Forwarded message ----------
Date: Mon, 19 Mar 2001 16:09:19 +0100 (CET)
From: Giacomo Mulas <gmulas at ca.astro.it>
To: Luca Moscadelli <mosca at ca.astro.it>
Subject: Large File Support on 32bit Linux systems

	With the release of the new stable kernel tree (2.4.x versions),
Large File Support (LFS) has been integrated at the kernel level. This
allows creating/accessing files larger than 2Gb on 32bit systems (such as
x86 based computers), provided that the libc on the system includes
user-level support for LFS. This means that the libc on the Linux system
must be version 2.1.7 or above (2.2.x is better) and that it has been
compiled using the headers of a kernel that has LFS support (patched 2.2.x
or 2.4.x). In most Linux distributions, either binary LFS-capable versions
of the libc are available or it is very easy to build them from sources (I
did it myself on a Debian system and it took me two instructions given in
a shell).
	Now, even if LFS capable kernel and libc are available, for
compatibility reasons the old (non-LFS) functions are used by default.
Programs compiled on systems which do not support LFS will *not* use LFS
even if run on an LFS capable computer. To use LFS, one has to recompile
the program with LFS enabled. This can be achieved in several ways, but
anyway some preprocessor macros must be defined. This does work with C
programs (I tested it myself), I cannot say whether it works with fortran
programs, I can only suppose it depends on the fortran compiler used and
on how it maps fortran file access calls to libc calls. Here follows an
excerpt from the info documentation of the GNU libc (node "feature test
macros"):

 - Macro: _LARGEFILE_SOURCE
     If this macro is defined some extra functions are available which
     rectify a few shortcomings in all previous standards.
     Specifically, the functions `fseeko' and `ftello' are available.
     Without these functions the difference between the ISO C interface
     (`fseek', `ftell') and the low-level POSIX interface (`lseek')
     would lead to problems.

     This macro was introduced as part of the Large File Support
     extension (LFS).

 - Macro: _LARGEFILE64_SOURCE
     If you define this macro an additional set of functions is made
     available which enables 32 bit systems to use files of sizes beyond
     the usual limit of 2GB.  This interface is not available if the
     system does not support files that large.  On systems where the
     natural file size limit is greater than 2GB (i.e., on 64 bit
     systems) the new functions are identical to the replaced functions.

     The new functionality is made available by a new set of types and
     functions which replace the existing ones.  The names of these new
     objects contain `64' to indicate the intention, e.g., `off_t' vs.
     `off64_t' and `fseeko' vs. `fseeko64'.

     This macro was introduced as part of the Large File Support
     extension (LFS).  It is a transition interface for the period when
     64 bit offsets are not generally used (see `_FILE_OFFSET_BITS').

 - Macro: _FILE_OFFSET_BITS
     This macro determines which file system interface shall be used,
     one replacing the other.  Whereas `_LARGEFILE64_SOURCE' makes the
     64 bit interface available as an additional interface,
     `_FILE_OFFSET_BITS' allows the 64 bit interface to replace the old
     interface.

     If `_FILE_OFFSET_BITS' is undefined, or if it is defined to the
     value `32', nothing changes.  The 32 bit interface is used and
     types like `off_t' have a size of 32 bits on 32 bit systems.

     If the macro is defined to the value `64', the large file interface
     replaces the old interface.  I.e., the functions are not made
     available under different names (as they are with
     `_LARGEFILE64_SOURCE').  Instead the old function names now
     reference the new functions, e.g., a call to `fseeko' now indeed
     calls `fseeko64'.

     This macro should only be selected if the system provides
     mechanisms for handling large files.  On 64 bit systems this macro
     has no effect since the `*64' functions are identical to the
     normal functions.

     This macro was introduced as part of the Large File Support
     extension (LFS).