From: malcolm AT concept-asa DOT de (Malcolm)
Subject: B19 - reading large files (NT, 95)
13 Nov 1998 17:07:26 -0800
Message-ID: <364C2404.57B6BA3C.cygnus.gnu-win32@concept-asa.de>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
To: gnu-win32 AT cygnus DOT com, malcolm AT concept-asa DOT de

I have some severe performance problems with NT when reading
large files. This problem does not only arise with gcc but also
withVisual C:

Suppose that I have a PentiumPro with 96MB memory and I wish to make a
copy of a large file - say 110MB, i.e.
% cp largeFile newFile
then, on the same hardware, Linux is about 4 or 5 times faster than
NT.

A test program I wrote, in pseudo code, is as follows:

    while ( read( fd_in, buffer, sizeof( buffer ) ) == sizeof ( buffer )
)
    {
        ... do something ...
        (void)write( fd_out, buffer, sizeof ( buffer ) ) ;
        ... do something else ...
    }
    close( fd_in ) ;
    close( fd_out ) ;
    exit( 0 ) ;

If "do something" and "do something else" are commented out
then performance with both gcc and Visual C on NT as compared with Linux
is similar to the use of "cp".

If "do something" is quite simple like
    dummy = (char*)malloc( 4096 ) ;
and "do something else"
    free( dummy ) ;
Then Linux is about 20 times faster than NT.

The problem is obviously (the NT task manager shows this clearly)
that both CygWin "cp" and my test program eat the available
memory and use it to extend the cache io buffers until there is
no available memory. At this point - i.e. half the file has been copied
- NT spends all its time swapping or paging. When this critical point
has been reached it requires about 3 or 4 minutes just to focus on
a window.

My actual application, of course, does rather more in "do something"
and "do something else" so that Linux is finished in 5 min. and
NT in more than one hour.

I presume the problem can be avoided at the programming level by
using some undocumented feature of NT since using NT's "copy"
command is just as fast as Linux's "cp". I had hoped that you would have
known how to do this in implementing CygWins's "cp" :-)

Any ideas would be gratefully received.

Malcolm

--

===============================
Malcolm Agnew
Concept asa
Gluckstr. 18
D-60318 Frankfurt/Main
Germany
Tel:    (**49) 69/59 70 297
Fax:    (**49) 69/59 09 01
Email:  malcolm AT concept-asa DOT de
Always :-)
===============================



-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request AT cygnus DOT com" with one line of text: "help".