Subject: Re: SPARSE files considered harmful - please revert
From: Martin Buchholz <[email-masked]>
To: "Max Bowsher" <[email-masked]>
Cc: <[email-masked]>
Date: Sun, 18 May 2003 17:21:46 -0700

>>>>> "Max" == Max Bowsher <[email-masked]> writes:

Max> Martin Buchholz wrote:
>>>>>>> "Max" == Max Bowsher <[email-masked]> writes:
>> 
Max> May I suggest a middle road? Why not let sparse files be configurable
>> as a Max> $CYGWIN option? This would allow those users who actually want
>> them to Max> enable them with minimal effort, but keep them off for most
>> users.
>> 
>> I suspect that SPARSE files are genuinely useful, when storing large
>> files that have holes in them.  But I can't imagine one ever wanting
>> to use SPARSE for all files, because most files aren't like that.  So
>> I don't think sparseness is a good candidate for being put into
>> $CYGWIN.

Max> Agreed. I was just trying to find some simple compromise. Have you reviewed
Max> the long conversation that went on in cygwin-patches in February? Based on
Max> the ease with which this patch was accepted, I'm conjecturing that the core
Max> developers won't want a simple reversion.

Certainly using $CYGWIN is a solution that works.  When you want to
create your sparse file, you do

env CYGWIN="$CYGWIN sparse" make-hole

But it's not very elegant.

>> We could have a much cleverer implementation of sparseness, if we kept
>> statistics on the number and size of zero bytes in a file while it was
>> being written.  When we did the close(), we could automatically
>> transform it into a sparse file.  But I don't think even that should
>> be the default behavior, because it would make all IO slower.

Max> And it wouldn't achieve Vaclav Heisman's original goal, either - he wanted
Max> to avoid the delay caused by Windows zero-filling a file when it was
Max> initially writted to at a large offset.

I see.

Here's another idea...

Whenever a program does a seek on a file descriptor that has O_CREAT set,
we can check if the new position is more than, say 32kb from the current
position.  If so, we can call DeviceIoControl(FSCTL_SET_SPARSE).
This is cheap to implement and might just make everyone happy.
If it works.

Martin

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/