X-Recipient: archive-cygwin AT delorie DOT com
DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:references:in-reply-to:from:date
	:message-id:subject:to:content-type; q=dns; s=default; b=JNXn9Yc
	p21k6ITPebMKxmr1GphZ0gjmj5NPbVOxCgP1cXTpQy2NekDW2sXWJBVJp5A/yl4m
	KgT/n0HK8ZkJgmq5AFhwdJTqaJ+TOKH54PmjAQeTRVP+DG0DLMT5xPXx1zSh3roL
	Pq+7+B0tahUoe7KpCjUwQn/oc1KdZBjFh8GE=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:references:in-reply-to:from:date
	:message-id:subject:to:content-type; s=default; bh=PD2dOgojGfuZm
	cAkYIMUkZlFfZo=; b=jvdZq7sIL5iPk94FVifZviO4xpx5qxGgiGkQZp6xPaYuN
	O4lxWmeqDDRqyh8OEh9oPSVwHl7xphiFeIu5+N/mR8K16ni2yklgQIfXALQ8JenO
	yi1v+08HuKNlmPjJGtdeaAFmZOvBmKDRT7ZKIgPGVhIGP9TkvrpjvObXmcZWtE=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Authentication-Results: sourceware.org; auth=none
X-Spam-SWARE-Status: No, score=-0.9 required=5.0 tests=AWL,BAYES_00,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,UNSUBSCRIBE_BODY autolearn=no version=3.3.1 spammy=arise
X-HELO: mail-oi1-f178.google.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=gmail.com; s=20161025;        h=mime-version:references:in-reply-to:from:date:message-id:subject:to;        bh=xqWX0/UBSCI+NDpXAoBu6eEEr7rXPx9auuyThFn6vUo=;        b=JglsRLRjnpN2rzTjIrOh4Al34eqqlRc7HFC+5L2Ii00hMEzMXcolhJnYvIG03B9mZQ         TtxMVmfmG3hvmT/ZA9Bv0UlDaCkCt/VXRdFo7E3DxASK0r/BiN7lZ7FADZJXk4wVKHeM         5QmTZ3Cf0frtmSRdE2PW/HsZVhs7Orw7sH1QOzckOxF92ulyA/iWKNqhfpSuSOE+mGY2         RkxbG8ecr4yHS9gfGHMu4Pivsf5OEIgBhhFpldp3pWr5SZCTu8RclmAkXA0E+CvwjtJQ         sxQjIC+/soolmLl2oY4VZ9GmPL9OmC50oEiNnkyGTjUQSSzqtMDNgdgetcOGXXd+oypD         q10Q==
MIME-Version: 1.0
References: <CALLhcm4QGY+eP0_CRiSbJwQ12kOetvTK=6-AtC17x7d+QhGKTw AT mail DOT gmail DOT com> <20190603115456 DOT GG3437 AT calimero DOT vinschen DOT de> <CALLhcm4qaha=XuBWEqag7QH1veA82UrB94emFtFMd26dZ_ZGZQ AT mail DOT gmail DOT com> <20190604131836 DOT GS3437 AT calimero DOT vinschen DOT de>
In-Reply-To: <20190604131836.GS3437@calimero.vinschen.de>
From: Stanislav Kascak <stanislav DOT kascak AT gmail DOT com>
Date: Tue, 4 Jun 2019 15:48:53 +0200
Message-ID: <CALLhcm4sHGmFwki+RuUOUA-GfnXzmOUX4Tr8Es-=291Bo+A4dg@mail.gmail.com>
Subject: Re: possible problem with memory allocation using calloc/mmap/munmap
To: cygwin AT cygwin DOT com, Stanislav Kascak <stanislav DOT kascak AT gmail DOT com>
Content-Type: text/plain; charset="UTF-8"

> > > > It seems that when mmap() is called with length argument exceeding
> > > > size of file, only memory to fit that file is allocated. munmap()
> > > > however frees the full specified length. Since (at least on my
> > > > computer) big chunk of memory allocated by calloc() is located after
> > > > mmap() allocation, munmap() frees even memory of that calloc().
> > >
> > > Ken's right.  Due to the differences between mapping files on Windows
> > > vs. Unix, Cygwin can't map beyond the file size + the remainder of the
> > > last page.  Cygwin tries to workaround that on 32 bit by allocating
> > > an anonymous mapping following the file mapping to keep the range free
> > > from other mappings.  But on 64 bit this workaround doesn't work anymore
> > > because the OS is missing an (undocumented) flag which allows to
> > > create mappings on 4K boundaries, rather than just on 64K boundaries.
> > >
> > > I know this situation is unsatisfying, but I have no easy workaround
> > > to allow this.  Cygwin could add the anonymous mapping on the next
> > > 64K boundary on 64 bit, but that would result in a hole in the mapping
> > > which seemed like a rather bad idea when porting mmap to 64 bit.
> > >
> > > Ken's also right that munmap is doing the right thing here.  If
> > > anything's wrong, it's mmap's workaround for mappings beyond the file
> > > length.  If only 64 bit would allow 4K-aligned mappings :(
> >
> > Thanks for the answer. It is appreciated.
> > I understand the problem and difficulty to resolve it. Maybe returning
> > an error from mmap (and putting a comment to code for its reason)
> > would be sufficient. mmap caller could just adjust requested
> > allocation size to file size. Without error, caller has no way of
> > knowing memory was not allocated and segfault is then thrown in an
> > unrelated memory segment which makes the root cause hard to track
> > down. But, I do not know all the implication that could result from
> > that, so evaluation of this approach is up to you.
>
> Given that most of the required code already exists for 32 bit systems
> (except under WOW64, suffering the same problem as the 64 bit WIndows
> environment), I hacked a bit on this code this morning and I got your
> testcase running fine.  The idea being that after a successful mmap the
> expectation that a matching munmap does *not* unmap unrelated mappings
> is valid.
>
> In more depth, here's what Cygwin does on 32 bit, assuming a file size
> of 100 bytes and a mapping request of 256K:
>
> First Cygwin mmaps the file.  This results in a 4K mapping in Windows:
>
>  file:    |-- 100b --|
>
>  mapping: |-- 4K --....--|
>
> Next Cygwin adds another mapping to fill up the range up to the next
> 64K allocation granularity boundary:
>
>  |-- file 4K --|-- filler 60K --|
>
> Eventually Cygwin adds another mapping to fullfill the entire mapping
> request:
>
>  |-- file 4K --|-- filler 60K --|-- filler 192K --|
>
> The problem on WOW64 and real 64 bit is that it's impossible to map
> the first filler.  However, this area in the VM will *never* be
> allocated by other application functions due to the allocation
> granularity of 64K!
>
> So my workaround for 64 bit and WOW64 is to just skip allocating the
> first filler:
>
>  |-- file 4K --|-- THE VOID 60K --|-- filler 192K --|
>
> The advantage is now that the following munmap of 256K will only
> unmap the map for the file and the filler, but not the region you
> calloced before, which formerly was accidentally mapped to the
> filler region.  This just can't happen anymore now.
>
> Would that be feasible?  If so I can push my patch and create a
> developer snapshot for testing.

Two questions arise when I'm thinking about workaround solution:
- what happens if caller tries to write to |-- THE VOID 60K --|. Since
this is unallocated, would there be a segfault?
- is it possible that some subsequent mem alloc request would return
region from |-- THE VOID 60K --| which could again cause segfault
after munmap?

Stanislav Kascak

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple