DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 4BFHUJoX3302972
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3B5603858D21
Message-ID: <d84f7f6c-5527-4f39-83a5-1aa16d8e451f@cornell.edu>
Date: Sun, 15 Dec 2024 12:29:42 -0500
User-Agent: Mozilla Thunderbird
Subject: Re: Atomic mmap replacement
To: cygwin AT cygwin DOT com
References: <66bf4f86-4618-b9a3-3e33-2c240b9204d0 AT cornell DOT edu>
 <20180219090042 DOT GC3417 AT calimero DOT vinschen DOT de>
 <e6b3bd42-f981-405e-b65b-529693598735 AT cornell DOT edu>
Content-Language: en-US
In-Reply-To: <e6b3bd42-f981-405e-b65b-529693598735@cornell.edu>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RitqeG14SkJkSGtvNTUvRE1OM3pLelpVejhEdnZ0QVRPc29hZHNJZ280NW9y?=
 =?utf-8?B?RThjdmdGR0kyN0lUZ3ZjTWJCTmpkbFpqYzdpWFl1K0tpTW40Z01rc3FpNkpK?=
 =?utf-8?B?d3N4T2M3TGNIK3dwa0JSa2pxbllHWmFiT21XOTZIcUp5ZE9pL1pvakVJeXpZ?=
 =?utf-8?B?djdTMzFvVXI0c3pqMjhHcHcrWjlaZ2h0UDlGQ2tiZmd2YzZESWQ2QTM5U1ZB?=
 =?utf-8?B?aWY4QnFlbmN1UXpUWkpLU0Fwb2ZuanNIeHA3RTd2TDE5Y2lHanlLWWFjSnFM?=
 =?utf-8?B?ZWVzaFFaQWx4RzRtS1A5cHBSVU9kTjZGUm9hbUE2d0dRNk1rNGp6NmlBcFlv?=
 =?utf-8?B?MFRhcElXWGxqNFJoMHM4ZHlZRWtCUysrYlhXSEdtOUY2Nm4yYkJHUkZYNVF1?=
 =?utf-8?B?V2RmNGdzZkw4YkU1NTZqR0MxUS9xYmo5TzdWYTh0MnlYYjdEZmF3T2hhbU1K?=
 =?utf-8?B?VVVWM1hLdmcvUWxlWEJHNUUzOEZNUzZpRHpSUlB2MHVLeXBSTEd1SisyM0xR?=
 =?utf-8?B?bUFSTG9RcFVUci9Od2NOampPZVR6RTNwL1ZNSEhDdjJ4bW5lRVFPTmdVSXRl?=
 =?utf-8?B?SmptcGNva3B4cUhJQ3FFdU1WWEs0MnBjOVJlaGo3QisyMFRUREh0YmdsU1lp?=
 =?utf-8?B?U1hiMkFtN2ZjZVJLRGdFZS9TZkYvUXVmbGZtVzRwVHFQVVJoZDhCUDBMamJQ?=
 =?utf-8?B?WElBZkR5UEd3aFpiUmNQc0FLUDlha1VhdzJ4ZWs0Y2o2T1hZQ3EwOGh5S1dM?=
 =?utf-8?B?eDhDMjhyUUFsZ3dPcm81TzNHQi8yQ20zYkViTG45YmNGWEk1ZDRVakRqR2RS?=
 =?utf-8?B?bE5tbDRjRTRIejM3YjdHcDlYbXFjYjFCTGN3TFVqR2JJK3NBeTFtMnZReVdC?=
 =?utf-8?B?ZDdnY2J2ZzNLSUh2ZTRVMm04RWR1NTRLRm5hT0hpQ3p5eVlONTluS3ZxNkZy?=
 =?utf-8?B?Rzh4ZGtob3pZSVppb1dNc0Q4NWpia0FDY0g1bC9VN0QrV0ZSMDJsMmNUQmo1?=
 =?utf-8?B?bnh3S1lSeVZvVUxVSVpzc2M3UUZlaEJEd2xqVmMwbGQ4ZGNIeGE5V1BwZ2FR?=
 =?utf-8?B?WmJ6eURQL0VDWjR6OC93THhFd05XaVpFdWhUWXFEYzUrSnp3cmJLd09PdnJs?=
 =?utf-8?B?dFBiVEtVRjR2RVFzYTVGalRuNVJZdFV3YStTRkhuUm9XNGtTdVRZUldoYUty?=
 =?utf-8?B?QXNDOU0yNUNTSDQvK2xwa25sQ1Z0RkpRb0lsSnlKL1VOYVNRMUNRdTBnYUFR?=
 =?utf-8?B?Zkp1TjFIcTdhQnE3Tno0cUU0V2hjM3FvNERaZy8zQklvR1J2VEdQSG80dlVx?=
 =?utf-8?B?UVNNUHRaemlrUlhrQWF6NXRGbjVhTmxUVnlsY21mdy9CcndYMXBTcWcvdUZw?=
 =?utf-8?B?bDJRbjlLcm5iZXMvYmpkZjhVbWJ6M1NIWVVTdGxoNWJaWFJsMm9OSUNEenJD?=
 =?utf-8?B?WlVLNVQyRVVSWUN0WVIyMnRHVzN2U1BNOGp6L2ZZaU9xMmkzbEg0OGZITlFP?=
 =?utf-8?B?RjhnN29UTTVITXpOaklOdGlONzlVN01odzhhU1RyQ0ZTR1BZallWY1ZoVGVw?=
 =?utf-8?B?ZUhaYkM1Y1VhcTNXSHFTUElSOFZxOUgrVFNHOXB5aWtIN0tvODQreVNFWmgr?=
 =?utf-8?B?WHFsMHFjTFAzdWNYRldCTE5IMjNyblM2NmtuTVBVMjlIUEpXV05vVmV6ZWp5?=
 =?utf-8?B?TU9tWFJwQ095TFlqOW92Q2czY0xxcFJVbGFIYzN0Y0xkTHpZcFMxYnY4aVNp?=
 =?utf-8?B?V29ndU1UR2NRbHlsMzNBTXFoTG9ZdytFc0wwaHhtb1kzckxqbWdweHBKWGRC?=
 =?utf-8?B?MlJtakVNOTF2THJvUUVNN1hhRlFCVC95TjdYdVJMSC9SWE4zb3VHdlNhRkVF?=
 =?utf-8?B?QlgyR0E4QW12YjNJR1F6S2ROUXl5ZTRtUHo1bzFwb3NGckNqQjZIUW1ORmlh?=
 =?utf-8?B?WDY0NTk4VHd2L0xqVVkxSDgybkpsY3Q5WjFCcW1JSDhnSk9WRkxKc3pPdkcy?=
 =?utf-8?B?THlsMXRIUmhLWkV4ckxKK002RGhQK2lENWZ1RGxRRmZUYjB0OTZqbklKYUF1?=
 =?utf-8?B?aFlqaTQ2UW1pUjJ3VjcwK0xDaVY3a0N2ZkVEbUxHbGhHamo0ZzlVNTFhUmFH?=
 =?utf-8?Q?O3xG77jY7C3QVxwarqEoI80oh?=
X-OriginatorOrg: cornell.edu
X-MS-Exchange-CrossTenant-Network-Message-Id: 30ce9eb7-e187-4f70-9338-08dd1d2e11ff
X-MS-Exchange-CrossTenant-AuthSource: SN6PR04MB5151.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Dec 2024 17:29:46.5773 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: W16ziV+GNzlN1LZPWtJrYagY+aGWw7EutAfoTzWJ2qN3g8/4unA/h/I58c0fyBst9A8zRyNChErDo5LBQY9SXQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR04MB7535
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Ken Brown via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Ken Brown <kbrown AT cornell DOT edu>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 4BFHUJoX3302972

On 12/14/2024 7:00 PM, Ken Brown via Cygwin wrote:
> Hi Corinna,
> 
> On 2/19/2018 4:00 AM, Corinna Vinschen wrote:
>> On Feb 17 22:37, Ken Brown wrote:
>>> Some code in emacs wants to reserve a chunk of address space with a big
>>> PROT_NONE anonymous mapping, and then carve it up into separate mappings
>>> associated to segments of a file.  This fails on Cygwin.
> 
> [...]
> 
>> Several limitations in the Windows kernel disallow this:
> 
> [...]
> 
>> - It also disallows to re-map any allocated or mapped mamory for another
>>    purpose.
>>
>> So this part of the POSIX specs for mmap:
>>
>>    "The mapping established by mmap() shall replace any previous mappings
>>     for those whole pages containing any part of the address space of the
>>     process starting at pa and continuing for len bytes"
>>
>> can't be implemented with Windows means.
> 
> I'm returning to this very old thread because of come up against another 
> application that wants to allocate a big block of memory and then 
> allocate pieces of it later.  I've looked at the documentation of 
> VirtualAlloc, and it seems that this should be possible:
> 
>     VirtualAlloc cannot reserve a reserved page. It can commit a page
>     that is already committed. This means you can commit a range of
>     pages, regardless of whether they have already been committed, and
>     the function will not fail.
> 
>     You can use VirtualAlloc to reserve a block of pages and then make
>     additional calls to VirtualAlloc to commit individual pages from
>     the reserved block. This enables a process to reserve a range of
>     its virtual address space without consuming physical storage until
>     it is needed.
> 
> The attached test case illustrates this.  Do you think it's feasible to 
> modify mmap to take advantage of this?  If you try to mmap a block 
> that's inside an already allocated block, I would want mmap to call 
> VirtualAlloc with MEM_COMMIT but not MEM_RESERVE, and this should succeed.
> 
> If you think this is feasible, I would be willing to work on it.  But in 
> that case I would appreciate some suggestions on how to implement it, 
> since I'm not yet very familiar with the mmap code.
It looks like a lot of the machinery for doing what I want is already 
present in mmap.cc.  If I want the initial allocation to reserve without 
committing [in the Windows sense of "reserve"], I just need to specify 
MAP_NORESERVE in the call to mmap [now we're using "noreserve" in the 
Linux sense].  Right?  Then future mmap calls to allocate memory within 
that first block could simply check for the noreserve flag and use 
MEM_COMMIT without MEM_RESERVE.  Obviously there are a lot of details 
that I haven't yet thought through, but I'm cautiously optimistic.

Ken

P.S. The conflicting meaning of "reserve" in Windows vs. Linux was very 
confusing to me at first.  There's probably nothing that can be done to 
make the code less confusing.

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple