DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 5232BZM82622855
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 7BB803858D28
Message-ID: <d812cd8e-37c9-499b-a26c-cfb02d87c763@anduin.net>
Date: Mon, 3 Mar 2025 03:10:48 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: pipe.cc: Missing FILE_SYNCHRONOUS_IO_NONALERT in call to
 NtOpenFile call in nt_create() - possibly leading to ENOSPC in UCRT
To: cygwin AT cygwin DOT com
References: <fa85130b-9763-4b36-a431-1641667be253 AT anduin DOT net>
 <Z73dBsUPT4UeAqt6 AT calimero DOT vinschen DOT de>
Content-Language: en-US
Autocrypt: addr=bird AT anduin DOT net; keydata=
 xjMEXmi1hRYJKwYBBAHaRw8BAQdA+UswELoxV9TRrA9wXuhDQx/nBBqfyM93OcWy/jnXR0XN
 JEtudXQgU3QuIE9zbXVuZHNlbiA8YmlyZEBhbmR1aW4ubmV0PsKWBBMWCAA+FiEE7y46McDW
 gR4aNBubwAsG7Zek51UFAl5otYUCGwMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AA
 CgkQwAsG7Zek51WDgQEA+ZNXiRxhbhcycILk6+PpFpA1J7dYqZWSvX1dGUkARHABAKpfwVbg
 cRXFxJkz9uhTixDtOOFYGWAjwQQnG6YLDzwMzjgEXmi1hRIKKwYBBAGXVQEFAQEHQLiSD2of
 92ORL5i0n6YFBpHoW9orQDGQYGEDp0sZCr0BAwEIB8J+BBgWCAAmFiEE7y46McDWgR4aNBub
 wAsG7Zek51UFAl5otYUCGwwFCQlmAYAACgkQwAsG7Zek51X1OwEAuSCsZoxd1pXxTtzjCyIZ
 srWjCEWz2K8l2AzHWNnvww8A/1/b/SPnb7kbqFcy/StMPa7QV1yyGWXm+pvK8TJFdUoG
In-Reply-To: <Z73dBsUPT4UeAqt6@calimero.vinschen.de>
Precedence: list
From: "knut st. osmundsen via Cygwin" <cygwin AT cygwin DOT com>
Reply-To: "knut st. osmundsen" <bird-cygwin AT anduin DOT net>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit

Hi Corinna,

On 25/02/2025 16:08, Corinna Vinschen via Cygwin wrote:
> Hi Knut,
>
> On Feb 25 01:51, knut st. osmundsen via Cygwin wrote:
>> Hi,
>>
>> I've been hunting an issue for some days now, where a non-cygwin program
>> using microsoft's UCRT sometimes end up with a sticky error on stdout when
>> running under cygwin perl with a pipe capturing stdout and stderr.  When the
>> problem triggers, the pipe buffer appears to be full and it really looks
>> like it's hitting the errno=ENOSPC/doserrno=0 situation at the tail end of
>> _write_nolock() in ucrt/lowio/write.cpp.
>>
>> I *think* the issue is that the write end of the pipe isn't configured to be
>> synchronous.  In winsup/cygwin/fhandler/pipe.cc, the nt_create() function
>> sets FILE_SYNCHRONOUS_IO_NONALERT when creating the _read_ end of the pipe
>> using NtCreateNamedPipeFile, citing some C# program compatibility need.
>> But, the call to NtOpenFile below that opens the _write_ end of the pipe
>> doesn't set it.  It does set the SYNCHRONIZE access right, but doesn't set
>> the FILE_SYNCHRONOUS_IO_NONALERT flag (last parameter, is zero). This is
>> akin to calling CreateFile with FILE_FLAG_OVERLAPPED, if I understand it
>> correctly.
> We can't make the write side of the pipe synchronous easily, because
> this means a pretty big rewrite of the current code.  Right now, if
> we'd add the FILE_SYNCHRONOUS_IO_NONALERT, you couldn't interrupt
> NtWriteFile with a signal.

Sorry, I didn't look at the rest of the code before firing off the 
email.  I totally understand it would be a major pain to rework that, if 
it's at all doable.

> We can add such a change to the TODO list for 3.7, using NtWriteFile
> in a thread or something like that.
>
> However, maybe there's a chance we can fix this for 3.6, if you would
> be able to create simple testcase in plain C, reproducing your issue,
> and the actual problem is not the FILE_SYNCHRONOUS_IO_NONALERT.

Been exploring the issue some more over the last few days.  I *think* 
I've gotten to the bottom of it now, but a testcase require more work. 
But no worries, this is stuff that has been broken for ages and the 
OS+UCRT vendor is really at blame here.

The problem is that when the pipe buffer goes full and there are two or 
more concurrent WriteFile calls from UCRT/whatever that isn't aware that 
it's an asynchronous handle, i.e. no OVERLAPPED parameter,  WriteFile 
will get a STATUS_PENDING back from the NtWriteFile call and follow that 
up with a NtWaitForSingleObject call on the pipe handle since it must 
not return while the IO_STATUS_BLOCK variable on the stack can still be 
written to by the kernel.  There is a potential race between the two 
threads calling NtWriteFile and NtWaitForSingleObject.  If the wait 
order is inverse of the write order, the wrong (*) WriteFile caller will 
be woken up when some ReadFile activity triggers the completion of the 
first WriteFile call.  So, the call that is woken up prematurely returns 
zero bytes read (initial value set by the kernel code) and runs the risk 
of stack corruption later on when the operation is actually completed.

I've got some incomplete proof of concept code for this that 
sporadically ends up with a corrupted security cookie or EBP. Took me 
some time to understand the occasional  stack corruption problem, as I 
obviously suspected a bug in the testcase first, but the code is fine 
and it can be anything other than unexpected writes by the kernel 
causing it.  This also tallies with the reported amount of readable 
bytes in the pipe after these events (when they don't cause stack 
corruption), as these account for the whole amount of the WriteFile 
calls returning zero bytes written. Once I get some time again, I'll try 
hammer the testcase code into shape and share it.

Kind Regards,
  bird.

(*) It is also possible they are both woken up, if a NotificationEvent 
(waking up all waiters, manual reset) is associated with the file object 
on the kernel side rather than a SynchronizationEvent (single wakeup, 
autoreset).  Haven't had time to check this yet, but I hope this isn't 
the case.

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple