X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:from:to:references:in-reply-to:subject:date
	:message-id:mime-version:content-type:content-transfer-encoding;
	q=dns; s=default; b=ObSE30xAP3PDIUhMoKf7fbC8IpgUVavzpsHASKN3Nvg
	JinYugBM15F8Enhi+/4nmjlIWg9pkdJ8tdm4BmNgTv4n+Q1bCA2klpI3jDbifchP
	pq5loPYcwflSCSF/PzjR+bBopJICCqQOI+6Dw0IdYNK1tvqTGvRFb5uA6qTyPCoY
	=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:from:to:references:in-reply-to:subject:date
	:message-id:mime-version:content-type:content-transfer-encoding;
	s=default; bh=VdB0uVt/XFs3n/nf9URJ0udt/cQ=; b=Qcfwj0zXtjUVIdYwj
	hf5wv8aKgMzwDDkxj950iR4JzAU6YEMgNM/dsJGanBeHhNqrXuQtkn3No/0SmY7W
	pGLX8m4nexZMKoR5TFu0e52GC8sXmxCHfM2tt/l6+1xqpCjxe9oVvhgdout2+WGt
	x7zjCOoC4ClOS5XE2lREMC1Jv0=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Virus-Found:	No
X-Spam-SWARE-Status:	No, score=-0.6 required=5.0 tests=AWL,BAYES_50,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS autolearn=ham version=3.3.2
X-HELO:	mout.perfora.net
From:	"James Johnston" <JamesJ AT motionview3d DOT com>
To:	<cygwin AT cygwin DOT com>
References:	<02ee01cefdce$9c343300$d49c9900$@motionview3d.com> <5F8AAC04F9616747BC4CC0E803D5907D0C455040 AT MLBXv04 DOT nih DOT gov>
In-Reply-To:	<5F8AAC04F9616747BC4CC0E803D5907D0C455040@MLBXv04.nih.gov>
Subject:	RE: Pipe syncronization and incompatibility between Cygwin and .NET Framework 4.0
Date:	Mon, 23 Dec 2013 19:14:54 -0000
Message-ID:	<034801cf0013$43acf220$cb06d660$@motionview3d.com>
MIME-Version:	1.0
X-IsSubscribed:	yes

(apologies to the list; I accidentally sent this reply directly to Anton and
so his reply that he just sent will appear out of order on the list
archive).

Hi Anton,

Thanks for the response.  Comments below...

> MSDN does not list pipe handles as waitable with WaitForMultipleObjectsEx
> (only a user comment at the end does).

That is true.  One would rightfully assume that it's not supported.  I have
assumed that myself in the past when working on asynchronous code for pipes
for a completely unrelated application.

However, their own .NET Framework 4.0 does this, so it must be at least
supported on some undocumented level.  This is the stock version of the
framework available on Windows 8; the older version that does not have the
wait, NETFX 2.0, is not installed by default.  And of course, the older
version doesn't have any of the new features available in NETFX 4.x - of
which there are many.  Sadly, "fixing" the code in the .NET Framework to use
a "supported" asynchronous mechanism is not realistic.  Nobody except
Microsoft can patch the .NET Framework, unfortunately... (it is not
(L)GPL!!)  The reference code is just for debugging and learning; you can't
use it to modify anything.

Here is the reference source code:

http://www.dotnetframework.org/default.aspx/4 AT 0/4 AT 0/untmp/DEVDIV_TFS/Dev10/R
eleases/RTMRel/ndp/clr/src/BCL/System/IO/__ConsoleStream AT cs/1305376/__Consol
eStream AT cs

Check line 192 for a call to the WaitForAvailableConsoleInput function.  I
was unable to find the source code for that function, which is an internal
C/C++ function (this is a mixed-mode assembly).  However, I was able to
determine that function calls PeekNamedPipe followed by
WaitForMultipleObjectsEx using the very handy freeware API Monitor from
rohitab.com.  Using this tool I also learned the exact parameters that they
pass to each of those functions, and then distill it into a C program that I
sent in my original message that reproduces the issue on my machine.
	
This WaitForAvailableConsoleInput function is new to .NET Framework 4.0, and
according to the comment, was added in order to prevent the ReadFile call
from blocking in order to support aborting a thread.  I think both of us can
probably suggest better ways of handling this that uses a supported
mechanism.  But this is how they did it. :(

As mentioned in the original message, this pattern of PeekNamedPipe &
WaitForMultipleObjectsEx doesn't cause any problems except with Cygwin for
reasons that are currently unknown to me.

> If you trace execution without
> PeekNamedPipe(), you'll see that WaitForMultipleObjectsEx() returns
> immediately, and that it is ReadFile() that blocks and waits for the input
> (which will be supplied after 3 seconds).

Right; this is very strange and contrary to what one would naturally expect
if WaitForMultipleObjectsEx was supported on named pipes (which it
apparently is at least on some undocumented level, if their own .NET
Framework 4.0 is doing it).  Intuitively, peeking something shouldn't affect
any synchronization.  Yet it does...  (Perhaps waiting on a pipe is not
"fully" supported and Cygwin found an edge case!)

> --quote--
> Is it possible to wait for data to be available on a non-GUI-based console
> input? The issue is that WaitFor* methods do not support PIPE handles and
> therefore STDIN is not supported if the process input is fed from another
> process, e.g. using the pipe | functionality of cmd.
> 
> The only way I know to query a pipe for data without actually reading the
> data is to use PeekNamedPipe(), but then you lose the ability to wait on a
> timeout effectively, because it has no timeout so you would have to
> implement a manual loop instead.
> --quote--

Indeed; there is no clean solution.  All the ideas that come to mind have
problems:

 * WaitForMultipleObjectsEx: not officially supported, as previously noted.
 * Overlapped I/O: perfect supported solution except requires the pipe to be
initialized to support that, which probably wasn't done.
 * PeekNamedPipe in a busy loop as suggested in the quote: (1) high CPU
utilization, (2) and/or high latency on the pipe, due to unnecessary
sleeping, (3) your code stays active in memory and can't be paged out to
disk, (4) all the above kills power efficiency...

I've used overlapped I/O myself in the past on an unrelated application, but
I had full control over the creation of the pipes and full control over the
reading/writing of the pipes - not the case here.

So I can see why they went for WaitForMultipleObjectsEx method... the other
methods have significant performance and/or functional limitations, and they
probably found out from a group within Microsoft that it's ok to wait on a
pipe anyway.

Best regards,

James Johnston


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -