X-Spam-Check-By: sourceware.org
From: ericblake AT comcast DOT net (Eric Blake)
To: Anders Brandén <x_anbr AT hotmail DOT com>, cygwin AT cygwin DOT com
Cc: lev DOT bishop AT yale DOT edu
Subject: bash process substitution [Was: Ref http://www.cygwin.com/ml/cygwin/2005-04/msg00651.html]
Date: Wed, 01 Feb 2006 05:04:34 +0000
Message-Id: <020120060504.21568.43E04161000EEE140000544022007610640A050E040D0C079D0A@comcast.net>
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com

[Picking a better subject line]

From: Anders Brandén
> Hi,
> 
> referring http://www.cygwin.com/ml/cygwin/2005-04/msg00651.html
> 
> I have a comment,
> 
> the problem seems to be more of a general kind(files that doesn't exist 
> already don't get created for writing) as these things happen on my Cygwin 
> system (running under Server 2003):

There is a distinct difference between disk files and pipes.

> 
> This is the new thing I've found, note that without the pipe(i.e. | cat) the 
> command runs just fine:
> 
> >tar -cf /proc/self/fd/1 syntax.c | cat
> >tar: /proc/self/fd/1: Cannot write: Bad file descriptor
> >tar: Error is not recoverable: exiting now

Here, the pipe operator tells bash to create a pipe and use the write
end of that pipe as tar's stdout, so tar is started with fd 1 as a pipe.
Inside tar, cygwin is apparently getting the read end of /proc/self/fd/1
instead of the write end, leading to tar complaining about a write
failure when it tries to put output on a read-only fd.

> 
> and this is what happens on my system with the command referred in the link 
> above, note that the error messages are the same!
> 
> >$ tar -cf >(cat) syntax.c
> >tar: /proc/self/fd/63: Cannot write: Bad file descriptor
> >tar: Error is not recoverable: exiting now

Here, bash creates a pipe for the command substitution, which
is named /proc/self/fd/63, then we are once again at the point
where tar opens the read end of fd 63 instead of the write end,
explaining the same error as before.

> 
> Both of these commands effectively creates a temporary file for both reading 
> and writing and that seems to be the problem, this command runs just fine 
> because I create a file for writing:
> 
> >$ tar -cf >(cat) syntax.c 63>temp

Whoa.  Your explanation has a hole.  Here, bash handles the
process substitution first, creating a pipe (fd 63), of which the
write end will be handed to tar and the read end handed to
cat.  Then bash does the redirection, opening the file temp
(whether pre-existing or not) for writing as fd 63.  Now
cat still has the read end of the pipe, but the write end has
been lost (so cat becomes a no-op), and tar now has a file
rather than a pipe at /proc/self/fd/63, and cygwin correctly
treats a file open for writing as a writable fd.

> 
> and, once created, this command runs fine too, however note that the 
> redirection of input also redirects output though it really shouldn't (try 
> it without having the temp file first, and then with an empty temp file and 
> check the contents afterward):
> 
> >$ tar -cf >(cat) syntax.c 63<temp

Similar to above - cat is a no-op process, since you threw
away the write end of the pipe, and now tar is handed a
file descriptor opened for reading.  If tar is able to write
to /proc/self/fd/63, then that might be a cygwin limitation
where files are treated as read-write instead of read-only.

> 
> So without the temp file, this fails every time:
> 
> >$ tar -cf >(cat) syntax.c 63<temp 63>temp
> >bash: temp: No such file or directory

The first redirection (63<temp) fails if temp does not
exist, because POSIX requires read redirection to
fail if the file does not exist.

> 
> But this always works:
> 
> >$ tar -cf >(cat) syntax.c 63>temp 63<temp

Here, the write redirection creates the file temp if it
did not already exist, then the read redirection
grabs the same file for reading; and you are back to
my earlier comment that if cygwin lets tar write to
a read-only file descriptor, then that is a bug/limitation
of cygwin.

> 
> Writing to temp gives >(cat temp) nothing to read:
> 
> >$ tar -cf >(cat temp) syntax.c 63>temp 63<temp

This is racy.  It is not guaranteed whether cat will
parse its arguments and try to open temp for reading
before bash has finished handling its redirections and
created temp for writing; then reopened temp for
reading.  You cannot reliably predict what will
happen here.

> 
> >$ tar -cf >(cat temp) syntax.c 63<temp 63>temp

Likewise racy.  Don't try it - it won't be predictable.

> 
> However there is obviously also something wrong with the redirection of 
> standard input for
> >(cat) because I get no output with either of these statements even after 
> >the temp file is created (writing to temp2 while attempting to read from 
> >temp):
> 
> >$ tar -cf >(cat) syntax.c 63>temp2 63<temp

Well of course - read my earlier comments - bash is hosing
the write end of the pipe, so cat is reading from nowhere,
and will print nothing.

> 
> >$ tar -cf >(cat) syntax.c 63<temp 63>temp2
> 
> So only these gives the expected output:
> 
> >$ tar -cf >(cat temp) syntax.c 63>temp2 63<temp

I don't know how you can call that expected, since it is
trying to rely on a data race.

> 
> >$ tar -cf >(cat temp) syntax.c 63<temp 63>temp2
> 
> Hope this helps in pinpointing the problem.

What problem?  The original mail was about cygwin treating
/proc/self/fd/nnn as read-only if it refers to a pipe, whether
or not the process was handed the write end of that pipe; but
the last half of your email has been complaining about fds
referring to files and not pipes.

--
Eric Blake
volunteer cygwin bash maintainer

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/