X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:subject:from:to:references:message-id:date
	:mime-version:in-reply-to:content-type
	:content-transfer-encoding; q=dns; s=default; b=p21evyKg+UjaPM7B
	SRXBbfBK4++twAes3SWCO79rqq4QsqWeKf8kAQ2bMKn5sV/WIjJLm8KLjU7DbuiL
	d/wyi8MTn7doxBIDzFLS1Dq5+5B1aH1GF59y+TXRAPO1D9xXREVNWbPppTsvOn+x
	iRO6SLr3axDlIHud49wvrEY+8zI=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:subject:from:to:references:message-id:date
	:mime-version:in-reply-to:content-type
	:content-transfer-encoding; s=default; bh=fESiwY2cFXn9yrXhYgRC+p
	W/Jqc=; b=c0GoXX+kJj7gno2+VkJo5sCx4sSzms6kTNsymTY2C3JgcxhNYam5AP
	hStPhSKa+lVAHXNiIGN/Hqj9XHbcfhusP0xmgYLiz3tisg1lfHzntleaJNYVfwEi
	XD0GX+9LGNPg5cU3oEekr8NnczudAzAJK4ZwphGUqWo1xU3yht8xA=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Spam-SWARE-Status:	No, score=-1.1 required=5.0 tests=BAYES_00,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL autolearn=no version=3.3.2 spammy=H*r:sk:sonic30, HX-HELO:sk:sonic30, Hx-spam-relays-external:sk:sonic30, H*RU:sk:sonic30
X-HELO:	sonic303-4.consmr.mail.bf2.yahoo.com
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1531878760; bh=ah/xITCoo1u8bw0l1JXaJk9z4IxBaWe3utBLC5xddlo=; h=Subject:From:To:References:Date:In-Reply-To:From:Subject; b=RoTEGaDWlLK/a06zbBuIeD9xi6/LcSoBgJBRCSidaDi4KPVKp1Pksfebp1UB1pcPI5p3pL86uqjCZAkZibTdF4fGI35N8Wwu/x3OiGoDRXA5K3VMREdzyZxkvUQhPfZrIUtoi7XeijV7Aq9KLcHvXBP+xHTaxP7NjTOokb9APxWAHUrKiAhMK6NGCvsIuDdLHIupIVcGwGm7DTWoWLIFE/jgNj7Vmtbg74mwxXY+f7n20egaGzgdWEeEodTpPQuNfMiL1lEBlX+0OU9Vmy25/J1bVe5QAEdA4PX9QXnvJXBRfhDb078itI2pkU3U7tkbw6OhW/KDptLQAfZjWusEKA==
Subject:	Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug
From:	BloomingAzaleas <rdbingham AT verizon DOT net>
To:	cygwin AT cygwin DOT com
References:	<69b0bc3c-7ead-920e-f04b-ec631c3453b7 AT verizon DOT net>
Message-ID:	<2beb175e-2291-6b12-6efd-c84704f6762f@verizon.net>
Date:	Tue, 17 Jul 2018 21:52:37 -0400
User-Agent:	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Lightning/0.9 Thunderbird/2.0.0.23 Mnenhy/0.7.6.666
MIME-Version:	1.0
In-Reply-To:	<69b0bc3c-7ead-920e-f04b-ec631c3453b7@verizon.net>
X-MIME-Autoconverted:	from quoted-printable to 8bit by delorie.com id w6I1quMT016230

Reply to Steven Penny <svnpenn at gmail dot com>:

    no mis-behaving: this is intended behavior - you yourself have given
    workarounds: either redirect output to a file that can be later read, or pipe to
    command grouping ala {} or () and read stdin from inside the subshell

    im not sure what you are asking here - it seems you have a grasp already of the
    "problem" and "solution" - so nothing more needs to be said really, as nothing
    needs to be fixed - another option is "lastpipe"


If the reported behavior is the intended behavior, then this 
complaint is now a documentation complaint.  Neither the Cygwin 
man pages (quoted in the original report) nor the upstream GNU 
documentation indicate any exception to stdin redirect with pipe 
in the doc paragraphs for the subject builtins.  What confuses 
the situation is that some stdin re-directions work for these 
builtins in a parent shell context (left-chevs)  but not others 
(pipes) UNLESS performed strictly at mid-night under a full moon 
using obscure incantation hidden in pieces throughout a 178-page 
pdf.  In the GNU doc, the sub-shell for builtins in a pipeline 
rule involved is obscured by burial in paragraph six of section 
3.7.3 Command Execution Environment.

I suggest the following adjustment to the man pages inserting a 
parenthetical cue regards behavior in pipes:

   mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u 
fd]  [-C  callback]  [-c quantum] [array]
   readarray  [-d delim] [-n count] [-O origin] [-s count] [-t] 
[-u fd] [-C callback] [-c        quantum] [array]
     Read lines from the standard input (see pipeline 
restriction) into the indexed array  variable  array,  or ...

  read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N 
nchars]  [-p  prompt]  [-t timeout] [-u fd] [name ...]
      One  line  is read from the standard input (see pipeline 
restriction), or ...

At least the reader gets a cue to modify expectations and 
research the "pipeline restriction" compared to all other 
commands that work just peachy with stdin pipe re-direction.  
Regards the shopt lastpipe doc single-sentence paragraph in GNU 
section 4.3.2 The Shopt Builtin, it could be improved by adding: 
"This option will change the parent shell context behavior of 
shell builtins mapfile, readarray and read when they are used as 
the last command of a pipeline."

As to the left-chev work-arounds, yes they work, but that would 
require some script re-structuring.  In any case, "what you are 
asking here" is that there appears to be a difference between a 
plain reading of the doc paragraphs for the builtins and the 
builtins behavior and so a bug against one or the other.  If this 
were a unix/linux kernel environment, the triage emphasis would 
be documentation wording rather than implementation bug.  Given 
the Cygwin context of necessary hidden emulation chicanery on 
Windows, triage emphasis is not that obvious.  I exerted the 
effort to construct a reasonably complete report with multiple 
illustration cases contrasted to the doc to improve the FOSS, be 
it implementation or documentation.  I didn't need too.

Regards.

On 7/15/2018 7:01 PM, BloomingAzaleas wrote:
> Windows 10 Pro 10.0.17134 N/A Build 17134 patched through 15 
> July 2018
> Cygwin 2.10.0(0.325/5/3)
> Bash 4.4.12(3)
>
> Cygwin man pages show:
>
>   mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u 
> fd]  [-C  callback]  [-c quantum] [array]
>   readarray  [-d delim] [-n count] [-O origin] [-s count] [-t] 
> [-u fd] [-C callback] [-c        quantum] [array]
>     Read lines from the standard input into the indexed array 
> variable  array,  or ...
>
>  read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N 
> nchars]  [-p  prompt]  [-t timeout] [-u fd] [name ...]
>      One  line  is read from the standard input, or ...
>
> So the expectation is that all bash stdin re-directions such as 
> pipes and left chevs should work.  For mapfile/readarry, the 
> output array variable MAPFILE is described as being created if 
> not array_var argument is given to mapfile.
>
> A) Searched cygwin email list with terms 'mapfile', 'readarray' 
> and 'read builtin'.  No obvious hits of recent vintage in 
> summary result list.
>
> B) Confirm, not using mapfile/readarry/read, that bash stdin 
> redirs work as expected.
>
> echo multi-line_arg | /bin/cat
>
> /bin/cat < some_unix_fmt_file
>
> /bin/cat <<END
> some
> lines
> END
>
> all work as expected.
>
> C) Test mapfile/readarray/read in various stdin redir 
> situations.  Basic harnesses are :
>
> mapfile/readarry HARNESS:
> stdin or fd 0 feed to mapfile
> then
> : "${MAPFILE[@]:?MAPFILE null or unset}"
> if get past the ${:?} test then dump MAPFILE with a simple /for 
> /loop as: for ent in "${MAPFILE[@]}" ; do echo ="${ent}"= ; done
>
> read HARNESS:
> stdin feed to 'read foo'
> then
> : "${foo:?foo null or unset}"
> if get past the ${:?} test then dump variable foo with echo 
> ="${foo}"=
>
> CASES:
> echo multi-line_arg | mapfile
> cat multi-line_unix_fmt_file | mapfile
> RESULT:
> Fail with MAPFILE null or unset
> If 'for' loop forced, "${MAPFILE[@]}" expands to zero words.
>
> CASE:
> echo multi-line_arg | read foo
> cat multi-line_unix_fmt_file | read foo
>  RESULT:
> Fail with variable foo null or unset
>
> CASES: left chev re-directions
> mapfile < some_unix_fmt_file
> mapfile <<END multi-line_here_doc
> read < some_unix_fmt_file
> read <<END multi-line_here_doc
> RESULT:
> Work as expected
>
> CASES: mapfile or read in a sub-shell
> echo multi-line_arg | ( mapfile ; stuff )
> echo multi-line_arg | ( read foo; stuff )
> RESULT:
> Work as expected.  However, since mapfile/read result vars are 
> not accessible in parent shell context, can force a notable 
> re-structuring of a script to use sub-shells.
>
> CONCLUSIONS:
> For bash builtins mapfile/readarray and read, stdin pipe 
> re-direction directly to the builtin fails as if the builtin 
> read()/fread() sees a silent pre-mature EOF on the pipe.  In 
> the sub-shell tests, the pipe re-direction happens for the 
> sub-shell as a whole before commands internal to the sub-shell 
> get processed for individual re-dirs, so no re-dir processing 
> directly for the builtin.
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -