From: "Matthew Kidd" <matthew DOT j DOT kidd AT gmail DOT com>
To: <cygwin AT cygwin DOT com>
Subject: Re: Spurious 'grep: writing output' in Cygwin 1.7.5-1 in certain cases
Date: Mon, 19 Apr 2010 18:12:29 -0700
Message-ID: <D9DAFBB4212E4955BE3AB73C7E249EEC@ghc.local>
MIME-Version: 1.0
Content-Type: text/plain; 	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com


It just says 'grep: writing output' without any clarification. And
for the record I probably should have said we saw this on a XP SP3
box. Also I think what I said about it printing the error out twice
for every match that grep finds is wrong. It is probably just once
for each match beyond the first 10 listed by head (the default number
or line for head) which in our case happens to be a large number
(~140,000) consistent with what you are saying about data pushed to
onto a after the reader has exited.

I am a bit puzzled by the 'this is expected' response. This annoyance
does not occur under Cygwin 1.5. And if I do the same things (head
or less) on Debian (2.6.26-2) using the same file I do not see this
behavior either. Nor do I ever remember seeing this behavior under
any other flavor of Unix (DEC Ultrix, AIX, SGI, Sun), though I admit
it has been 15 years since I have used some of those flavors. I
probably didn't try it with ~500 MB file on those flavors but surely
tried it with say ~10 MB file.

Is Cygwin 1.7 simply being more compliant to the standard(s)? Or is
there some way that a pipe reader can indicate that it has finished
cleanly and that further attempts to write on the pipe should not be
treated as errors? I'm not that knowledgeable about the details of
pipe handling.

In the case of piping to less, there are ~140,000 rows found by grep,
containing in aggregate ~8 MB of data. That is not small but the XP
box in question has 2 GB of memory. I would think that less could
easily buffer that amount of data.


----------------------------------------------------------------------
From: Eric Blake <eblake at redhat dot com>
To: cygwin at cygwin dot com, GNU <bug-grep at gnu dot org>
Date: Mon, 19 Apr 2010 17:37:55 -0600
Subject: Re: Spurious 'grep: writing output' in Cygwin 1.7.5-1 in certain
cases
References: <CE4D3C6371BA4722B73A7015133FC9B6 AT ghc DOT local>

[adding bug-grep]

On 04/19/2010 03:19 PM, Matthew Kidd wrote:
> We are seeing grep emit 'grep: writing output' multiple times in
> certain cases where the output of grep is piped to another program.
> Specifically, we see it in the following cases:
> 
>   grep thisandthat foo.txt | head
>   grep thisandthat foo.txt | less

Expected, if you are blocking SIGPIPE, although it might be improved a
bit.  Grep is telling you that it encountered a write failure due to
EPIPE, and is thus about to exit with non-zero status.  Did you paste
the full error message?  On Linux, I see:

grep: writing output: Broken pipe

with the errno being converted to a string to make it obvious.  If
cygwin is not printing the EPIPE errno string as part of the error
message that you are seeing, then that could be improved; it's hard to
tell from your report, though.

> There appear to be two 'grep: writing output' messages for each
> line grepped.

That is indeed annoying; but it is not cygwin-specific.  It would be
nicer to print a failure message exactly once in the case of EPIPE write
failure, rather than once per matching line.  Would you care to
contribute a patch?

> Not all pipe operations exhibit this behavior. For example with the
> same file, we do not see it in either of these cases:
> 
>   grep thisandthat foo.txt | last
>   grep thisandthat foo.txt | wc

Also to be expected - it is a factor of how much data the kernel can
push through a pipe.  You only get EPIPE write failures if SIGPIPE is
being ignored, and if the writer (grep) tries to push more data into the
pipe after the reader is done.  As readers, last and wc consume the
entire stream, but head and less can exit early, in which case the
kernel pushes back to the writer.  Another thing to note; you will only
get the errors for large files; for smaller files, the entire file can
be written into the pipe before the reader exits.

> Moreover a non-piped grep does not exhibit this behavior:
> 
>   grep thisandthat foo.txt

Of course, because there is no EPIPE write failures if stdout is not
a pipe.

> Note: in this case 'grep: writing output' does not appear in
> out.txt; my guess is that they are being sent to STDERR when they
> do occur.

Yes, POSIX requires that error messages, such as diagnosis of write
failure due to EPIPE, goes to stderr.

By the way, if you don't like this message, then consider changing your
environment to not ignore SIGPIPE.  Then grep will silently die from a
signal, instead of noisily exiting with an error message.

-- 
Eric Blake   eblake AT redhat DOT com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple