DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:in-reply-to:references:from:date
	:message-id:subject:to:content-type; q=dns; s=default; b=qsZnibM
	n/Q4EjqB/x1lso09NHwkv+7uNwXa/ZKldlyn59F0JRVKzgjHF6GyaMOkL9W3KMK6
	4PqV6N3PUbQOxRdsNbkAiKksHR9RUFo7LNzZH+d11+SgNt3bBT0M84Ub2+qKn057
	HJEDgpKRYiAWTAC0pWA3J5VzVP1GfR8gTWRg=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
MIME-Version: 1.0
In-Reply-To: <1495612367.2760331.986814392.79C77EB2@webmail.messagingengine.com>
References: <1495612367 DOT 2760331 DOT 986814392 DOT 79C77EB2 AT webmail DOT messagingengine DOT com>
From: Erik Bray <erik DOT m DOT bray AT gmail DOT com>
Date: Wed, 24 May 2017 13:34:19 +0200
Message-ID: <CAOTD34Z0q9_vcgMEBuUj2HL7==qVng=m6vGhmxRo1GGv=5pFrQ@mail.gmail.com>
Subject: Re: Bug: grep behaves incorrectly under the locale C.UTF-8, if a file contains Umlaut characters
To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset="UTF-8"

On Wed, May 24, 2017 at 9:52 AM, Ronald Fischer wrote:
> I have a file X which contains ASCII text, but also in some lines German
> umlaut characters. The file is classified as:
>
>      $ file X
>      X: ISO-8859 text, with CRLF line terminators
>
> If I grep the file using, say,
>
>      $ grep  .  X  >Y
>
> (i.e. select every non-empty line and write the result to Y), this works
> fine, if LANG is set to one of: UTF-8, C, C.de_DE, C.en_EN, en_EN,
> de_DE.
>
> However, if LANG is set to C.UTF-8, two things happen:
>
> - grep classifies the file as binary file and produces the error message
> "Binary file X matches"
>
> - Both the grepped lines (i.e. in our example the non-empty lines) AND
> the error message end up in the standard output (i.e. in file Y).
>
> IMO, there are several problems with this:
>
> 1. It's hard to see, why an umlaut character makes the file X binary
> under encoding C.UTF-8, but not under encoding UTF-8 or C.en_EN

Only one of these, "UTF-8", specifies an *encoding*.  Further, I don't
think just "UTF-8" or "C.en_EN" are valid locale specifiers.  You can
read more about how Cygwin handles locales here:

https://cygwin.com/cygwin-ug-net/setup-locale.html

Certainly, if you set something like LANG="C.UTF-8" it will use UTF-8
to decode the text in the file and, failing that, treat it as binary.
If you know it's ISO-8859-1 you can use either LANG= or
LC_CTYPE="C.ISO-8859-1".  Or, if you know the language, you should be
able to use LC_CTYPE="de_DE".  Each language+territory has a default
encoding associated with it.  For German I think it is one of the
ISO-8859-* variants.  Or you can use the variant "de_DE AT euro" which
forces ISO-8859-15 which includes the euro symbol, among other
possibilities.  So I think your main problem here is just not
specifying your locale correctly.

> 2. If grep classifies a file as binary, I think the desired behaviour
> would be to NOT produce any output, unless the -a flag has been
> supplied.
>
> 3. If grep writes a message "Binary file ... matches", this message
> should go to stderr, not stdout. The stdout is supposed to contain only
> a subset of the input lines.

I would tend to agree with this, but this is normal behavior of grep
(on Linux too), so I would take it up with the authors of grep.

Best,
Erik

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple