X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:in-reply-to:references:from:date
	:message-id:subject:to:content-type; q=dns; s=default; b=qsZnibM
	n/Q4EjqB/x1lso09NHwkv+7uNwXa/ZKldlyn59F0JRVKzgjHF6GyaMOkL9W3KMK6
	4PqV6N3PUbQOxRdsNbkAiKksHR9RUFo7LNzZH+d11+SgNt3bBT0M84Ub2+qKn057
	HJEDgpKRYiAWTAC0pWA3J5VzVP1GfR8gTWRg=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:in-reply-to:references:from:date
	:message-id:subject:to:content-type; s=default; bh=e4qRDCVxiNFYV
	7q2Rc1RyURYSUQ=; b=B8NMCHCeInNmruz2LoO1WqUOCnOcsJb7DVfIcXZUIatqa
	4FyuZJ0g6AUjGAiAv9DvCFnu05bMTLKFjKWWRjiPg1lFiZvFspogll2Twt54v8vv
	ohIWG6yekOpJeeKhcW8DpSvWE7mFU2OgVEyzW7sIYS2mClc5d98HMQlOEE+yeI=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Virus-Found:	No
X-Spam-SWARE-Status:	No, score=2.8 required=5.0 tests=AWL,BAYES_50,FREEMAIL_FROM,LIKELY_SPAM_SUBJECT,RCVD_IN_DNSWL_NONE,RCVD_IN_SORBS_SPAM,SPF_PASS autolearn=no version=3.3.2 spammy=german, German, ronald, Ronald
X-HELO:	mail-yw0-f181.google.com
X-Google-DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=diiXbv+TNDxOVPvQRyzXtQU7/OIwsM522sCxNTiAxtY=; b=O0J7LfL+Ym+bpizgB/YvrbCRCHFZGYo0DZgEDFg2nUqFhzB6I4/9sJdWypzuh4ZBiu PhFmSngZLod90cuZCtX5Rn5BwNP3qcpMCBAhYeiyyal0UQYaVlv5wA4JewKq2iHvv2qC EFsJ8IftJU8cWkFEgUumXt2SeOEBvCHts2q0tm35jr+1AoIhmZU8XC14pe8pc4Y7b5fJ i3CMyN/t/hU6kADVu3NITRJx438XfHQ/d7ot0pXDvsL2NxeyY8JIEB/8VuJkMIbdlbf2 oEo7Vcerj/SR18WW1vKgsl/1sMsNjVoQ9G6s5APCiyLWwH5eji6McHlDakpEovRqSxcE fWXg==
X-Gm-Message-State:	AODbwcAtQVisHxWDKcW4kNEdHDuNtorqJks1+JKQPeQDkVRwIuIKT+W3 lUgQtTjlE7V7zOOciRO+wetsHbzqsHyqkrc=
X-Received:	by 10.129.175.91 with SMTP id x27mr28642122ywj.192.1495625659881; Wed, 24 May 2017 04:34:19 -0700 (PDT)
MIME-Version:	1.0
In-Reply-To:	<1495612367.2760331.986814392.79C77EB2@webmail.messagingengine.com>
References:	<1495612367 DOT 2760331 DOT 986814392 DOT 79C77EB2 AT webmail DOT messagingengine DOT com>
From:	Erik Bray <erik DOT m DOT bray AT gmail DOT com>
Date:	Wed, 24 May 2017 13:34:19 +0200
Message-ID:	<CAOTD34Z0q9_vcgMEBuUj2HL7==qVng=m6vGhmxRo1GGv=5pFrQ@mail.gmail.com>
Subject:	Re: Bug: grep behaves incorrectly under the locale C.UTF-8, if a file contains Umlaut characters
To:	cygwin AT cygwin DOT com
X-IsSubscribed:	yes

On Wed, May 24, 2017 at 9:52 AM, Ronald Fischer wrote:
> I have a file X which contains ASCII text, but also in some lines German
> umlaut characters. The file is classified as:
>
>      $ file X
>      X: ISO-8859 text, with CRLF line terminators
>
> If I grep the file using, say,
>
>      $ grep  .  X  >Y
>
> (i.e. select every non-empty line and write the result to Y), this works
> fine, if LANG is set to one of: UTF-8, C, C.de_DE, C.en_EN, en_EN,
> de_DE.
>
> However, if LANG is set to C.UTF-8, two things happen:
>
> - grep classifies the file as binary file and produces the error message
> "Binary file X matches"
>
> - Both the grepped lines (i.e. in our example the non-empty lines) AND
> the error message end up in the standard output (i.e. in file Y).
>
> IMO, there are several problems with this:
>
> 1. It's hard to see, why an umlaut character makes the file X binary
> under encoding C.UTF-8, but not under encoding UTF-8 or C.en_EN

Only one of these, "UTF-8", specifies an *encoding*.  Further, I don't
think just "UTF-8" or "C.en_EN" are valid locale specifiers.  You can
read more about how Cygwin handles locales here:

https://cygwin.com/cygwin-ug-net/setup-locale.html

Certainly, if you set something like LANG="C.UTF-8" it will use UTF-8
to decode the text in the file and, failing that, treat it as binary.
If you know it's ISO-8859-1 you can use either LANG= or
LC_CTYPE="C.ISO-8859-1".  Or, if you know the language, you should be
able to use LC_CTYPE="de_DE".  Each language+territory has a default
encoding associated with it.  For German I think it is one of the
ISO-8859-* variants.  Or you can use the variant "de_DE AT euro" which
forces ISO-8859-15 which includes the euro symbol, among other
possibilities.  So I think your main problem here is just not
specifying your locale correctly.

> 2. If grep classifies a file as binary, I think the desired behaviour
> would be to NOT produce any output, unless the -a flag has been
> supplied.
>
> 3. If grep writes a message "Binary file ... matches", this message
> should go to stderr, not stdout. The stdout is supposed to contain only
> a subset of the input lines.

I would tend to agree with this, but this is normal behavior of grep
(on Linux too), so I would take it up with the authors of grep.

Best,
Erik

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -