DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:content-type:mime-version:subject:from
	:in-reply-to:date:content-transfer-encoding:message-id
	:references:to; q=dns; s=default; b=nXn7UBq6kSBgZ1BxSGQME5qy82uy
	SoX1UWebQ+Fuvs1/uK+W7DfM8Ev5lF7SpCHdRuIDjpiR/Ix3o82kdWTsVLfXem6t
	jvy+2VfBmWUMqQPDpqkp9G21PVnzHaUfKBQny/JWb9ETpf+E36y+Aq7+ACUnN7KZ
	occ+PFuyV66noKA=
Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm
Sender: cygwin-owner@cygwin.com
Mail-Followup-To: cygwin@cygwin.com
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
Subject: Re: grep treating my text files as binary!
From: Warren Young <wyml@etr-usa.com>
In-Reply-To: <549C5A6B.2000509@towo.net>
Date: Fri, 26 Dec 2014 19:00:38 -0700
Message-Id: <27CE6A0A-9845-4A1C-A0F8-C0236B95A1E3@etr-usa.com>
References: <XnsA40D81CA1FAA8davidrayninfocouk@80.91.229.13> <549B4258.5050509@redhat.com> <XnsA40DECB2AE256davidrayninfocouk@80.91.229.13> <549C5A6B.2000509@towo.net>
To: The Cygwin Mailing List <cygwin@cygwin.com>
Content-Transfer-Encoding: 8bit

On Dec 25, 2014, at 11:41 AM, Thomas Wolff <towo@towo.net> wrote:

> In any case the argument is quite artificial since the new behaviour
> hits many files that are in fact text files.

Please define the term “text file” in a way that allows a C programmer to write a program that automatically does the correct thing for all members of the class “text file” without involving locales, or an equivalent mechanism.

Just because you, the human, can use your superior pattern matching skills to see that a file is a “text file” doesn’t mean that a relatively dumb program like grep(1) can.  You can’t expect someone to build an AI system into grep(1) just to get automatic locale detection.

If grep runs into a byte sequence that makes it think it is not legal for your current locale, it must treat the file as raw bytes, unless you give it -a.

If you don’t like this behavior, say “alias grep=grep -a” in your ~/.bashrc, and forget the change ever happened.  It’ll be on you when some non-text file gets treated as text and grep spams your terminal with binary garbage, though.

This isn’t really a Cygwin problem.  It just happens to affect it more than other *ix systems because there are two sets of rules on the same system, and they may conflict.  But, if I go and copy a UTF-16LE file to a Linux box, I’m not going to complain to the grep bug list when grep doesn’t automatically do the right thing with it while $LANG contains UTF-8.

Ultimately, the proper solution is to use UTF-8 on all systems you use Cygwin on.  Many Unicode-aware native Windows programs that deal with text files can cope with UTF-8.  

If you have one of those that demands UTF-16LE, iconv(1) can make the conversion for you.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple