X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org C5DF43858C33
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1710621688;
	bh=qsBK1Gcpn6ydxbHb+5ReX26vzbFTqZwIX/rFzX/E/ac=;
	h=Date:Subject:To:References:In-Reply-To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	From;
	b=F+wxYpVTOYQD0oWAF4owMv5QV4+UpYt30fzKMwthcgM2pL4FNMVMgxZfUEDRXmAiY
	57NGBSvr4fwaoCQFycSR0E0dKCIVJJ2o0IBKgBekKeq5EogSrAx85juJFejsYhlDVP
	7XCIsO6kn1FVjPh49pilkStAQNRc8OleWXD/iFDY=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org 4646F3858C98
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org 4646F3858C98
ARC-Seal:	i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710621636; cv=none;
	b=kFpj1R4VLTxdprp4H0lNiYYrPfHYZcOWISzjt9+ncaCNlMPGmCoczA31URj3CE4CHyPac99Iuf5P//hQ+Oyp2QJ6Mg7XcRI1DelbnwTvnfBwzVO4Frv04Q0AQ33YHx0jp7ri1gw7A2fO5DOvTEilktmQ8HjerBttMVsA9EmJ+M4=
ARC-Message-Signature:	i=1; a=rsa-sha256; d=sourceware.org; s=key;
	t=1710621636; c=relaxed/simple;
	bh=1hQAlRLdqoNtd2mb/LL/TPE5c6EW9e+sINdx7mM27D0=;
	h=Message-ID:Date:MIME-Version:Subject:To:From;
	b=wJSabPbs++W5PkuUN1j3/5Ri0KnazfxmSmnJlXGTLJXoGBhpC3lPyKWmB2Okds5nT2xNGyL9ZqNR2Tvo2XSrbslx4TSFF9wm0bj5QNhk9pKP39O19o1TEZIveNC9R/7SJRDYVo0KIwq6fBPFm5GAjfF0heXnZRpM2CRDDYEWvGI=
ARC-Authentication-Results:	i=1; server2.sourceware.org
Message-ID:	<9179d07b-71bf-4e66-9dd5-ce5c47e2a89d@SystematicSW.ab.ca>
Date:	Sat, 16 Mar 2024 14:40:31 -0600
MIME-Version:	1.0
User-Agent:	Mozilla Thunderbird
Subject:	Re: The grep 3.11 application when used in perl-regexp mode appears
	to now be broken
To:	cygwin AT cygwin DOT com
References:	<CAEFTnVP=CV_7v=-yRgfsaxk6qbRE=s_LPPZonKPW8CdLiEvfZw AT mail DOT gmail DOT com>
	<115196859 DOT 4162286 DOT 1710616115289 AT mail DOT yahoo DOT com>
Organization:	Systematic Software
In-Reply-To:	<115196859.4162286.1710616115289@mail.yahoo.com>
X-Rspamd-Queue-Id:	33C222F
X-Spam-Status:	No, score=-2.1 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS,
	KAM_SHORT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,
	SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE,
	UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6
X-Rspamd-Server:	rspamout08
X-Stat-Signature:	aanrawnpiexnm3tk34q567fjt1bi9jwg
X-Session-Marker:	427269616E2E496E676C69734053797374656D6174696353572E61622E6361
X-Session-ID:	U2FsdGVkX1/C31Gxfm8K/TELkPWzF4MQM4+n6wAkIxI=
X-HE-Tag:	1710621632-886404
X-HE-Meta:	U2FsdGVkX1/5apfyntKduiPWn3OUy/n0tGJPZB2eN5yA855j12+WpvuaRwgmWb1l9qHxwZ+JmxPAc3yuyyOFxbQM3Qo/8GAQwdU+JC5iku6hebWGKKhDpehH3HqsWwx+WeECflApJ2uz5lg+/ypExelv5lcK4UKdn8n0oHtqXX+K2fZFrk9PcohYsu+tANfzVxnQvZ3C1SXndT2IN3Fc/KkvZ7fGyy63AyGFgRMJGd7FxpOneM7JPW498pDOI5oc1e6jvoJG7m5+fXufYtrFDpQEOp62OVmFt76dGxOjxiDDzEXnQim/YbcF15qYUmwkoqgAYX8qgp3FRN326kUoERl5DnlyUv+BjzORCskvUDQ3SAWR55dylNBfFOGBlSrA4JJWmW7+dFeSlEtgIrc+ES/fnxrvbFQU5IEKoux3S2+DJ9EhfW/KNb632FykkjuDfGStCuIGviOIQFJHaq8UyGgINQCloNQDIRllmlnfGSRRQfjtXihjVVvc2CoD2ctrqtXQsY8XXAYqS8TFk9gDmwZgNUM49Ol394UDIdMuTdLoelnE7AYf0QDRlxAo2g7o+mJIm0hrXCXYgaupzlQTyA==
X-Spam-Checker-Version:	SpamAssassin 3.4.6 (2021-04-09) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<https://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Brian Inglis via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	cygwin AT cygwin DOT com
Cc:	Brian Inglis <Brian DOT Inglis AT SystematicSW DOT ab DOT ca>
Errors-To:	cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 42GKfUs33891694

On 2024-03-16 13:08, Kevin Schnitzius via Cygwin wrote:
> On Saturday, March 16, 2024 at 02:02:31 PM EDT, Michael Goldshteyn via Cygwin wrote:
>> $ grep -c -P '000$' a
>> 0
> 
>> # Now you may be thinking, OK, it's because of the CR/LF line ending
> 
> $ LC_ALL=en_US grep -c --binary-files=text -P '000$' a
> 0
> $ LC_ALL=en_US grep -c --binary-files=text -P '000\r$' a
> 1
> 
> It is the an EOL issue; it is also a bug.
> 
> "By default, under MS-DOS and MS-Windows, grep guesses
> whether a file is text or binary as described for the  --binary-files  option.   If
> grep decides the file is a text file, it strips the CR characters from the original
> file  contents  (to  make  regular  expressions  with  ^  and  $  work  correctly)."
> 
> The current release is not stripping EOL characters correctly in the case of DOS text files.

> On Saturday, March 16, 2024 at 02:02:31 PM EDT, Michael Goldshteyn via Cygwin wrote:
> I just updated my Cygwin64 installation, which includes the grep
> utility and its behavior has changed. It no longer works like it used to
> for Perl reg-ex matching, as demonstrated below:
> 
> Simple test cases:
> ======================
> $ ls -l a
> -rwxr-xr-x 1 Michael None 6 Mar 16 12:15 a
> 
> $ hexdump -C a
> 00000000  31 30 30 30 0d 0a                                |1000..|
> 00000006
> 
> # Notice the CR/LF encoding after the "1000" text, as is the case for DOS
> text files
> 
> # Now let's test grep regular match
> $ grep --version
> grep (GNU grep) 3.11
> Packaged by Cygwin (3.11-1)
> Copyright (C) 2023 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> https://gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> Written by Mike Haertel and others; see
> <https://git.savannah.gnu.org/cgit/grep.git/tree/AUTHORS>.
> 
> grep -P uses PCRE2 10.43 2024-02-16
> 
> $ grep '000' a
> 1000
> 
> # Match using pcre2
> $ grep -P '000' a
> 1000
> 
> # OK, so far so good
> $ grep -P '000$' a
> # No match
> 
> # Put another way
> $ grep -c -P '000$' a
> 0
> 
> # Now you may be thinking, OK, it's because of the CR/LF line ending
> # But, I present the following
> $ pcre2grep --version
> pcre2grep version 10.43 2024-02-16
> 
> $ pcre2grep '000$' a
> 1000
> 
> # As a further cross-check, the same version of the cygpcre2-8-0.Dll is
> used for both grep.exe and pcre2grep.exe, as shown below with an "=>"
> annotation added by me to direct you to the Dll in question:
> 
> $ ldd grep.exe
>          ntdll.dll => /cygdrive/c/Windows/SYSTEM32/ntdll.dll (0x7ffa87d50000)
>          KERNEL32.DLL => /cygdrive/c/Windows/System32/KERNEL32.DLL
> (0x7ffa87700000)
>          KERNELBASE.dll => /cygdrive/c/Windows/System32/KERNELBASE.dll
> (0x7ffa85570000)
>          cygwin1.dll => /usr/bin/cygwin1.dll (0x7ff9c84d0000)
>          cygintl-8.dll => /usr/bin/cygintl-8.dll (0x5ee2d0000)
> =>        cygpcre2-8-0.dll => /usr/bin/cygpcre2-8-0.dll (0x5ec2b0000)
>          cygiconv-2.dll => /usr/bin/cygiconv-2.dll (0x3dff10000)
> 
> $ ldd pcre2grep.exe
>          ntdll.dll => /cygdrive/c/Windows/SYSTEM32/ntdll.dll (0x7ffa87d50000)
>          KERNEL32.DLL => /cygdrive/c/Windows/System32/KERNEL32.DLL
> (0x7ffa87700000)
>          KERNELBASE.dll => /cygdrive/c/Windows/System32/KERNELBASE.dll
> (0x7ffa85570000)
> =>        cygpcre2-8-0.dll => /usr/bin/cygpcre2-8-0.dll (0x5ec2b0000)
>          cygbz2-1.dll => /usr/bin/cygbz2-1.dll (0x3ed560000)
>          cygwin1.dll => /usr/bin/cygwin1.dll (0x7ff9c84d0000)
>          cygz.dll => /usr/bin/cygz.dll (0x5ebb10000)
> 
> # For what it's worth, I also checked into what versions of libintl8 and
> libiconv-2 I have, and these are as follows:
> # libintl8 0.22.4-1
> # libiconv2 1.17-1
> 
> # And as an addition cross-check, I will include the following "complete
> hack":
> $ strings cygintl-8.dll | pcre2grep '^\d\.\d\d'
> 0.22.4
> 0.22.4
> 
> $ strings cygiconv-2.dll | pcre2grep '^\d\.\d\d'
> 1.17
> 1.17
> 
> # For completeness, here is my CYGWIN environment variable setting and some
> other info:
> $ echo "$CYGWIN"
> glob:ignorecase winsymlinks:native pipe_byte
> $ echo "$CYGWIN64_DIR"
> c:\cygwin64
> $ which grep
> /usr/bin/grep
> $ which pcre2grep
> /usr/bin/pcre2grep
> # No aliases are set up for these, either
> $ alias grep pcre2grep
> bash: alias: grep: not found
> bash: alias: pcre2grep: not found
> ======================
> Further comments:
> I do not know with which version of grep.exe this misbehavior (or at least
> misaligned behavior with respect to grep2pcre) of the '-P' switch began. I
> discovered it after updating my Cygwin64 install to use the latest grep
> version, which likely also picked up the latest version of PCRE2 and
> other dependencies along the way.
> 
> Thank you for looking into this and/or providing constructive comments on
> the source of the issue,

You must have updated from very old packages, as this has been the case since 
2017-02 when Cygwin 2.7 was released, with gawk 4.1.4-3, grep 3.0-2, and sed 4.4 
operating strictly according to POSIX:

https://cygwin.com/pipermail/cygwin-announce/2017-February/007795.html
https://cygwin.com/pipermail/cygwin-announce/2017-February/007796.html
https://cygwin.com/pipermail/cygwin-announce/2017-February/007797.html

Similar changes may have occurred in some coreutils around that time.

Since then, all of us with current releases have been using Cygwin text mounts, 
d2u/dos2unix, BRE '\r\?$' ERE '\r?$', adding gawk 'sub(/\r$/,"")', sed 
's/\r$//', to strip CRs before LFs at EoL, if necessary.

Please note that Cygwin does not consider its releases "Windows" versions but 
POSIX versions.

It appears that pcre2 and thus pcre2grep may default line terminators to 
"anycrlf" to work with files from any environment, and is not standardized by 
POSIX as of 202X.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -