X-Recipient: archive-cygwin@delorie.com X-SWARE-Spam-Status: No, hits=-1.3 required=5.0 tests=AWL,BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE X-Spam-Check-By: sourceware.org MIME-Version: 1.0 In-Reply-To: <4BF55F87.4060407@towo.net> References: <20100520123926.GA1432@onderneming10.xs4all.nl> <4BF55F87.4060407@towo.net> Date: Thu, 20 May 2010 20:46:05 +0300 Message-ID: Subject: Re: sed doesn't like LANG= anymore From: Andy Koppe To: "cygwin@cygwin.com" Content-Type: text/plain; charset=UTF-8 X-IsSubscribed: yes Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Delivered-To: mailing list cygwin@cygwin.com On Thursday, May 20, 2010, Thomas Wolff: > With LANG=anything-unknown, the charmap is set to ASCII, so it works (as there is at least no multibyte character then). Anything above 0x7F is invalid with charset ASCII though (since 1.7.2). But perhaps sed skips the multibyte conversion functions when in the C locale. > Considering the described effect, I doubt that a UTF-8 decoder should swallow an ASCII byte after an incomplete UTF-8 sequence; > it should rather stop at the last UTF-8 sequence byte, and consider any subsequent initial UTF-8 or ASCII byte as a new character. 0xE5 is a valid initial byte of a UTF-8 sequence, hence mbtowc returns -2 ("incomplete") after that and -1 ("invalid") on encountering the following ASCII byte. I think it would be wrong to ignore the encoding error, and it's up to the application to back up and feed in the same byte again if it wants to. Andy -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple