X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 5A6003858031
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1637660935;
	bh=IEHlye4YN/qx79B/6o11uEgBvhrE0+DCiQ+Igch663Y=;
	h=Date:To:Subject:References:In-Reply-To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	From;
	b=CEWICnhqYEhWOgcf6XkoLRqMdcZddwxXh5WLnTKVRoc8tq86veHkNPLMNCFLNG+KO
	j0vFY7jevJ/XgoH5cWuw05F8bbl3h8mvpmbvLEEIgcyoV+zcUDYoITI064FNUMdssk
	5sB1GFoxNxWO/krWY2ibI/05jJl3zOmrcC2Q7BtA=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.1 sourceware.org BD8E53858405
Date:	Tue, 23 Nov 2021 10:48:21 +0100
To:	cygwin AT cygwin DOT com
Subject:	Re: possible snprintf() regression in 3.3.2
Message-ID:	<YZy45cStTTgLp0gO@calimero.vinschen.de>
Mail-Followup-To:	cygwin AT cygwin DOT com
References:	<20211118203538 DOT a049809d57731fe375801c15 AT nifty DOT ne DOT jp>
	<YZZSzPhbqF6cQAiu AT calimero DOT vinschen DOT de>
	<fa647f83-c5ed-6dea-fcba-c8b799abe12f AT gmail DOT com>
	<YZZi3rI5msk6ksAV AT calimero DOT vinschen DOT de>
	<7545bb24-43de-cd7d-0764-55c85f1af957 AT gmx DOT com>
	<20211121001613 DOT GH10332 AT venus DOT tony DOT develop-help DOT com>
	<YZtyJDXdrMzVR2lJ AT calimero DOT vinschen DOT de>
	<YZuVRjYoG40cEGDV AT calimero DOT vinschen DOT de>
	<20211122232302 DOT GI10332 AT venus DOT tony DOT develop-help DOT com>
	<20211123173409 DOT 0db4d5ccd94501ce1b8f69ea AT nifty DOT ne DOT jp>
MIME-Version:	1.0
In-Reply-To:	<20211123173409.0db4d5ccd94501ce1b8f69ea@nifty.ne.jp>
X-Provags-ID:	V03:K1:2hRQSh2WgqNSf1Xb9oicxbL6Lcphv8xc8hN4XOkBr5wLzUnynTY
	t1E4QDeGK0YTEO+wqBlORCveikenqbiJaAcsuTGdczo5ps21qtHXsDIFVGKS9fg7UELrBa5
	WDBhw3txJSBwSuLzZmT/Yqxy+quU+XnD09d1jJUkW4e4fxTX6mqIE+ikgCg2TpWPFTmKWzL
	G6HxnwKa7KFs4/NEVTdpw==
X-UI-Out-Filterresults:	notjunk:1;V03:K0:rZ/nccIp5rs=:9+7s5uMaeovFK3Ktswok8s
	6gzInM62acyHTucD3JjZsIQHx0L2Ld3t3m/Y9OSWedTmoZuDfDFopZWB85Ynyb3EAVEpVqRsX
	5LzkzhwfWFzMY+9V8i1lXi3OAm50k6A2PEncLtMgyWO6bhFyX/K8Sh+uhH+bovEobLB1CbFVz
	gcBncIIrTs6w4Tvq9J9nuxL1mXvlfj9rorSyzS82s+YDgmjLXck0RkDdJIECK6LEBK4MlY6M/
	2S/C4KQRvUO42ObabUsM0NodRxTisCDiaDfhL9sybE87lzy08J3/GzCkztcjbVRcxSsWsHvCG
	4Vu53Q15rbAVtHldYm4ZAY2LBVetLOTXlif3VDGvNnka27xrst3ZODMmFHX+HIzXjM47HGN2p
	0D86TM0hb3EZj6j45ScIue6dXgNvcWtuDGlwxfdc6TvKQvrCIfdPV/95BVERf+ysE5rvLDF2W
	jDd6OyS3RlJOkZQppupqCuRfjwIRR9FQoWr7SGwvCQ+IIDZ/PdwROccHqE+W+28D1LBTIptTu
	FT2gECvoz9Kfh8kZbdvqZS3PEgKJ9e2l0DSg8a4yHkXWy8P/uviZ98bPe5NqGxR/PWgiBD+v+
	0h6zwEdDYpvjDdSeD7dFwIwGu0MpeQKI3lYYxTPJl7+hSRs9AhLIJqKW6X7t9tje64rZOH7lR
	7GY742hyDDQHJvYNKmmnXM5kmNjkaixC7QzSaeygtZngZCORU7oi6tb6bIqrk6wOBbkjpbs8A
	MHLvu16j8WLzTPzq
X-Spam-Status:	No, score=-99.0 required=5.0 tests=BAYES_00,
	GOOD_FROM_CORINNA_CYGWIN, KAM_DMARC_NONE, KAM_DMARC_STATUS, KAM_LOTSOFHASH,
	KAM_NUMSUBJECT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,
	SPF_NEUTRAL, TXREP autolearn=ham autolearn_force=no version=3.4.4
X-Spam-Checker-Version:	SpamAssassin 3.4.4 (2020-01-24) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<https://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Corinna Vinschen via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	cygwin AT cygwin DOT com
Cc:	Corinna Vinschen <corinna-cygwin AT cygwin DOT com>
Errors-To:	cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>

On Nov 23 17:34, Takashi Yano via Cygwin wrote:
> On Tue, 23 Nov 2021 10:23:02 +1100
> Tony Cook wrote:
> > On Mon, Nov 22, 2021 at 02:04:06PM +0100, Corinna Vinschen via Cygwin wrote:
> > > On Nov 22 11:34, Corinna Vinschen via Cygwin wrote:
> > > > On Nov 21 11:16, Tony Cook wrote:
> > > > > A simple option would be to use an small auto fixed buffer for most
> > > > > conversions, but use malloc() for %f formats for numbers greater in
> > > > > magnitude than some limit, though it would also need to be adjusted
> > > > > for the precision (ndigits here), since they take extra space.
> > > > > 
> > > > > This would avoid using the optional-to-implement VLA feature too.
> > > > 
> > > > Good idea.  I guess I create a simple fix doing just that.
> > > 
> > > I created a patch:
> > > https://sourceware.org/git/?p=newlib-cygwin.git;a=commitdiff;h=68faeef4be71
> > I don't think this solves the fundamental problem.
> > 
> > Simply looking at ndigits isn't enough for %f.
> > 
> > For %f with a large number (like 9e99), the buffer size required is
> > ndigits plus (roughly) log10(n), which we can further estimate
> > with log2(n)*146/485 (log2(10) is 3.32 ~== 485/146)
> > 
> > I think something more like:
> > 
> >   size_t outsize;
> >   if (mode == 3) {        /* %f */
> >     int expon = (e[NI-1] & 0x7fff) - (EXONE - 1); /* exponent part of float */
> >     /* log2(10) approximately 485/146 */
> >     outsize = expon * 146 / 485 + ndigits + 10;
> >   }
> >   else { /* %g/%e */
> >     outsize = ndigits + MAX_EXP_DIGITS + 10;
> >   }
> >   if (outsize > NDEC_SML) {
> >     outbuf = (char *)_malloc_r(ptr, outsize);
> >   }
> > 
> > You'll probably need to pass outsize into etoasc() rather than
> > calculating it.
> > 
> > See https://github.com/Perl/perl5/blob/blead/sv.c#L13295 for code in
> > perl that calculates the buffer size needed for %f (precis aka ndigits
> > is added at line 13385).
> 
> I guess Corinna thinks that 'ndigits' keeps the total number
> of digits to be printed.

No, I don't.  It's the requested decimal precision.

However, the fun fact is that ldtoa in newlib is more than 20 years old,
with only minor changes in 2003.  My patches don't change the basic
mechanism of ldtoa.  I just don't have enough knowledge of floating
point arithmetic to do that.  My patches only try to raise the number of
*possible* digits by raising the matching macro and raising the size of
the single, local digit buffer accordingly.

If the above crashed, then probably because the buffer was too small.
That should be fixed now, because the second patch fixes the buffer size
and the computation based on the buffer size.  If that's not the
problem, then, in theory, the same would have occured with the old code.

If my patches are inadequate, we can revert the patches and then the
precision will be restricted to 42 digits again, as before, see the
thread https://sourceware.org/pipermail/newlib/2021/018626.html

For everything else, we either need somebody who knows how to change the
current ldtoa to "do the right thing", whatever that is, or somebody who
takes a stab at replacing ldtoa with another, better alternative.

> However, in reality, for example in the case:
> snprintf(buf, sizeof(buf), "%.3f", 1234567890123456.789);
> 'ndigits' is only 3 even though total digits will be 20.
> 
> So, Tony thinks current code does not correct.
> 
> However, I think something is wrong with interpretation
> of 'ndigits' in dltoa.c.
> 
> printf("%.3f\n", sqrt(2)*1e70);
> printf("%.50f\n", sqrt(2)*1e70);
> 
> outputs
> 
> 14142135623730951759073108307330633613786387000000000000000000000000000.000
> 14142135623730951759073108307330633613786386978891021459448717416650727.13402790000888758223149296720949629080194006476078
> 
> Is this as intended?

On Linux I see

14142135623730951759073108307330633613786387161811679011529922516615168.000
14142135623730951759073108307330633613786387161811679011529922516615168.00000000000000000000000000000000000000000000000000

The newlib output for .3f probably suffers from the fact that ldtoa
chooses the small buffer, which restricts the number of digits to 43 or
44.  But keep in mind that ldtoa *always* restricted the output to 42,
so you never got a more precise output anyway.  Every digit beyond digit
42 is only printed due to the bigger buffer sizes.

So, what newlib and, in extension, Cygwin really needs at this point are
patches to the existing ldtoa or a change to gdtoa or equivalent.

https://cygwin.com/acronyms/#SHTDI
https://cygwin.com/acronyms/#PTC


Corinna

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -