DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 59564dku1989775
Authentication-Results: delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results: delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter: OpenDKIM Filter v2.11.0 delorie.com 59564dku1989775
Authentication-Results: delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=HXAGxxWs
X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org AD1D53858C60
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1759644277;
	bh=qchCIM7/T/VXTHlKnxnL80FTaJg+OT3bmMldxZXsasY=;
	h=Subject:To:References:Date:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	 From;
	b=HXAGxxWslnJCY31cynkwGI8WYsP7wS5HiITjijfiO1uTuF6gthX/sXzuFtBEuK1bY
	 xSoXK2UpiNefLtblxwwu216ihrSCo4BvxSYqkJ21TbD5pObqrWDGv3ZtACF3FOFm3g
	 JOrRhO0It+dNSfxvIcUjIk/FtzzxMVteBlS3wJ9M=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E70483858D1E
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org E70483858D1E
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1759644218; cv=none;
 b=tTV9EpBUVd5h+Xqo0HrK4uxHnjU55OUNN6bPMAE0IycUbgBrWB0+siS8XmEH2cqJZdf1v8X8lBUpGuGSy5Pkm5IL188pQd2+MqCnMzsVYv/Xa1m8CC6GwSEHwC/9tD1PXZXupxCoJ5YsUzKmy/SHuKGIT9h6kEE/YnnzHTJVxY0=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1759644218; c=relaxed/simple;
 bh=ZhMApsp55icXu5WA77qbCcqgC4Qq/mSyndFq2OGqjSQ=;
 h=DKIM-Signature:Subject:To:From:Message-ID:Date:MIME-Version;
 b=Fj+pNtwcrqtSUYKiyUHTLBPkGADEi6B/NkUew5Nd4Lhwz19zrQqCqfXkcfxgRUm3s+OlbpQH62EChPogSet64VAeOpaRpPBaqVBG8BJXOr7SUzXnNJZC0rX2Jl1/YYmu/j4uDkFEVkPaJ7e2ulLiDCix0yErVuvuaaUiTYVuFoM=
ARC-Authentication-Results: i=1; server2.sourceware.org
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E70483858D1E
Subject: Re: bash hangs when executing a script with a here document larger
 than 64 KiB
To: cygwin AT cygwin DOT com
References: <6844953 DOT 4vTCxPXJkl AT nimes>
 <6f388d6d-e0b1-46a6-a4fb-9f1e008345a7 AT SystematicSW DOT ab DOT ca>
 <aOH61MY8W2lZMWWb AT dimstar DOT local DOT net>
Organization: WiseMo A/S
Message-ID: <7551137c-8ba5-afd8-b7aa-30a18661cc6d@wisemo.com>
Date: Sun, 5 Oct 2025 08:03:34 +0200
X-Mailer: Epyrus/2.1.3
MIME-Version: 1.0
In-Reply-To: <aOH61MY8W2lZMWWb@dimstar.local.net>
Content-Language: en-US
X-Content-Filtered-By: Mailman/MimeDel 2.1.30
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Jakob Bohm via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Jakob Bohm <jb AT wisemo DOT com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 59564dku1989775

On 10/5/2025 6:57 AM, Duncan Roe via Cygwin wrote:
> On Thu, Sep 18, 2025 at 02:17:12PM -0600, cygwin wrote:
>> On 2025-09-18 13:09, Bruno Haible via Cygwin wrote:
>>> Please, can you handle this bug report, specific to Cygwin?
>>> The GNU bash maintainer cannot reproduce it, as he doesn't use Cygwin.
>>>
>>> https://lists.gnu.org/archive/html/bug-bash/2025-09/msg00186.html
>> Confirm it hangs in bash but not with dash/ash:
>>
>> $ dash ./here-toobig.sh
>> here-toobig.sh: error: cannot find input file: 'Makefile.in'
>> $ grep -nC3 echo\ \'BEGIN here-toobig.sh
>> 861-  ac_cs_awk_cr=$ac_cr
>> 862-fi
>> 863-
>> 864:echo 'BEGIN {' >"$ac_tmp/subs1.awk" &&
>> 865-cat >>"$ac_tmp/subs1.awk" <<\_ACAWK &&
>> 866-S["gltests_libgnu_LIBOBJDEPS"]=""
>> 867-S["gltests_libgnu_LTLIBOBJS"]=""
>> $ awk '866 <= FNR && FNR <= 2710' here-toobig.sh | wc -lwmcL
>>     1845    2041   65563   65563     177
>>
>> and Cygwin works with 64KB "pages" and temp buffer sizes.
>>
> This only affects here documents over 64KB by 128 bytes or less. The underlying
> cause is a bug in Windows or Cygwin which damages the build process.
>
> Bash tries to send here document data to the target via a pipe. It has to write
> the entire document to the pipe before any of it get read. Bash checks that the
> system it's running on can accept that much data in a pipe, otherwise bash uses
> a temporary file.
>
> In redir.c, bash first checks that the here document is smaller than the pipe
> capacity calculated at build time. This value is wrong because of the bug.
>
> Bash then re-checks the pipe capacity using fcntl(F_GETPIPE_SZ) if that fcntl is
> available. Under cygwin it is not.
>
> The build process runs psize.aux (built from builtins/psize.c) to get pipe
> capacity. Below is a condensed free-standing version of psize.c (no bash
> includes nor GNU header):-
>
> | /* psize.c - Find pipe size. */
> |
> | /* Copyright (C) 1987, 1991 Free Software Foundation, Inc.
> |    Copyright (C) 2025 Duncan Roe
> | */
> |
> | /*  Write output in 128-byte chunks until we get a sigpipe or write gets an
> |     EPIPE.  Then report how many bytes we wrote.  We assume that this is the
> |     pipe size. */
> |
> | #include <unistd.h>
> | #include <stdio.h>
> | #include <signal.h>
> | #include <errno.h>
> | #include <stdlib.h>
> | #include <string.h>
> | #include <unistd.h>
> |
> | int nw;
> |
> | static void
> | sigpipe (int sig, siginfo_t *siginfo, void *ucontext)
> | {
> |   fprintf (stderr, "%d\n", nw);
> |   exit (0);
> | }
> |
> | int
> | main (int argc, char **argv)
> | {
> |   char buf[128];
> |   register int i;
> |   static struct sigaction act;
> |
> |   for (i = 0; i < 128; i++)
> |     buf[i] = ' ';
> |
> |   memset(&act, 0, sizeof act);
> |   act.sa_sigaction = sigpipe;
> |   act.sa_flags = SA_SIGINFO;
> |   sigemptyset(&act.sa_mask);
> |   sigaddset(&act.sa_mask, SIGPIPE);
> |   sigaction(SIGPIPE, &act, NULL);
> |
> |   nw = 0;
> |   for (;;)
> |     {
> |       int n;
> |       n = write (1, buf, 128);
> |       nw += n;
> |     }
> |   return (0);
> | }
>
> `make` runs this as `psize.aux | sleep 3` which gives plenty of time to fill the
> pipe before getting SIGPIPE when it outputs how much it has written.
>
> To see what goes wrong, first build psize.exe:-
>
> | gcc psize.c -g3 -gdwarf-5 -o psize
>
> Now run psize with a longer sleep time so you can attach gdb to it and enter a
> few commands (I used sleep 300 and killed sleep once gdb was ready)(blank lines
> omitted):-
>
> | Script started on 2025-10-04 21:24:09+10:00 [TERM="xterm-direct" TTY="/dev/pty1" COLUMNS="88" LINES="98"]
> | 21:24:11$ gdb -q -p $(pgrep psize)
> | Attaching to process 9764
> | [New Thread 9764.0x11c0]
> | [New Thread 9764.0x38ac]
> | [New Thread 9764.0x3740]
> | Reading symbols from /home/dunc/tests/psize/psize.exe...
> | (gdb) watch nw
> | Hardware watchpoint 1: nw
> | (gdb) b sigpipe
> | Breakpoint 2 at 0x100401096: file psize.c, line 24.
> | (gdb) c
> | Continuing.
> | [Thread 9764.0x3740 exited with code 0]
> | [Switching to Thread 9764.0x50c]
> | Thread 1 "psize" hit Hardware watchpoint 1: nw
> | Old value = 65536
> | New value = 65664
> | main (argc=1, argv=0xa000004e0) at psize.c:47
> | 47          {
> | (gdb) n
> | 49            n = write (1, buf, 128);
> | (gdb)
> | Thread 1 "psize" received signal SIGPIPE, Broken pipe.
> | 0x00007ffeeefcd574 in ntdll!ZwWaitForSingleObject ()
> |    from /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll
> | (gdb) signal SIGPIPE
> | Continuing with signal SIGPIPE.
> | Thread 1 "psize" hit Breakpoint 2, sigpipe (sig=13, siginfo=0x7ffffbdd0,
> |     ucontext=0x7ffffbf50) at psize.c:24
> | 24        fprintf (stderr, "%d\n", nw);
> | (gdb) n
> | 25        exit (0);
> | (gdb)
> | [Thread 9764.0x50c exited with code 0]
> | [Thread 9764.0x38ac exited with code 0]
> | [Inferior 1 (process 9764) exited normally]
> | (gdb) q
> | 21:27:59$ s
> | exit
> | Script done on 2025-10-04 21:28:17+10:00 [COMMAND_EXIT_CODE="0"]
>
> SIGPIPE interrupts write() which returns 128 as if it had succeeded!
>
> The signal is only delivered to psize after it tries another write().
>
> There has to be a bug in there somewhere.
The bug is in understanding that the design of read(2), write(2), 
fread(3) etc.
all requirethat upon approaching the limit, they must first succeed within
their limit before theycan return 0 or raise a signal to indicate actually
hitting the limit .

Also check if the Cygwin implementation of write() contains a hidden buffer
thatpretends to write beyond the actual kernel level pipe limit, in which
case the testprogram needs to somehow bypass that without becoming
complicated.

>
> If the bug is to persist, I suggest to munge psize.c:-
>
> | *** builtins/psize.c.bu Fri Aug 15 05:56:53 2008
> | --- builtins/psize.c    Sun Oct  5 14:06:50 2025
> | ***************
> | *** 51,57 ****
> |   sigpipe (sig)
> |        int sig;
> |   {
> | !   fprintf (stderr, "%d\n", nw);
> |     exit (0);
> |   }
> |
> | --- 51,57 ----
> |   sigpipe (sig)
> |        int sig;
> |   {
> | !   fprintf (stderr, "%d\n", nw - 128);
> |     exit (0);
> |   }
> |
>
> N.B. blank lines (after removing 1st 2 chars) have 2 trailing spaces.
>
> Cheers ... Duncan.
>
Subtracting 128 (the granularity of the test code), seems very arbitrary 
and kludgy.

An obvious simplification is to disable SIGPIPE and use a loop that ends 
when 0or -1
is returned, like this:

/* The following code was written directly in the mail and not tested */

   /* TODO: ignore SIGPIPE instead of setting a handler */

   nw = 0;
   for(;;)
     {
       i = write (1, buf, 128);
       if (i <= 0)
         break;
       nw += i;
     }
   /* These are the error codes in the POSIX write(3P) man page
    * associated with hitting the pipe limit or similar,
    * maybe this code should accept a few more error codes in
    * real world use for this.
    */
   if (errno == EPIPE || errno == ENOSPC || errno == EAGAIN ||
       errno == EFBIG || errno == ENOBUFS)
     fprintf (stderr, "%d\n", nw);
   else
     perror (argv[0]);

Another simplification would be for psize.c to open its own pipe and
simply never read from it, thus removing any dependency on how the build
time shell and sleep programs handle the pipe.

-- 
Jakob Bohm, CIO, partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Soborg, Denmark. direct: +45 31 13 16 10 
<tel:+4531131610>
This message is only for its intended recipient, delete if misaddressed.
WiseMo - Remote Service Management for PCs, Phones and Embedded

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple