| delorie.com/archives/browse.cgi | search |
| DMARC-Filter: | OpenDMARC Filter v1.4.2 delorie.com 6932R1LG3458523 |
| Authentication-Results: | delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com |
| Authentication-Results: | delorie.com; spf=pass smtp.mailfrom=cygwin.com |
| DKIM-Filter: | OpenDKIM Filter v2.11.0 delorie.com 6932R1LG3458523 |
| Authentication-Results: | delorie.com; |
| dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=gYH8X9Vs | |
| X-Recipient: | archive-cygwin AT delorie DOT com |
| DKIM-Filter: | OpenDKIM Filter v2.11.0 sourceware.org E25D64BAE7E9 |
| DKIM-Signature: | v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com; |
| s=default; t=1790994403; | |
| bh=B7IdPDBGRCiNj8NyCg8t3vN/TnsXMuRcDC/IL+uxRj0=; | |
| h=To:References:In-Reply-To:Subject:Date:List-Id:List-Unsubscribe: | |
| List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: | |
| From; | |
| b=gYH8X9VsMEn6G5K+PWIHfkSdJh+OYUo3cnTrYOwdagRcV5VrgpMHlYHuHKQ2MjAFx | |
| wGKsexVA0oY4oYVOgmmcGypTzYq4WQL6Ek3D0NSXGzXUn5NobIXyFQQLDnUoRmQhdu | |
| 8TUJx3KlAHxOD73/4BGG4PV3VozsJsTYrs7X4NG8= | |
| X-Original-To: | cygwin AT cygwin DOT com |
| Delivered-To: | cygwin AT cygwin DOT com |
| DMARC-Filter: | OpenDMARC Filter v1.4.2 sourceware.org 5F3174BA903C |
| ARC-Filter: | OpenARC Filter v1.0.0 sourceware.org 5F3174BA903C |
| ARC-Seal: | i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1790994353; cv=none; |
| b=ZtREztDNpcRRBSoLopGNO7n/is7A6vOMLyXI5A6jDj3dYaYnY0Py+PJZ7HtntjzIh+su3NNZxWxB6njh1OSwhuDiLdShjCrSKOx1Z0wN3+xeY+KSTsVcOK8NKRaltsOa3pMFiBxT+6Sag8BSp7bUJB7JlOSbdId+W1FsKbCJADE= | |
| ARC-Message-Signature: | i=1; a=rsa-sha256; d=sourceware.org; s=key; |
| t=1790994353; c=relaxed/simple; | |
| bh=B7XuYyxuO3DNI30BvjO6OiAe8w9Lp6BNuGY8GzpQVy0=; | |
| h=From:To:Subject:Date:Message-ID:MIME-Version; | |
| b=Jvn5eXgyenASdCikuRU1os0A47LpH7d34vCMdP7QAqgX5B70FkakE5IyQ8PhfwzIsEo3Wc/gqtPuvTifdqZTCwsM6ltjiS3HmDKicNHJ6kV1lrnbmCD4NOyX+ubG5f+OXUUvB0eeEr9Q9kOD0YxPgeVIQjkSfl0HL4SGlBG7qZQ= | |
| ARC-Authentication-Results: | i=1; sourceware.org |
| DKIM-Filter: | OpenDKIM Filter v2.11.0 sourceware.org 5F3174BA903C |
| To: | <Brian DOT Inglis AT SystematicSW DOT ab DOT ca> |
| References: | <000001dd33b2$6a4f2570$3eed7050$@nickpopoff.net> |
| <20260824194051 DOT e61e716cbdbe3a1c246fa51d AT nifty DOT ne DOT jp> | |
| In-Reply-To: | <20260824194051.e61e716cbdbe3a1c246fa51d@nifty.ne.jp> |
| Subject: | =?utf-8?Q?RE:_Regression_in_Cygwin_3.6.9:_?= |
| =?utf-8?Q?Deadlock_when_spawning_many_conc?= | |
| =?utf-8?Q?urrent_Windows-native_processes_?= | |
| =?utf-8?Q?=28nmake.exe=29_=E2=80=94_works_in_3.6.5_1?= | |
| Date: | Fri, 2 Oct 2026 22:25:51 -0400 |
| Message-ID: | <009401dd52de$82792e20$876b8a60$@nickpopoff.net> |
| MIME-Version: | 1.0 |
| X-Mailer: | Microsoft Outlook 16.0 |
| Thread-Index: | AQISnvyw/+o0LnckBBDy9covbFvZ3gIB2zR+AlkR9pc= |
| X-BeenThere: | cygwin AT cygwin DOT com |
| X-Mailman-Version: | 2.1.30 |
| List-Id: | General Cygwin discussions and problem reports <cygwin.cygwin.com> |
| List-Archive: | <https://cygwin.com/pipermail/cygwin/> |
| List-Post: | <mailto:cygwin AT cygwin DOT com> |
| List-Help: | <mailto:cygwin-request AT cygwin DOT com?subject=help> |
| List-Subscribe: | <https://cygwin.com/mailman/listinfo/cygwin>, |
| <mailto:cygwin-request AT cygwin DOT com?subject=subscribe> | |
| From: | Nick Popoff via Cygwin <cygwin AT cygwin DOT com> |
| Reply-To: | Nick Popoff <nick AT nickpopoff DOT net> |
| Cc: | cygwin AT cygwin DOT com |
| Sender: | "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com> |
| X-MIME-Autoconverted: | from base64 to 8bit by delorie.com id 6932R1LG3458523 |
Not sure what you mean by "massive graphic blob". What blob? I reported a comprehensive bug report in great detail. Even, illustrated the problem with real bash code (see below). Here is even a more simple illustration:
Steps to Reproduce
1. Install Cygwin 3.6.9
2. Launch a Cygwin shell
3. Start 15 parallel Windows-native builds, e.g.:
Code
for i in {1..15}; do
(nmake.exe -f Makefile$i &)
done
wait
What can be a more simple illustration. Each makefile above runs hundreds ASN.1 regression tests. They started concurrently and after a certain number of nmake.exe processes running, everything stops, no movement. Cannot be killed with Ctrl-C. Only completely closing the Cygwin console kills the processes. This problem was introduced recently. There was no problem in Cygwin 3.6.5.
Nick
===============
Hi Nick,
Attaching a massive graphic blob with little data does not really highlight
where the issue is.
Perhaps try a simple Cygwin command line (with procps-ng installed):
$ procps -Aeflw --forest | tee issue.log
and feel free to trim the output options, and edit the content to highlight
relevant processes where the issues exist.
FYI: This is the actual script sample that 3.6.9.-1 cannot handle properly:
.........
if [[ $case == all || ( -a "$tmp/$daemon$i.lst" && -s "$tmp/$daemon$i.lst" ) ]]; then
if [[ "$1" == -test ]]; then
[[ -n $ra || -n $bc ]] && Wait
{
time nmake -a -nologo -k -f $mutil/$daemon$i.mak \
makefile=$daemon$i update=no case=$case msdebug=$msdebug dir=. nrt=$nrt $dllver
} >_test$i.log 2>&1 &
found=yes
elif [[ "$1" == -update ]]; then
[[ -n $ra || -n $bc ]] && Wait
{
time nmake -a -nologo -k -f $mutil/$daemon$i.mak \
update=yes case=$case dir=$dir makefile=$daemon$i msdebug=$msdebug nrt=$nrt $dllver
} >_test$i.log 2>&1 &
found=yes
fi
else
rm -f $TMP/$daemon$i.lst
fi
i=$((1+$i))
done
if [[ $found == no && $difs == yes ]]; then
echo "$DAEMON: $(date "$format"): ERROR: Error in makefiles. No tests found."
return 203
elif [[ $difs == no ]]; then
echo "$DAEMON: $(date "$format"): WARNING: No tests found to update."
return 0
fi
wait
.........
Where Wait() is a throttling function that allows to run concurrently a given number (like 15 in this case) of 'nmake' processes, and the low case 'wait' is the bash wait.
Nick
> -----Original Message-----
> From: Takashi Yano <takashi DOT yano AT nifty DOT ne DOT jp>
> Sent: Monday, August 24, 2026 6:41 AM
> To: cygwin AT cygwin DOT com
> Cc: Nick Popoff <nick AT nickpopoff DOT net>
> Subject: Re: Regression in Cygwin 3.6.9: Deadlock when spawning many
> concurrent Windows-native processes (nmake.exe) — works in 3.6.5 1
>
> Thanks for the report.
>
> I guess the patch:
> https://cygwin.com/pipermail/cygwin-patches/2026q3/015286.html
> fixes the issue. Please wait for test version of cygwin, where the patch
> applied. The patch is now under review, so hopefully it will become soon.
>
> On Mon, 24 Aug 2026 06:22:06 -0400
> Nick Popoff via Cygwin <cygwin AT cygwin DOT com> wrote:
>
> > Hello,
> >
> >
> >
> > Regression in Cygwin 3.6.9: Deadlock when spawning many concurrent
> > Windows-native processes (nmake.exe) — works in 3.6.5‑1
> >
> > Version affected: Cygwin 3.6.9 (latest as of August 2026)
> >
> > Version NOT affected: Cygwin 3.6.5‑1 (stable, no issues)
> >
> >
> >
> > Summary
> >
> > Cygwin 3.6.9 introduces a regression in process management that causes
> complete build deadlock when spawning many concurrent Windows-native
> processes (specifically nmake.exe). The same workload has run flawlessly for
> 20+ years on all previous Cygwin versions, including 3.6.5‑1. Under 3.6.9, the
> build freezes once some parallel job limit is reached, leaving dozens of idle
> sh.exe processes and stalled nmake.exe processes. Ctrl‑C kills the Windows
> nmake.exe processes but Cygwin’s sh.exe instances remain stuck, and the
> terminal never returns control. Reverting to 3.6.5‑1 immediately restores
> correct behavior. This is a reproducible regression in Cygwin’s
> process/signal/pipe handling.
> >
> >
> >
> > Environment
> >
> > * Windows Server 2025 (WS2025)
> > * Heavy parallel build workload
> > * 15 concurrent nmake.exe processes
> > * Each nmake.exe spawns many short-lived Windows-native tools
> > * Cygwin shell (sh.exe) orchestrates the build
> > * No Cygwin-native processes involved — all Windows-native
> >
> >
> >
> > Steps to Reproduce
> >
> > 1. Install Cygwin 3.6.9
> > 2. Launch a Cygwin shell
> > 3. Start 15 parallel Windows-native builds, e.g.:
> >
> > Code
> >
> > for i in {1..15}; do
> >
> > (nmake.exe -f Makefile$i &)
> >
> > done
> >
> > wait
> >
> > Each makefile repeatedly spawns short-lived Windows-native tools (ASN.1-
> compiler processes).
> >
> > 4. Observe behavior once all 15 jobs are active.
> >
> >
> >
> > Actual Behavior (3.6.9)
> >
> > * Build freezes completely once a certain parallel limit is reached
> > * All nmake.exe processes become idle
> > * Dozens of idle sh.exe accumulate
> > * No CPU usage
> > * No progress
> > * Ctrl‑C kills nmake.exe but does not kill sh.exe
> > * Cygwin console never returns
> > * Terminal must be forcibly closed
> > * No such behavior in 3.6.5‑1
> >
> > This is consistent with a signal-thread or pipe inheritance deadlock.
> >
> >
> >
> > Expected Behavior (3.6.5‑1 and earlier)
> >
> > * All 15 parallel nmake.exe processes run normally
> > * No idle sh.exe accumulation
> > * No deadlock
> > * Ctrl‑C cleanly terminates all processes
> > * Console returns immediately
> > * Build completes successfully
> >
> > This behavior has been stable for decades.
> >
> >
> >
> > Notes
> >
> > This workload is a strong stress test for:
> >
> > * spawn.cc
> > * sigproc.cc
> > * pipe inheritance
> > * process table updates
> > * console control handling
> > * synchronization primitives
> >
> > The regression appears only in 3.6.9.
> >
> >
> >
> > Conclusion
> >
> > Cygwin 3.6.9 introduces a reproducible deadlock when orchestrating many
> concurrent Windows-native processes via Cygwin’s shell. This is a regression
> from 3.6.5‑1. The issue affects real-world build systems and should be
> investigated.
> >
> > I can provide:
> >
> > * process snapshots
> > * strace logs
> > * Windows process tree dumps
> > * minimal reproduction scripts
> >
> > upon request.
> >
> >
> >
> > Nick
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > Problem reports: https://cygwin.com/problems.html
> > FAQ: https://cygwin.com/faq/
> > Documentation: https://cygwin.com/docs.html
> > Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
>
>
> --
> Takashi Yano <takashi DOT yano AT nifty DOT ne DOT jp>
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation: https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
| webmaster | delorie software privacy |
| Copyright © 2026 by DJ Delorie | Updated Sep 20 2026 |