X-Recipient: archive-cygwin AT delorie DOT com DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 1ECA63860C35 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com; s=default; t=1596787631; bh=hjgjir54IYiMJnzyvpipR/6wZuXQs/1wPimTocu58SI=; h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=qwvCyaEs1JDD+xmHqNbQyOEQPpRpSDdRxdD3oTWYr+Ml6LZ1GHQLezQGQ+H2DIe9P oDhl3wAgRcDVSpsGITl2B9c27891tvnmQ+z3qH2Qp11OKYaf5+9ILD03u8V1LYhj37 povT8FXotLf1M2O0pANIl9Pfh5NYqXP4DdM0lts0= X-Original-To: cygwin AT cygwin DOT com Delivered-To: cygwin AT cygwin DOT com DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 26EB43858D35 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=dcGwRTf/QrfUgDpt6mvopZaWNE667gD2SD3zAkEotS0=; b=G0Yvp4NM/kJZxegF6OvfyJMxniLcog/KgxwVHC6PbLsvb337BxrCPbXYrpViJq8xNG bqwF2eqPIwBGGxD8K12F/o+rJ5MUwSm7RUCPIa7eX9L83cipxy/k8tilr1XsaW7WbtEr segauW6/e17S7dE/n3BxxVwYrVEkxrW2n/HBvxVOhFYShRgx0WqOKZcKDNNHTG5uOK0u d7nRuCfXCgf8QOhHSiILvz26DXjjtcCOdISJfc1TU2i8hZurLQObCHbnmutyAo41MDae iIIFJdRysX+mPXqTj9ksk2SohDyMZSr1LoIHdetViPwe8zOIBh40V+2wMRalRcN+lvCv z7uw== X-Gm-Message-State: AOAM5332d8o2RadagLW9xG4hnvJ8KicKrPMFaWcuOUcjFb4Qrs58E1U0 AJQMHi7LAgPt47SljtJ406pxN3Lh5JDvXFZS4zjVhdB18vw= X-Google-Smtp-Source: ABdhPJxo0GtS5OH3k1xnbLcNXzE8BWdkPCzNqVGR6njf9lEAOoH6OaQsEskYJeBmDPaI7WSzPc65AtFtUIwISbYK/CA= X-Received: by 2002:a05:6402:1a26:: with SMTP id be6mr7642947edb.162.1596787625881; Fri, 07 Aug 2020 01:07:05 -0700 (PDT) MIME-Version: 1.0 References:

<323895c2-0df4-f8da-791e-03579f2a8caf AT cornell DOT edu> In-Reply-To: <323895c2-0df4-f8da-791e-03579f2a8caf@cornell.edu> Date: Fri, 7 Aug 2020 10:06:54 +0200 Message-ID: Subject: Re: name pipe problem: 1 writer, multiple concurrent readers To: cygwin AT cygwin DOT com X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin AT cygwin DOT com X-Mailman-Version: 2.1.29 List-Id: General Cygwin discussions and problem reports List-Archive: List-Post: List-Help: List-Subscribe: , From: =?utf-8?q?Morten_Kj=C3=A6rulff_via_Cygwin?= Reply-To: =?UTF-8?Q?Morten_Kj=C3=A6rulff?= Content-Type: text/plain; charset="utf-8" Sender: "Cygwin" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 07787eD6020140 On Thu, Jul 16, 2020 at 9:26 PM Ken Brown wrote: > > On 7/3/2020 7:09 AM, Ken Brown via Cygwin wrote: > > On 7/2/2020 1:50 PM, Morten Kjærulff via Cygwin wrote: > >> I think we got a new release around the beginning of June, right? > >> You said that there were still issues (I can confirm). > >> If it can help, here is the output I see today of above scripts: > >> > >> $ ./tp.sh > > [...] > >> 0 [fifo_reader] diff 1806 C:\cygwin\bin\diff.exe: *** fatal > >> error - Can't update my handlers, Win32 error 87 > > > > Thanks for the report and the simple test case. Obviously I still have more > > work to do on this. > > Hi Morten, > > I've attempted to fix the bugs (see > https://cygwin.com/pipermail/cygwin-patches/2020q3/010380.html). With these > patches installed, I no longer see a fatal error or hanging diff processes. But > your script still doesn't work as you expect. On a typical run of the parallel > part, 6 or 7 of the 10 diff processes see the FIFO t.pip as empty. > > Here's a sample run under strace, so that I could see what was going on. 7 of > the 10 diff processes saw t.pip as empty on this run. > > $ strace -o tpip.sh.strace sh -c ./tpip.sh > PID PPID PGID WINPID TTY UID STIME COMMAND > 1307 1306 1307 10932 pty1 197609 17:50:12 /usr/bin/bash > 18426 1 18426 9360 cons0 197609 06:47:31 /usr/bin/sh > 18429 18426 18426 13900 cons0 197609 06:47:32 /usr/bin/ps > 1306 1 1306 3768 ? 197609 17:50:11 /usr/bin/mintty > 18424 1307 18424 21840 pty1 197609 06:47:31 /usr/bin/strace > result1 start > 10 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > result1 end > 0a1,2 > > line1 > > line2 > 0a1,2 > > line1 > > line2 > 0a1,2 > > line1 > > line2 > 0a1,2 > > line1 > > line2 > 0a1,2 > > line1 > > line2 > 0a1,2 > 0a1,2 > > line1 > > line1 > > line2 > > line2 > result2 start > 10 > 0 > 1 > 1 > 0 > 1 > 1 > 1 > 1 > 1 > 0 > result2 end > PID PPID PGID WINPID TTY UID STIME COMMAND > 18480 18430 18426 15580 cons0 197609 06:47:33 /usr/bin/cp > 1307 1306 1307 10932 pty1 197609 17:50:12 /usr/bin/bash > 18484 18426 18426 21264 cons0 197609 06:47:44 /usr/bin/ps > 18430 18426 18426 23472 cons0 197609 06:47:32 /usr/bin/sh > 18426 1 18426 9360 cons0 197609 06:47:31 /usr/bin/sh > 1306 1 1306 3768 ? 197609 17:50:11 /usr/bin/mintty > 18424 1307 18424 21840 pty1 197609 06:47:31 /usr/bin/strace > > I'm attaching your script for ease of reference, and I'm attaching an excerpt > from the strace output, to which I've added a few comments. The excerpt shows > all open, close, read, and write system calls involving t.pip. > > Here's a summary of what you can see from those system calls in the parallel > part of the script. In what follows, I've called the diff processes diff-1, > diff-2,..., diff-10, and similarly for the cp processes (although there are only > four of them). > > 1. cp-1 tries to open t.pip for writing and blocks. It unblocks when diff-1 > opens t.pip for reading, and both processes run to completion as expected. > > 2. diff-2, diff-3, diff-4, and diff-5 try to open t.pip for reading, and they > block until cp-2 opens it for writing. Then cp-2 writes 12 bytes to t.pip and > closes it, and the four diff processes all try to read. diff-4 gets there first > and reads the 12 bytes; it reads once more and sees EOF because there is no data > available in the pipe and there are no writers open[1], so it considers those 12 > bytes to constitute the file t.pip. It later exits with success. > > diff-2, diff-3, and diff-5 all complete their reads before cp-3 opens t.pip. > They see EOF for the same reason as above, so t.pip appears empty and they exit > with failure. > > 3. diff-6, diff-7, diff-8, diff-9, and diff-10 try to open t.pip for reading, > and they block until cp-3 opens it for writing. Then cp-3 writes 12 bytes to > t.pip and closes it, and the five diff processes all try to read. diff-10 gets > there first and reads 12 bytes followed by EOF; it later exits with success. > > diff-6, diff-7, diff-8, and diff-9 all complete their reads before cp-4 opens > t.pip. They see EOF, so t.pip appears empty and they exit with failure. > > 4. cp-4 tries to open t.pip and blocks because there are no more diff processes. > > I've run your script on Linux a few times, and it usually[2] behaves as you > expect, with all diff processes succeeding. For reasons I don't understand, the > diff and cp processes apparently alternate most of the time on Linux, rather > than having 4 or 5 diff processes lumped together between the cp processes as on > Cygwin. > > If someone can figure out the reason for the difference, and if it turns out to > be related to the FIFO code, I could try to modify the code to make Cygwin > behave more like Linux. > > Ken > > [1] From https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html: > > When attempting to read from an empty pipe or FIFO: > > * If no process has the pipe open for writing, read() shall return 0 to > indicate end-of-file. > > [2] But I did have one Linux run in which one of the ten diff processes saw an > empty t.pip and failed as on Cygwin. Hi, Also from https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html: "The behavior of multiple concurrent reads on the same pipe, FIFO, or terminal device is unspecified." Would that be the reason why it also fails on Linux? If I change the script to: #diff t.pip t.txt flock t.pip.lock diff t.pip t.txt It seems to work as I want it to, both on cygwin and Linux. My goal is to convert my ~/.netrc to a pipe. The writer would decrypt an `gpg2 -e --default-recipient-self` encrypted version. As a workaround, I tried wrappers for all commands accessing ~/.netrc, like this for curl: #!/bin/sh flock ~/.netrc.rlock /usr/bin/curl "$@" It seems to work, but not really a fine "solution". Is there a more general way of serializing access to a file/pipe? /Morten -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation: https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple