Mail Archives: cygwin/2020/08/07/04:07:58
| X-Recipient: | archive-cygwin AT delorie DOT com
|
| DKIM-Filter: | OpenDKIM Filter v2.11.0 sourceware.org 1ECA63860C35
|
| DKIM-Signature: | v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
|
| s=default; t=1596787631;
|
| bh=hjgjir54IYiMJnzyvpipR/6wZuXQs/1wPimTocu58SI=;
|
| h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe:
|
| List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
|
| From;
|
| b=qwvCyaEs1JDD+xmHqNbQyOEQPpRpSDdRxdD3oTWYr+Ml6LZ1GHQLezQGQ+H2DIe9P
|
| oDhl3wAgRcDVSpsGITl2B9c27891tvnmQ+z3qH2Qp11OKYaf5+9ILD03u8V1LYhj37
|
| povT8FXotLf1M2O0pANIl9Pfh5NYqXP4DdM0lts0=
|
| X-Original-To: | cygwin AT cygwin DOT com
|
| Delivered-To: | cygwin AT cygwin DOT com
|
| DMARC-Filter: | OpenDMARC Filter v1.3.2 sourceware.org 26EB43858D35
|
| X-Google-DKIM-Signature: | v=1; a=rsa-sha256; c=relaxed/relaxed;
|
| d=1e100.net; s=20161025;
|
| h=x-gm-message-state:mime-version:references:in-reply-to:from:date
|
| :message-id:subject:to:content-transfer-encoding;
|
| bh=dcGwRTf/QrfUgDpt6mvopZaWNE667gD2SD3zAkEotS0=;
|
| b=G0Yvp4NM/kJZxegF6OvfyJMxniLcog/KgxwVHC6PbLsvb337BxrCPbXYrpViJq8xNG
|
| bqwF2eqPIwBGGxD8K12F/o+rJ5MUwSm7RUCPIa7eX9L83cipxy/k8tilr1XsaW7WbtEr
|
| segauW6/e17S7dE/n3BxxVwYrVEkxrW2n/HBvxVOhFYShRgx0WqOKZcKDNNHTG5uOK0u
|
| d7nRuCfXCgf8QOhHSiILvz26DXjjtcCOdISJfc1TU2i8hZurLQObCHbnmutyAo41MDae
|
| iIIFJdRysX+mPXqTj9ksk2SohDyMZSr1LoIHdetViPwe8zOIBh40V+2wMRalRcN+lvCv
|
| z7uw==
|
| X-Gm-Message-State: | AOAM5332d8o2RadagLW9xG4hnvJ8KicKrPMFaWcuOUcjFb4Qrs58E1U0
|
| AJQMHi7LAgPt47SljtJ406pxN3Lh5JDvXFZS4zjVhdB18vw=
|
| X-Google-Smtp-Source: | ABdhPJxo0GtS5OH3k1xnbLcNXzE8BWdkPCzNqVGR6njf9lEAOoH6OaQsEskYJeBmDPaI7WSzPc65AtFtUIwISbYK/CA=
|
| X-Received: | by 2002:a05:6402:1a26:: with SMTP id
|
| be6mr7642947edb.162.1596787625881;
|
| Fri, 07 Aug 2020 01:07:05 -0700 (PDT)
|
| MIME-Version: | 1.0
|
| References: | <CA+7cx1p-vH6D9MsQ=zhx2nNDU+8S_oCXmpvNeMp0rwMr2L3RFw AT mail DOT gmail DOT com>
|
| <a9ffd5f2-e5f6-4d36-95ba-0ad2a7ad11e9 AT cornell DOT edu>
|
| <CA+7cx1pjvJOzMM3bwspdWGHowC+dSkPx+SrOH6wtyRO367hCJQ AT mail DOT gmail DOT com>
|
| <c05337b7-e422-2589-882a-46557e2e9ed1 AT cornell DOT edu>
|
| <CA+7cx1o2huaWH6s4cuwzyYQVuFgAJNBsi_bVWDdCzn-W-EAhUQ AT mail DOT gmail DOT com>
|
| <e3077fac-26a6-fca2-3c3a-53a47a947b9c AT cornell DOT edu>
|
| <323895c2-0df4-f8da-791e-03579f2a8caf AT cornell DOT edu>
|
| In-Reply-To: | <323895c2-0df4-f8da-791e-03579f2a8caf@cornell.edu>
|
| Date: | Fri, 7 Aug 2020 10:06:54 +0200
|
| Message-ID: | <CA+7cx1pwmX6R3e5crWxfSY15xtM6g7=8T-VY0xFzwWGD1A2-EA@mail.gmail.com>
|
| Subject: | Re: name pipe problem: 1 writer, multiple concurrent readers
|
| To: | cygwin AT cygwin DOT com
|
| X-Spam-Status: | No, score=-0.7 required=5.0 tests=BAYES_00, DKIM_SIGNED,
|
| DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,
|
| SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2
|
| X-Spam-Checker-Version: | SpamAssassin 3.4.2 (2018-09-13) on
|
| server2.sourceware.org
|
| X-BeenThere: | cygwin AT cygwin DOT com
|
| X-Mailman-Version: | 2.1.29
|
| List-Id: | General Cygwin discussions and problem reports <cygwin.cygwin.com>
|
| List-Archive: | <https://cygwin.com/pipermail/cygwin/>
|
| List-Post: | <mailto:cygwin AT cygwin DOT com>
|
| List-Help: | <mailto:cygwin-request AT cygwin DOT com?subject=help>
|
| List-Subscribe: | <https://cygwin.com/mailman/listinfo/cygwin>,
|
| <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
|
| From: | =?utf-8?q?Morten_Kj=C3=A6rulff_via_Cygwin?= <cygwin AT cygwin DOT com>
|
| Reply-To: | =?UTF-8?Q?Morten_Kj=C3=A6rulff?= <mortenkjarulff AT gmail DOT com>
|
| Sender: | "Cygwin" <cygwin-bounces AT cygwin DOT com>
|
| X-MIME-Autoconverted: | from base64 to 8bit by delorie.com id 07787eD6020140
|
On Thu, Jul 16, 2020 at 9:26 PM Ken Brown wrote:
>
> On 7/3/2020 7:09 AM, Ken Brown via Cygwin wrote:
> > On 7/2/2020 1:50 PM, Morten Kjærulff via Cygwin wrote:
> >> I think we got a new release around the beginning of June, right?
> >> You said that there were still issues (I can confirm).
> >> If it can help, here is the output I see today of above scripts:
> >>
> >> $ ./tp.sh
> > [...]
> >> 0 [fifo_reader] diff 1806 C:\cygwin\bin\diff.exe: *** fatal
> >> error - Can't update my handlers, Win32 error 87
> >
> > Thanks for the report and the simple test case. Obviously I still have more
> > work to do on this.
>
> Hi Morten,
>
> I've attempted to fix the bugs (see
> https://cygwin.com/pipermail/cygwin-patches/2020q3/010380.html). With these
> patches installed, I no longer see a fatal error or hanging diff processes. But
> your script still doesn't work as you expect. On a typical run of the parallel
> part, 6 or 7 of the 10 diff processes see the FIFO t.pip as empty.
>
> Here's a sample run under strace, so that I could see what was going on. 7 of
> the 10 diff processes saw t.pip as empty on this run.
>
> $ strace -o tpip.sh.strace sh -c ./tpip.sh
> PID PPID PGID WINPID TTY UID STIME COMMAND
> 1307 1306 1307 10932 pty1 197609 17:50:12 /usr/bin/bash
> 18426 1 18426 9360 cons0 197609 06:47:31 /usr/bin/sh
> 18429 18426 18426 13900 cons0 197609 06:47:32 /usr/bin/ps
> 1306 1 1306 3768 ? 197609 17:50:11 /usr/bin/mintty
> 18424 1307 18424 21840 pty1 197609 06:47:31 /usr/bin/strace
> result1 start
> 10
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> result1 end
> 0a1,2
> > line1
> > line2
> 0a1,2
> > line1
> > line2
> 0a1,2
> > line1
> > line2
> 0a1,2
> > line1
> > line2
> 0a1,2
> > line1
> > line2
> 0a1,2
> 0a1,2
> > line1
> > line1
> > line2
> > line2
> result2 start
> 10
> 0
> 1
> 1
> 0
> 1
> 1
> 1
> 1
> 1
> 0
> result2 end
> PID PPID PGID WINPID TTY UID STIME COMMAND
> 18480 18430 18426 15580 cons0 197609 06:47:33 /usr/bin/cp
> 1307 1306 1307 10932 pty1 197609 17:50:12 /usr/bin/bash
> 18484 18426 18426 21264 cons0 197609 06:47:44 /usr/bin/ps
> 18430 18426 18426 23472 cons0 197609 06:47:32 /usr/bin/sh
> 18426 1 18426 9360 cons0 197609 06:47:31 /usr/bin/sh
> 1306 1 1306 3768 ? 197609 17:50:11 /usr/bin/mintty
> 18424 1307 18424 21840 pty1 197609 06:47:31 /usr/bin/strace
>
> I'm attaching your script for ease of reference, and I'm attaching an excerpt
> from the strace output, to which I've added a few comments. The excerpt shows
> all open, close, read, and write system calls involving t.pip.
>
> Here's a summary of what you can see from those system calls in the parallel
> part of the script. In what follows, I've called the diff processes diff-1,
> diff-2,..., diff-10, and similarly for the cp processes (although there are only
> four of them).
>
> 1. cp-1 tries to open t.pip for writing and blocks. It unblocks when diff-1
> opens t.pip for reading, and both processes run to completion as expected.
>
> 2. diff-2, diff-3, diff-4, and diff-5 try to open t.pip for reading, and they
> block until cp-2 opens it for writing. Then cp-2 writes 12 bytes to t.pip and
> closes it, and the four diff processes all try to read. diff-4 gets there first
> and reads the 12 bytes; it reads once more and sees EOF because there is no data
> available in the pipe and there are no writers open[1], so it considers those 12
> bytes to constitute the file t.pip. It later exits with success.
>
> diff-2, diff-3, and diff-5 all complete their reads before cp-3 opens t.pip.
> They see EOF for the same reason as above, so t.pip appears empty and they exit
> with failure.
>
> 3. diff-6, diff-7, diff-8, diff-9, and diff-10 try to open t.pip for reading,
> and they block until cp-3 opens it for writing. Then cp-3 writes 12 bytes to
> t.pip and closes it, and the five diff processes all try to read. diff-10 gets
> there first and reads 12 bytes followed by EOF; it later exits with success.
>
> diff-6, diff-7, diff-8, and diff-9 all complete their reads before cp-4 opens
> t.pip. They see EOF, so t.pip appears empty and they exit with failure.
>
> 4. cp-4 tries to open t.pip and blocks because there are no more diff processes.
>
> I've run your script on Linux a few times, and it usually[2] behaves as you
> expect, with all diff processes succeeding. For reasons I don't understand, the
> diff and cp processes apparently alternate most of the time on Linux, rather
> than having 4 or 5 diff processes lumped together between the cp processes as on
> Cygwin.
>
> If someone can figure out the reason for the difference, and if it turns out to
> be related to the FIFO code, I could try to modify the code to make Cygwin
> behave more like Linux.
>
> Ken
>
> [1] From https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html:
>
> When attempting to read from an empty pipe or FIFO:
>
> * If no process has the pipe open for writing, read() shall return 0 to
> indicate end-of-file.
>
> [2] But I did have one Linux run in which one of the ten diff processes saw an
> empty t.pip and failed as on Cygwin.
Hi,
Also from https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html:
"The behavior of multiple concurrent reads on the same pipe, FIFO, or
terminal device is unspecified."
Would that be the reason why it also fails on Linux?
If I change the script to:
#diff t.pip t.txt
flock t.pip.lock diff t.pip t.txt
It seems to work as I want it to, both on cygwin and Linux.
My goal is to convert my ~/.netrc to a pipe. The writer would decrypt
an `gpg2 -e --default-recipient-self` encrypted version.
As a workaround, I tried wrappers for all commands accessing ~/.netrc,
like this for curl:
#!/bin/sh
flock ~/.netrc.rlock /usr/bin/curl "$@"
It seems to work, but not really a fine "solution".
Is there a more general way of serializing access to a file/pipe?
/Morten
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation: https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
- Raw text -