MIME-Version: 1.0
Date: Wed, 29 Jul 2009 17:16:24 +1200
Message-ID: <db95995b0907282216g6f90bcb0t5ee80873a383eb29@mail.gmail.com>
Subject: bash pipeline exit code - possible race condition?
From: David Antliff <david DOT antliff AT gmail DOT com>
To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com

I've noticed a strange problem with bash pipelines in Cygwin that
might indicate some sort of race condition. I cannot reproduce the
problem on a Linux system, but it seems easy to reproduce in Cygwin.

I'm running the following command in a bash script that builds some software:

/usr/bin/make --version | head -1

I then test the exit code ($?) against zero and abort if it's non-zero.

Before this executes I have set:
set -o pipefail

This is to ensure that if any command in a pipeline fails, the entire
command will return a failed error code. I.e. if make returns
non-zero, then $? will be non-zero. Without "pipefail", $? is set to
the exit code of the last process in the pipeline (head), regardless
of the error code of preceding programs (make).

I am doing this because I want the script to log the version of Make
that it is using, as well as verify that the make binary is present.
Most of the time this works fine, $? is zero, and the script continues
happily.

However occasionally $? comes back as the value "141". I think that
this value is coming from make because if I turn pipefail off, the
error code is always zero and never 141, and there's nothing else in
the pipeline. QED.

This is very curious (and causes my script to break occasionally) so I
wrote a little script to test this out:

#!/bin/bash
set -o pipefail
err_count=0
count=0
while : ; do
    count=$((count + 1))
    /usr/bin/make --version | head -1 > /dev/null
    EC=$?
    if (( EC != 0 )) ; then
        err_count=$((err_count+1))
        echo "$count : $EC  ($err_count)"
    fi
done


This prints results like this:
$ ./test-make.sh
84 : 141  (1)
2686 : 141  (2)
10762 : 141  (3)
10767 : 141  (4)
10768 : 141  (5)
10769 : 141  (6)
10770 : 141  (7)
29849 : 141  (8)
30003 : 141  (9)
30004 : 141  (10)
30005 : 141  (11)
30154 : 141  (12)


As you can see, it's not very common, but it does happen - about 1 in
every 3000 in that run.

But I noticed after my screensaver came on that the number of
incidents rockets up to a much higher rate:
30307 : 141  (13)
30523 : 141  (14)
30524 : 141  (15)
30587 : 141  (16)
30588 : 141  (17)
30759 : 141  (18)
30912 : 141  (19)
30914 : 141  (20)
31064 : 141  (21)
31455 : 141  (22)
31519 : 141  (23)
31522 : 141  (24)
31611 : 141  (25)
31612 : 141  (26)
31613 : 141  (27)
31824 : 141  (28)
31972 : 141  (29)
32062 : 141  (30)
32128 : 141  (31)
32669 : 141  (32)
33340 : 141  (33)
33490 : 141  (34)
33579 : 141  (35)
33580 : 141  (36)
33946 : 141  (37)
etc etc etc

I suspected that high system load might exacerbate the problem, so to
test this I tried running a high-CPU-load compilation process (FPGA
synthesis) in parallel, and yes I can confirm that this seems to cause
the incidence rate to increase dramatically.

I don't know what error code 141 means because I'm not really sure
where it's coming from. I downloaded and browsed the GNU make source
but I wasn't able to find anything obvious. If it's any help, I
noticed that hitting ctrl-c sometimes generates error code 130 just
before termination. Maybe it's related to a broken pipe or something.

So it looks like this might be some sort of race condition involving
the bash pipe, since it seems timing-affected.

I've tried this under Linux (Ubuntu 8.04, GNU make 3.81, bash
3.2.39(1)-release, head (GNU coreutils) 6.10) and it does not happen
at all, as far as I can see.


I am running Cygwin on Windows XP (service pack 3).

$ uname -srv
CYGWIN_NT-5.1 1.5.25(0.156/4/2) 2008-06-12 19:34


$ /usr/bin/make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i686-pc-cygwin

$ bash --version
GNU bash, version 3.2.49(22)-release (i686-pc-cygwin)
Copyright (C) 2007 Free Software Foundation, Inc.

$ head --version
head (GNU coreutils) 6.10
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by David MacKenzie and Jim Meyering.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple