Mail Archives: cygwin/2008/07/25/13:12:18
--_----------=_1216915906174220
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="ISO-8859-1"
MIME-Version: 1.0
X-Mailer: MessagingEngine.com Webmail Interface
Date: Thu, 24 Jul 2008 16:11:46 UT
Ran into a weird issue using python on cygwin. When trying to use the
stdout handle from a subprocess spawned using the subprocess.Popen
class, it hangs when accessed via a thread. This behavior does /not/
occur on other platforms -- nor when using os.popen from threads on
cygwin. Nor when using subprocess.Popen from cygwin without threads.
I know I could avoid this by using subprocess.Popen(...).communicate(),
but in my /real/ application, I want line-by-line access to the sub's
output, whereas communicate() gobbles the entire output and returns a
list of strings for stdout, and for stderr. From the help text:
communicate() returns a tuple (stdout, stderr).
Note: The data read is buffered in memory, so do not use this
method if the data size is large or unlimited.
So, I don't want to do /that/.
I've attached a test case to demonstrate the issue. I was hoping (a)
somebody has seen this before, and has a simple solution for me, or (b)
the python maintainer (Jason?) could take a look, and maybe raise a bug
report upstream.
I'm using 2.5.1-2, but I see the same behavior on cygwin using 2.5.2.
When run on native windows, this test case requires Fping.exe
(http://www.kwakkelflap.com/fping.html) because -- at least on Vista --
the native ping is b0rked. When native ping is called on Vista from a
threaded context, if a host is unreachable, it reports summary info for
the previously-accessed reachable host. Very strange -- but not related
to the problem at hand. For cygwin, of course, the test case uses
/usr/bin/ping.
Test case --help output:
Usage: ./testprog.py [htsS]
Tests os.popen vs. subprocess.Popen for various platforms
On native windows: requires Fping.exe
(http://www.kwakkelflap.com/fping.html)
In operation, this program 'pings' ten IP addresses in the current
network, including localhost. It is used to demonstrate that the
.stdout member of subprocess.Popen() does not work on Cygwin when
using threads
-h, --help print this message
-t, --threads use threads (default is seqential)
-s, --subprocess use subprocess.Popen().stdout (default is os.popen)
-S, --shell when using subprocess.Popen(), execute the target
using
the shell (/bin/sh or cmd.exe), just like os.popen
does.
default for subprocess.Popen is to use an
execv-style list.
Behavior: native windows cygwin
unix
<no args> (os.popen, sequential) OK OK
OK
-t (os.popen, threaded) OK OK
OK
-s -S (subprocess, sequential, via shell) OK OK
OK
-s (subprocess, sequential, execv) OK OK
OK
-t -s -S (subprocess, threaded, via shell) OK hangs
OK
-t -s (subprocess, threaded, execv) OK hangs
OK
--_----------=_1216915906174220
Content-Disposition: attachment; filename="testprog.py"
Content-Transfer-Encoding: base64
Content-Type: text/x-python; name="testprog.py"
MIME-Version: 1.0
X-Mailer: MessagingEngine.com Webmail Interface
Date: Thu, 24 Jul 2008 16:11:46 UT
IyEvdXNyL2Jpbi9weXRob24KIiIiClVzYWdlOiAlcyBbaHRzU10KClRlc3Rz
IG9zLnBvcGVuIHZzLiBzdWJwcm9jZXNzLlBvcGVuIGZvciB2YXJpb3VzIHBs
YXRmb3JtcwpPbiBuYXRpdmUgd2luZG93czogcmVxdWlyZXMgRnBpbmcuZXhl
IChodHRwOi8vd3d3Lmt3YWtrZWxmbGFwLmNvbS9mcGluZy5odG1sKQpJbiBv
cGVyYXRpb24sIHRoaXMgcHJvZ3JhbSAncGluZ3MnIHRlbiBJUCBhZGRyZXNz
ZXMgaW4gdGhlIGN1cnJlbnQKbmV0d29yaywgaW5jbHVkaW5nIGxvY2FsaG9z
dC4gIEl0IGlzIHVzZWQgdG8gZGVtb25zdHJhdGUgdGhhdCB0aGUKLnN0ZG91
dCBtZW1iZXIgb2Ygc3VicHJvY2Vzcy5Qb3BlbigpIGRvZXMgbm90IHdvcmsg
b24gQ3lnd2luIHdoZW4KdXNpbmcgdGhyZWFkcwoKICAtaCwgLS1oZWxwICAg
ICAgICBwcmludCB0aGlzIG1lc3NhZ2UKICAtdCwgLS10aHJlYWRzICAgICB1
c2UgdGhyZWFkcyAoZGVmYXVsdCBpcyBzZXFlbnRpYWwpCiAgLXMsIC0tc3Vi
cHJvY2VzcyAgdXNlIHN1YnByb2Nlc3MuUG9wZW4oKS5zdGRvdXQgKGRlZmF1
bHQgaXMgb3MucG9wZW4pCiAgLVMsIC0tc2hlbGwgICAgICAgd2hlbiB1c2lu
ZyBzdWJwcm9jZXNzLlBvcGVuKCksIGV4ZWN1dGUgdGhlIHRhcmdldCB1c2lu
ZwogICAgICAgICAgICAgICAgICAgIHRoZSBzaGVsbCAoL2Jpbi9zaCBvciBj
bWQuZXhlKSwganVzdCBsaWtlIG9zLnBvcGVuIGRvZXMuCiAgICAgICAgICAg
ICAgICAgICAgZGVmYXVsdCBmb3Igc3VicHJvY2Vzcy5Qb3BlbiBpcyB0byB1
c2UgYW4gZXhlY3Ytc3R5bGUgbGlzdC4KCkJlaGF2aW9yOiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIG5hdGl2ZSB3aW5kb3dzICBj
eWd3aW4gIHVuaXgKICA8bm8gYXJncz4gKG9zLnBvcGVuLCBzZXF1ZW50aWFs
KSAgICAgICAgICAgICAgICAgICBPSyAgICAgICAgICAgT0sgICAgIE9LCiAg
LXQgICAgICAgIChvcy5wb3BlbiwgdGhyZWFkZWQpICAgICAgICAgICAgICAg
ICAgICAgT0sgICAgICAgICAgIE9LICAgICBPSwogIC1zIC1TICAgICAoc3Vi
cHJvY2Vzcywgc2VxdWVudGlhbCwgdmlhIHNoZWxsKSAgICAgIE9LICAgICAg
ICAgICBPSyAgICAgT0sKICAtcyAgICAgICAgKHN1YnByb2Nlc3MsIHNlcXVl
bnRpYWwsIGV4ZWN2KSAgICAgICAgICBPSyAgICAgICAgICAgT0sgICAgIE9L
CiAgLXQgLXMgLVMgIChzdWJwcm9jZXNzLCB0aHJlYWRlZCwgdmlhIHNoZWxs
KSAgICAgICAgT0sgICAgICAgICAgaGFuZ3MgICBPSwogIC10IC1zICAgICAo
c3VicHJvY2VzcywgdGhyZWFkZWQsIGV4ZWN2KSAgICAgICAgICAgIE9LICAg
ICAgICAgIGhhbmdzICAgT0sgCiIiIiAKaW1wb3J0IG9zCmltcG9ydCByZQpp
bXBvcnQgdGltZQppbXBvcnQgc3lzCmltcG9ydCBnZXRvcHQKZnJvbSB0aHJl
YWRpbmcgaW1wb3J0IFRocmVhZAppbXBvcnQgc29ja2V0CmltcG9ydCBzdHJ1
Y3QKaW1wb3J0IHN1YnByb2Nlc3MKCiMgTk9URTogT1BUX3VzZV9zaGVsbCBp
cyBhbHdheXMgdHJlYXRlZCBhcyBUcnVlIGlmIE9QVF91c2Vfc3VicHJvY2Vz
cyBpcyBGYWxzZQojICh0aGF0IGlzLCBvcy5wb3BlbiBhbHdheXMgdXNlcyB0
aGUgc2hlbGwhKQpPUFRfdXNlX3RocmVhZHM9RmFsc2UKT1BUX3VzZV9zdWJw
cm9jZXNzPUZhbHNlCk9QVF91c2Vfc2hlbGw9RmFsc2UKCmlmIHN5cy5wbGF0
Zm9ybSA9PSAnd2luMzInOgogICAgZGVmIHBpbmdfY21kbGluZShpcCwgc2hl
bGw9RmFsc2UpOgogICAgICAgICMgd2luZG93cyBwaW5nIGlzIGIwcmtlZCBv
biBWaXN0YSwgc28gd2UncmUgZm9yY2VkIHRvIHVzZSBhIGRpZmZlcmVudAog
ICAgICAgICMgcGluZyBwcm9ncmFtLiBDYW4ndCB1c2UgY3lnd2luIHBpbmcs
IG11c3QgYmUgYSBuYXRpdmUgcHJvZ3JhbS4KICAgICAgICAjICAgIEZwaW5n
IGZyb20gaHR0cDovL3d3dy5rd2Fra2VsZmxhcC5jb20vZnBpbmcuaHRtbCBz
ZWVtcyBwcmV0dHkKICAgICAgICAjIGdvb2QsIGlzIHZpcnVzIGZyZWUsIGFu
ZCBpcyBub3QgYjBya2VkLiAKICAgICAgICBpZiBzaGVsbDoKICAgICAgICAg
ICAgcmV0dXJuICJGcGluZy5leGUgIiArIGlwICsgIiAtaSAtbiAyIgogICAg
ICAgIHJldHVybiBbIkZwaW5nLmV4ZSAiLCBpcCwgIi1pIiwgIi1uIiwgIjIi
XQplbGlmIHN5cy5wbGF0Zm9ybSA9PSAnY3lnd2luJzoKICAgIGRlZiBwaW5n
X2NtZGxpbmUoaXAsIHNoZWxsPUZhbHNlKToKICAgICAgICAjIGN5Z3dpbidz
IHBpbmcgaXMgYWxzbyBvZGQKICAgICAgICBpZiBzaGVsbDoKICAgICAgICAg
ICAgcmV0dXJuICJwaW5nLmV4ZSAtcSAiICsgaXAgKyAiIDU2IDIiCiAgICAg
ICAgcmV0dXJuIFsicGluZy5leGUiLCAiLXEiLCBpcCwgIjU2IiwgIjIiXQpl
bHNlOgogICAgZGVmIHBpbmdfY21kbGluZShpcCwgc2hlbGw9RmFsc2UpOgog
ICAgICAgIGlmIHNoZWxsOgogICAgICAgICAgICByZXR1cm4gInBpbmcgLXEg
LWMyICIgKyBpcAogICAgICAgIHJldHVybiBbInBpbmciLCAiLXEiLCAiLWMy
IiwgaXBdCgpkZWYgZ2V0X3BpcGVfc3VicHJvY2Vzc19Qb3BlbihpcCk6CiAg
ICBnbG9iYWwgT1BUX3VzZV9zaGVsbAogICAgcHJpbnQgPj5zeXMuc3RkZXJy
LCAiVXNpbmcgc3VicHJvY2Vzcy5Qb3BlbiAoc2hlbGw9JXMpIiAlIE9QVF91
c2Vfc2hlbGwKICAgIGNobGRfc3RkaW4gPSBvcy5vcGVuKG9zLmRldm51bGws
IG9zLk9fUkRPTkxZKTsKICAgIHJldHVybiBzdWJwcm9jZXNzLlBvcGVuKHBp
bmdfY21kbGluZShpcCwgc2hlbGw9T1BUX3VzZV9zaGVsbCksCiAgICAgICAg
c2hlbGw9T1BUX3VzZV9zaGVsbCwKICAgICAgICBzdGRpbj1jaGxkX3N0ZGlu
LAogICAgICAgIHN0ZG91dD1zdWJwcm9jZXNzLlBJUEUsCiAgICAgICAgc3Rk
ZXJyPXN1YnByb2Nlc3MuU1RET1VUKS5zdGRvdXQKCmRlZiBnZXRfcGlwZV9v
c19wb3BlbihpcCk6CiAgICBnbG9iYWwgT1BUX3VzZV9zaGVsbAogICAgcHJp
bnQgPj5zeXMuc3RkZXJyLCAiVXNpbmcgb3MucG9wZW4gKHNoZWxsPSVzKSIg
JSBPUFRfdXNlX3NoZWxsCiAgICByZXR1cm4gb3MucG9wZW4ocGluZ19jbWRs
aW5lKGlwLCBzaGVsbD1PUFRfdXNlX3NoZWxsKSwgInIiKQoKZGVmIGdldF9w
aXBlKGlwKToKICAgIGdsb2JhbCBPUFRfdXNlX3N1YnByb2Nlc3MKICAgIGlm
IE9QVF91c2Vfc3VicHJvY2VzczoKICAgICAgICByZXR1cm4gZ2V0X3BpcGVf
c3VicHJvY2Vzc19Qb3BlbihpcCkKICAgIGVsc2U6CiAgICAgICAgcmV0dXJu
IGdldF9waXBlX29zX3BvcGVuKGlwKQoKY2xhc3MgdGVzdGl0KFRocmVhZCk6
CiAgIGRlZiBfX2luaXRfXyAoc2VsZixpcCk6CiAgICAgIFRocmVhZC5fX2lu
aXRfXyhzZWxmKQogICAgICBzZWxmLmlwID0gaXAKICAgICAgc2VsZi5zdGF0
dXMgPSAtMQogICBkZWYgcnVuKHNlbGYpOgogICAgICBwaW5nYWxpbmcgPSBn
ZXRfcGlwZShzZWxmLmlwKQogICAgICB3aGlsZSAxOgogICAgICAgIGxpbmUg
PSBwaW5nYWxpbmcucmVhZGxpbmUoKQogICAgICAgIGlmIG5vdCBsaW5lOiBi
cmVhawogICAgICAgIGlnb3QgPSByZS5maW5kYWxsKHRlc3RpdC5saWZlbGlu
ZSxsaW5lKQogICAgICAgIGlmIGlnb3Q6CiAgICAgICAgICAgc2VsZi5zdGF0
dXMgPSBpbnQoaWdvdFswXSkKCmlmIHN5cy5wbGF0Zm9ybSA9PSAnd2luMzIn
OgogICAgdGVzdGl0LmxpZmVsaW5lID0gcmUuY29tcGlsZShyIlJlY2VpdmVk
ID0gKFxkKSIpCiAgICBsaWZlbGluZSA9IHJlLmNvbXBpbGUociJSZWNlaXZl
ZCA9IChcZCkiKQplbGlmIHN5cy5wbGF0Zm9ybSA9PSAnY3lnd2luJzoKICAg
IHRlc3RpdC5saWZlbGluZSA9IHJlLmNvbXBpbGUociIoXGQpIHBhY2tldHMg
cmVjZWl2ZWQiKQogICAgbGlmZWxpbmUgPSByZS5jb21waWxlKHIiKFxkKSBw
YWNrZXRzIHJlY2VpdmVkIikKZWxzZToKICAgIHRlc3RpdC5saWZlbGluZSA9
IHJlLmNvbXBpbGUociIoXGQpIHJlY2VpdmVkIikKICAgIGxpZmVsaW5lID0g
cmUuY29tcGlsZShyIihcZCkgcmVjZWl2ZWQiKQpyZXBvcnQgPSAoIk5vIHJl
c3BvbnNlIiwiUGFydGlhbCBSZXNwb25zZSIsIkFsaXZlIiwiSW50ZXJuYWwg
RXJyb3IiKQoKCmRlZiBuYWl2ZV9nZXRfaXBfYWRkcigpOgogICAgcmV0dXJu
IHNvY2tldC5nZXRob3N0YnluYW1lKHNvY2tldC5nZXRob3N0bmFtZSgpKQoK
ZGVmIGRvdHRlZFF1YWRUb051bShpcCk6CiAgICAiY29udmVydCBkZWNpbWFs
IGRvdHRlZCBxdWFkIHN0cmluZyB0byBsb25nIGludGVnZXIiCiAgICByZXR1
cm4gc3RydWN0LnVucGFjaygnIUwnLHNvY2tldC5pbmV0X2F0b24oaXApKVsw
XQoKZGVmIG51bVRvRG90dGVkUXVhZChuKToKICAgICJjb252ZXJ0IGxvbmcg
aW50IHRvIGRvdHRlZCBxdWFkIHN0cmluZyIKICAgIHJldHVybiBzb2NrZXQu
aW5ldF9udG9hKHN0cnVjdC5wYWNrKCchTCcsbikpCiAgICAgIApkZWYgbWFr
ZU1hc2sobik6CiAgICAicmV0dXJuIGEgbWFzayBvZiBuIGJpdHMgYXMgYSBs
b25nIGludGVnZXIiCiAgICByZXR1cm4gKDFMPDxuKS0xCgpkZWYgaXBUb05l
dEFuZEhvc3QoaXAsIG1hc2tiaXRzKToKICAgICJyZXR1cm5zIHR1cGxlIChu
ZXR3b3JrLCBob3N0KSBkb3R0ZWQtcXVhZCBhZGRyZXNzZXMgZ2l2ZW4gSVAg
YW5kIG1hc2sgc2l6ZSIKICAgICMgKGJ5IEdyZWcgSm9yZ2Vuc2VuKQoKICAg
IG4gPSBkb3R0ZWRRdWFkVG9OdW0oaXApCiAgICBtID0gbWFrZU1hc2sobWFz
a2JpdHMpCgogICAgaG9zdCA9IG4gJiBtCiAgICBuZXQgPSBuIC0gaG9zdAoK
ICAgIHJldHVybiBudW1Ub0RvdHRlZFF1YWQobmV0KSwgbnVtVG9Eb3R0ZWRR
dWFkKGhvc3QpCgpkZWYgbWFpbl90aHJlYWRzKGhvc3RzKToKICAgIHByaW50
ID4+c3lzLnN0ZGVyciwgIlVzaW5nIHRocmVhZHMiCiAgICBwaW5nbGlzdCA9
IFtdCiAgICBmb3IgaXAgaW4gaG9zdHM6CiAgICAgICBjdXJyZW50ID0gdGVz
dGl0KGlwKQogICAgICAgcGluZ2xpc3QuYXBwZW5kKGN1cnJlbnQpCiAgICAg
ICBjdXJyZW50LnN0YXJ0KCkKICAgIAogICAgZm9yIHBpbmdsZSBpbiBwaW5n
bGlzdDoKICAgICAgIHBpbmdsZS5qb2luKCkKICAgICAgIHByaW50ICJTdGF0
dXMgZnJvbSAiLHBpbmdsZS5pcCwiaXMiLHJlcG9ydFtwaW5nbGUuc3RhdHVz
XQoKZGVmIG1haW5fbm90aHJlYWRzKGhvc3RzKToKICAgIHByaW50ID4+c3lz
LnN0ZGVyciwgIk5vIHRocmVhZHMiCiAgICBmb3IgaXAgaW4gaG9zdHM6CiAg
ICAgICBwaW5nYWxpbmcgPSBnZXRfcGlwZShpcCkKICAgICAgIHdoaWxlIDE6
CiAgICAgICAgICAgbGluZSA9IHBpbmdhbGluZy5yZWFkbGluZSgpCiAgICAg
ICAgICAgaWYgbm90IGxpbmU6IGJyZWFrCiAgICAgICAgICAgaWdvdCA9IHJl
LmZpbmRhbGwobGlmZWxpbmUsbGluZSkKICAgICAgICAgICBpZiBpZ290Ogog
ICAgICAgICAgICAgICBwcmludCAiU3RhdHVzIGZyb20gIixpcCwiIGlzICIs
cmVwb3J0W2ludChpZ290WzBdKV0KCmRlZiBtYWluKGFyZ3Y9Tm9uZSk6CiAg
ICBnbG9iYWwgT1BUX3VzZV90aHJlYWRzLCBPUFRfdXNlX3N1YnByb2Nlc3Ms
IE9QVF91c2Vfc2hlbGwKICAgIGlmIGFyZ3YgaXMgTm9uZToKICAgICAgICBh
cmd2ID0gc3lzLmFyZ3YKIAogICAgb3B0cywgYXJncyA9IGdldG9wdC5nZXRv
cHQoYXJndlsxOl0sICJodHNTIixbImhlbHAiLCAidGhyZWFkcyIsICJzdWJw
cm9jZXNzIiwgInNoZWxsIl0pCiAgICBmb3Igb3B0LCB2YWx1ZSBpbiBvcHRz
OgogICAgICAgIGlmIG9wdCBpbiAoJy1oJywgJy0taGVscCcpOgogICAgICAg
ICAgICBwcmludCBfX2RvY19fICUgYXJndlswXQogICAgICAgICAgICByZXR1
cm4gMAogICAgICAgIGlmIG9wdCBpbiAoJy10JywgJy0tdGhyZWFkcycpOgog
ICAgICAgICAgICBPUFRfdXNlX3RocmVhZHMgPSBUcnVlCiAgICAgICAgaWYg
b3B0IGluICgnLXMnLCAnLS1zdWJwcm9jZXNzJyk6CiAgICAgICAgICAgIE9Q
VF91c2Vfc3VicHJvY2VzcyA9IFRydWUKICAgICAgICBpZiBvcHQgaW4gKCct
UycsICctLXNoZWxsJyk6CiAgICAgICAgICAgIE9QVF91c2Vfc2hlbGwgPSBU
cnVlCgogICAgIyBOT1RFOiBpZiB1c2luZyBvcy5wb3BlbiAodGhhdCBpcywg
T1BUX3VzZV9zdWJwcm9jZXNzIGlzIGZhbHNlKQogICAgIyAgICAgICB0aGVu
IHdlICphbHdheXMqIHVzZSB0aGUgc2hlbGwKICAgIGlmIG5vdCBPUFRfdXNl
X3N1YnByb2Nlc3M6CiAgICAgICAgT1BUX3VzZV9zaGVsbCA9IFRydWUKCiAg
ICAjIGNvbXB1dGUgcmFuZ2Ugb2YgaXAgYWRkcmVzc2VzCiAgICBteWlwID0g
bmFpdmVfZ2V0X2lwX2FkZHIoKQogICAgKG5ldHdrLCBoc3QpID0gaXBUb05l
dEFuZEhvc3QobXlpcCwgOCkKICAgIGhzdGQgPSBkb3R0ZWRRdWFkVG9OdW0o
aHN0KQogICAgbmV0d2tkID0gZG90dGVkUXVhZFRvTnVtKG5ldHdrKQogICAg
aWYgaHN0ZCA+PSAyNTA6CiAgICAgICAgaG9zdHJhbmdlID0gcmFuZ2UoMjQ1
LDI1NCkKICAgIGVsc2U6CiAgICAgICAgaG9zdHJhbmdlID0gcmFuZ2UoaHN0
ZCAtIChoc3RkICUgMTApICsgMSwKICAgICAgICAgICAgICAgICAgICAgICAg
ICBoc3RkIC0gKGhzdGQgJSAxMCkgKyAxMSkKCiAgICBob3N0cyA9IFtdIAog
ICAgZm9yIGhvc3QgaW4gaG9zdHJhbmdlOgogICAgICAgaXAgPSBudW1Ub0Rv
dHRlZFF1YWQobmV0d2tkICsgaG9zdCkKICAgICAgIGhvc3RzLmFwcGVuZChp
cCkKIAogICAgcHJpbnQgdGltZS5jdGltZSgpCiAgICBpZiBPUFRfdXNlX3Ro
cmVhZHM6CiAgICAgICAgbWFpbl90aHJlYWRzKGhvc3RzKQogICAgZWxzZToK
ICAgICAgICBtYWluX25vdGhyZWFkcyhob3N0cykKICAgIHByaW50IHRpbWUu
Y3RpbWUoKQoKCmlmIF9fbmFtZV9fID09ICJfX21haW5fXyI6CiAgICBzeXMu
ZXhpdChtYWluKCkpCgo=
--_----------=_1216915906174220
Content-Type: text/plain; charset=us-ascii
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
--_----------=_1216915906174220--
- Raw text -