X-Recipient: archive-cygwin AT delorie DOT com X-Spam-Check-By: sourceware.org Message-Id: <1216915906.17422.1265168403@webmail.messagingengine.com> From: "Charles Wilson" To: cygwin AT cygwin DOT com Content-Transfer-Encoding: 7bit Content-Type: multipart/mixed; boundary="_----------=_1216915906174220"; charset="ISO-8859-1" MIME-Version: 1.0 X-Mailer: MessagingEngine.com Webmail Interface Subject: cygwin python: subprocess.Popen().stdout + threads == hang Date: Thu, 24 Jul 2008 12:11:46 -0400 Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT cygwin DOT com Mail-Followup-To: cygwin AT cygwin DOT com Delivered-To: mailing list cygwin AT cygwin DOT com --_----------=_1216915906174220 Content-Disposition: inline Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="ISO-8859-1" MIME-Version: 1.0 X-Mailer: MessagingEngine.com Webmail Interface Date: Thu, 24 Jul 2008 16:11:46 UT Ran into a weird issue using python on cygwin. When trying to use the stdout handle from a subprocess spawned using the subprocess.Popen class, it hangs when accessed via a thread. This behavior does /not/ occur on other platforms -- nor when using os.popen from threads on cygwin. Nor when using subprocess.Popen from cygwin without threads. I know I could avoid this by using subprocess.Popen(...).communicate(), but in my /real/ application, I want line-by-line access to the sub's output, whereas communicate() gobbles the entire output and returns a list of strings for stdout, and for stderr. From the help text: communicate() returns a tuple (stdout, stderr). Note: The data read is buffered in memory, so do not use this method if the data size is large or unlimited. So, I don't want to do /that/. I've attached a test case to demonstrate the issue. I was hoping (a) somebody has seen this before, and has a simple solution for me, or (b) the python maintainer (Jason?) could take a look, and maybe raise a bug report upstream. I'm using 2.5.1-2, but I see the same behavior on cygwin using 2.5.2. When run on native windows, this test case requires Fping.exe (http://www.kwakkelflap.com/fping.html) because -- at least on Vista -- the native ping is b0rked. When native ping is called on Vista from a threaded context, if a host is unreachable, it reports summary info for the previously-accessed reachable host. Very strange -- but not related to the problem at hand. For cygwin, of course, the test case uses /usr/bin/ping. Test case --help output: Usage: ./testprog.py [htsS] Tests os.popen vs. subprocess.Popen for various platforms On native windows: requires Fping.exe (http://www.kwakkelflap.com/fping.html) In operation, this program 'pings' ten IP addresses in the current network, including localhost. It is used to demonstrate that the .stdout member of subprocess.Popen() does not work on Cygwin when using threads -h, --help print this message -t, --threads use threads (default is seqential) -s, --subprocess use subprocess.Popen().stdout (default is os.popen) -S, --shell when using subprocess.Popen(), execute the target using the shell (/bin/sh or cmd.exe), just like os.popen does. default for subprocess.Popen is to use an execv-style list. Behavior: native windows cygwin unix (os.popen, sequential) OK OK OK -t (os.popen, threaded) OK OK OK -s -S (subprocess, sequential, via shell) OK OK OK -s (subprocess, sequential, execv) OK OK OK -t -s -S (subprocess, threaded, via shell) OK hangs OK -t -s (subprocess, threaded, execv) OK hangs OK --_----------=_1216915906174220 Content-Disposition: attachment; filename="testprog.py" Content-Transfer-Encoding: base64 Content-Type: text/x-python; name="testprog.py" MIME-Version: 1.0 X-Mailer: MessagingEngine.com Webmail Interface Date: Thu, 24 Jul 2008 16:11:46 UT IyEvdXNyL2Jpbi9weXRob24KIiIiClVzYWdlOiAlcyBbaHRzU10KClRlc3Rz IG9zLnBvcGVuIHZzLiBzdWJwcm9jZXNzLlBvcGVuIGZvciB2YXJpb3VzIHBs YXRmb3JtcwpPbiBuYXRpdmUgd2luZG93czogcmVxdWlyZXMgRnBpbmcuZXhl IChodHRwOi8vd3d3Lmt3YWtrZWxmbGFwLmNvbS9mcGluZy5odG1sKQpJbiBv cGVyYXRpb24sIHRoaXMgcHJvZ3JhbSAncGluZ3MnIHRlbiBJUCBhZGRyZXNz ZXMgaW4gdGhlIGN1cnJlbnQKbmV0d29yaywgaW5jbHVkaW5nIGxvY2FsaG9z dC4gIEl0IGlzIHVzZWQgdG8gZGVtb25zdHJhdGUgdGhhdCB0aGUKLnN0ZG91 dCBtZW1iZXIgb2Ygc3VicHJvY2Vzcy5Qb3BlbigpIGRvZXMgbm90IHdvcmsg b24gQ3lnd2luIHdoZW4KdXNpbmcgdGhyZWFkcwoKICAtaCwgLS1oZWxwICAg ICAgICBwcmludCB0aGlzIG1lc3NhZ2UKICAtdCwgLS10aHJlYWRzICAgICB1 c2UgdGhyZWFkcyAoZGVmYXVsdCBpcyBzZXFlbnRpYWwpCiAgLXMsIC0tc3Vi cHJvY2VzcyAgdXNlIHN1YnByb2Nlc3MuUG9wZW4oKS5zdGRvdXQgKGRlZmF1 bHQgaXMgb3MucG9wZW4pCiAgLVMsIC0tc2hlbGwgICAgICAgd2hlbiB1c2lu ZyBzdWJwcm9jZXNzLlBvcGVuKCksIGV4ZWN1dGUgdGhlIHRhcmdldCB1c2lu ZwogICAgICAgICAgICAgICAgICAgIHRoZSBzaGVsbCAoL2Jpbi9zaCBvciBj bWQuZXhlKSwganVzdCBsaWtlIG9zLnBvcGVuIGRvZXMuCiAgICAgICAgICAg ICAgICAgICAgZGVmYXVsdCBmb3Igc3VicHJvY2Vzcy5Qb3BlbiBpcyB0byB1 c2UgYW4gZXhlY3Ytc3R5bGUgbGlzdC4KCkJlaGF2aW9yOiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIG5hdGl2ZSB3aW5kb3dzICBj eWd3aW4gIHVuaXgKICA8bm8gYXJncz4gKG9zLnBvcGVuLCBzZXF1ZW50aWFs KSAgICAgICAgICAgICAgICAgICBPSyAgICAgICAgICAgT0sgICAgIE9LCiAg LXQgICAgICAgIChvcy5wb3BlbiwgdGhyZWFkZWQpICAgICAgICAgICAgICAg ICAgICAgT0sgICAgICAgICAgIE9LICAgICBPSwogIC1zIC1TICAgICAoc3Vi cHJvY2Vzcywgc2VxdWVudGlhbCwgdmlhIHNoZWxsKSAgICAgIE9LICAgICAg ICAgICBPSyAgICAgT0sKICAtcyAgICAgICAgKHN1YnByb2Nlc3MsIHNlcXVl bnRpYWwsIGV4ZWN2KSAgICAgICAgICBPSyAgICAgICAgICAgT0sgICAgIE9L CiAgLXQgLXMgLVMgIChzdWJwcm9jZXNzLCB0aHJlYWRlZCwgdmlhIHNoZWxs KSAgICAgICAgT0sgICAgICAgICAgaGFuZ3MgICBPSwogIC10IC1zICAgICAo c3VicHJvY2VzcywgdGhyZWFkZWQsIGV4ZWN2KSAgICAgICAgICAgIE9LICAg ICAgICAgIGhhbmdzICAgT0sgCiIiIiAKaW1wb3J0IG9zCmltcG9ydCByZQpp bXBvcnQgdGltZQppbXBvcnQgc3lzCmltcG9ydCBnZXRvcHQKZnJvbSB0aHJl YWRpbmcgaW1wb3J0IFRocmVhZAppbXBvcnQgc29ja2V0CmltcG9ydCBzdHJ1 Y3QKaW1wb3J0IHN1YnByb2Nlc3MKCiMgTk9URTogT1BUX3VzZV9zaGVsbCBp cyBhbHdheXMgdHJlYXRlZCBhcyBUcnVlIGlmIE9QVF91c2Vfc3VicHJvY2Vz cyBpcyBGYWxzZQojICh0aGF0IGlzLCBvcy5wb3BlbiBhbHdheXMgdXNlcyB0 aGUgc2hlbGwhKQpPUFRfdXNlX3RocmVhZHM9RmFsc2UKT1BUX3VzZV9zdWJw cm9jZXNzPUZhbHNlCk9QVF91c2Vfc2hlbGw9RmFsc2UKCmlmIHN5cy5wbGF0 Zm9ybSA9PSAnd2luMzInOgogICAgZGVmIHBpbmdfY21kbGluZShpcCwgc2hl bGw9RmFsc2UpOgogICAgICAgICMgd2luZG93cyBwaW5nIGlzIGIwcmtlZCBv biBWaXN0YSwgc28gd2UncmUgZm9yY2VkIHRvIHVzZSBhIGRpZmZlcmVudAog ICAgICAgICMgcGluZyBwcm9ncmFtLiBDYW4ndCB1c2UgY3lnd2luIHBpbmcs IG11c3QgYmUgYSBuYXRpdmUgcHJvZ3JhbS4KICAgICAgICAjICAgIEZwaW5n IGZyb20gaHR0cDovL3d3dy5rd2Fra2VsZmxhcC5jb20vZnBpbmcuaHRtbCBz ZWVtcyBwcmV0dHkKICAgICAgICAjIGdvb2QsIGlzIHZpcnVzIGZyZWUsIGFu ZCBpcyBub3QgYjBya2VkLiAKICAgICAgICBpZiBzaGVsbDoKICAgICAgICAg ICAgcmV0dXJuICJGcGluZy5leGUgIiArIGlwICsgIiAtaSAtbiAyIgogICAg ICAgIHJldHVybiBbIkZwaW5nLmV4ZSAiLCBpcCwgIi1pIiwgIi1uIiwgIjIi XQplbGlmIHN5cy5wbGF0Zm9ybSA9PSAnY3lnd2luJzoKICAgIGRlZiBwaW5n X2NtZGxpbmUoaXAsIHNoZWxsPUZhbHNlKToKICAgICAgICAjIGN5Z3dpbidz IHBpbmcgaXMgYWxzbyBvZGQKICAgICAgICBpZiBzaGVsbDoKICAgICAgICAg ICAgcmV0dXJuICJwaW5nLmV4ZSAtcSAiICsgaXAgKyAiIDU2IDIiCiAgICAg ICAgcmV0dXJuIFsicGluZy5leGUiLCAiLXEiLCBpcCwgIjU2IiwgIjIiXQpl bHNlOgogICAgZGVmIHBpbmdfY21kbGluZShpcCwgc2hlbGw9RmFsc2UpOgog ICAgICAgIGlmIHNoZWxsOgogICAgICAgICAgICByZXR1cm4gInBpbmcgLXEg LWMyICIgKyBpcAogICAgICAgIHJldHVybiBbInBpbmciLCAiLXEiLCAiLWMy IiwgaXBdCgpkZWYgZ2V0X3BpcGVfc3VicHJvY2Vzc19Qb3BlbihpcCk6CiAg ICBnbG9iYWwgT1BUX3VzZV9zaGVsbAogICAgcHJpbnQgPj5zeXMuc3RkZXJy LCAiVXNpbmcgc3VicHJvY2Vzcy5Qb3BlbiAoc2hlbGw9JXMpIiAlIE9QVF91 c2Vfc2hlbGwKICAgIGNobGRfc3RkaW4gPSBvcy5vcGVuKG9zLmRldm51bGws IG9zLk9fUkRPTkxZKTsKICAgIHJldHVybiBzdWJwcm9jZXNzLlBvcGVuKHBp bmdfY21kbGluZShpcCwgc2hlbGw9T1BUX3VzZV9zaGVsbCksCiAgICAgICAg c2hlbGw9T1BUX3VzZV9zaGVsbCwKICAgICAgICBzdGRpbj1jaGxkX3N0ZGlu LAogICAgICAgIHN0ZG91dD1zdWJwcm9jZXNzLlBJUEUsCiAgICAgICAgc3Rk ZXJyPXN1YnByb2Nlc3MuU1RET1VUKS5zdGRvdXQKCmRlZiBnZXRfcGlwZV9v c19wb3BlbihpcCk6CiAgICBnbG9iYWwgT1BUX3VzZV9zaGVsbAogICAgcHJp bnQgPj5zeXMuc3RkZXJyLCAiVXNpbmcgb3MucG9wZW4gKHNoZWxsPSVzKSIg JSBPUFRfdXNlX3NoZWxsCiAgICByZXR1cm4gb3MucG9wZW4ocGluZ19jbWRs aW5lKGlwLCBzaGVsbD1PUFRfdXNlX3NoZWxsKSwgInIiKQoKZGVmIGdldF9w aXBlKGlwKToKICAgIGdsb2JhbCBPUFRfdXNlX3N1YnByb2Nlc3MKICAgIGlm IE9QVF91c2Vfc3VicHJvY2VzczoKICAgICAgICByZXR1cm4gZ2V0X3BpcGVf c3VicHJvY2Vzc19Qb3BlbihpcCkKICAgIGVsc2U6CiAgICAgICAgcmV0dXJu IGdldF9waXBlX29zX3BvcGVuKGlwKQoKY2xhc3MgdGVzdGl0KFRocmVhZCk6 CiAgIGRlZiBfX2luaXRfXyAoc2VsZixpcCk6CiAgICAgIFRocmVhZC5fX2lu aXRfXyhzZWxmKQogICAgICBzZWxmLmlwID0gaXAKICAgICAgc2VsZi5zdGF0 dXMgPSAtMQogICBkZWYgcnVuKHNlbGYpOgogICAgICBwaW5nYWxpbmcgPSBn ZXRfcGlwZShzZWxmLmlwKQogICAgICB3aGlsZSAxOgogICAgICAgIGxpbmUg PSBwaW5nYWxpbmcucmVhZGxpbmUoKQogICAgICAgIGlmIG5vdCBsaW5lOiBi cmVhawogICAgICAgIGlnb3QgPSByZS5maW5kYWxsKHRlc3RpdC5saWZlbGlu ZSxsaW5lKQogICAgICAgIGlmIGlnb3Q6CiAgICAgICAgICAgc2VsZi5zdGF0 dXMgPSBpbnQoaWdvdFswXSkKCmlmIHN5cy5wbGF0Zm9ybSA9PSAnd2luMzIn OgogICAgdGVzdGl0LmxpZmVsaW5lID0gcmUuY29tcGlsZShyIlJlY2VpdmVk ID0gKFxkKSIpCiAgICBsaWZlbGluZSA9IHJlLmNvbXBpbGUociJSZWNlaXZl ZCA9IChcZCkiKQplbGlmIHN5cy5wbGF0Zm9ybSA9PSAnY3lnd2luJzoKICAg IHRlc3RpdC5saWZlbGluZSA9IHJlLmNvbXBpbGUociIoXGQpIHBhY2tldHMg cmVjZWl2ZWQiKQogICAgbGlmZWxpbmUgPSByZS5jb21waWxlKHIiKFxkKSBw YWNrZXRzIHJlY2VpdmVkIikKZWxzZToKICAgIHRlc3RpdC5saWZlbGluZSA9 IHJlLmNvbXBpbGUociIoXGQpIHJlY2VpdmVkIikKICAgIGxpZmVsaW5lID0g cmUuY29tcGlsZShyIihcZCkgcmVjZWl2ZWQiKQpyZXBvcnQgPSAoIk5vIHJl c3BvbnNlIiwiUGFydGlhbCBSZXNwb25zZSIsIkFsaXZlIiwiSW50ZXJuYWwg RXJyb3IiKQoKCmRlZiBuYWl2ZV9nZXRfaXBfYWRkcigpOgogICAgcmV0dXJu IHNvY2tldC5nZXRob3N0YnluYW1lKHNvY2tldC5nZXRob3N0bmFtZSgpKQoK ZGVmIGRvdHRlZFF1YWRUb051bShpcCk6CiAgICAiY29udmVydCBkZWNpbWFs IGRvdHRlZCBxdWFkIHN0cmluZyB0byBsb25nIGludGVnZXIiCiAgICByZXR1 cm4gc3RydWN0LnVucGFjaygnIUwnLHNvY2tldC5pbmV0X2F0b24oaXApKVsw XQoKZGVmIG51bVRvRG90dGVkUXVhZChuKToKICAgICJjb252ZXJ0IGxvbmcg aW50IHRvIGRvdHRlZCBxdWFkIHN0cmluZyIKICAgIHJldHVybiBzb2NrZXQu aW5ldF9udG9hKHN0cnVjdC5wYWNrKCchTCcsbikpCiAgICAgIApkZWYgbWFr ZU1hc2sobik6CiAgICAicmV0dXJuIGEgbWFzayBvZiBuIGJpdHMgYXMgYSBs b25nIGludGVnZXIiCiAgICByZXR1cm4gKDFMPDxuKS0xCgpkZWYgaXBUb05l dEFuZEhvc3QoaXAsIG1hc2tiaXRzKToKICAgICJyZXR1cm5zIHR1cGxlIChu ZXR3b3JrLCBob3N0KSBkb3R0ZWQtcXVhZCBhZGRyZXNzZXMgZ2l2ZW4gSVAg YW5kIG1hc2sgc2l6ZSIKICAgICMgKGJ5IEdyZWcgSm9yZ2Vuc2VuKQoKICAg IG4gPSBkb3R0ZWRRdWFkVG9OdW0oaXApCiAgICBtID0gbWFrZU1hc2sobWFz a2JpdHMpCgogICAgaG9zdCA9IG4gJiBtCiAgICBuZXQgPSBuIC0gaG9zdAoK ICAgIHJldHVybiBudW1Ub0RvdHRlZFF1YWQobmV0KSwgbnVtVG9Eb3R0ZWRR dWFkKGhvc3QpCgpkZWYgbWFpbl90aHJlYWRzKGhvc3RzKToKICAgIHByaW50 ID4+c3lzLnN0ZGVyciwgIlVzaW5nIHRocmVhZHMiCiAgICBwaW5nbGlzdCA9 IFtdCiAgICBmb3IgaXAgaW4gaG9zdHM6CiAgICAgICBjdXJyZW50ID0gdGVz dGl0KGlwKQogICAgICAgcGluZ2xpc3QuYXBwZW5kKGN1cnJlbnQpCiAgICAg ICBjdXJyZW50LnN0YXJ0KCkKICAgIAogICAgZm9yIHBpbmdsZSBpbiBwaW5n bGlzdDoKICAgICAgIHBpbmdsZS5qb2luKCkKICAgICAgIHByaW50ICJTdGF0 dXMgZnJvbSAiLHBpbmdsZS5pcCwiaXMiLHJlcG9ydFtwaW5nbGUuc3RhdHVz XQoKZGVmIG1haW5fbm90aHJlYWRzKGhvc3RzKToKICAgIHByaW50ID4+c3lz LnN0ZGVyciwgIk5vIHRocmVhZHMiCiAgICBmb3IgaXAgaW4gaG9zdHM6CiAg ICAgICBwaW5nYWxpbmcgPSBnZXRfcGlwZShpcCkKICAgICAgIHdoaWxlIDE6 CiAgICAgICAgICAgbGluZSA9IHBpbmdhbGluZy5yZWFkbGluZSgpCiAgICAg ICAgICAgaWYgbm90IGxpbmU6IGJyZWFrCiAgICAgICAgICAgaWdvdCA9IHJl LmZpbmRhbGwobGlmZWxpbmUsbGluZSkKICAgICAgICAgICBpZiBpZ290Ogog ICAgICAgICAgICAgICBwcmludCAiU3RhdHVzIGZyb20gIixpcCwiIGlzICIs cmVwb3J0W2ludChpZ290WzBdKV0KCmRlZiBtYWluKGFyZ3Y9Tm9uZSk6CiAg ICBnbG9iYWwgT1BUX3VzZV90aHJlYWRzLCBPUFRfdXNlX3N1YnByb2Nlc3Ms IE9QVF91c2Vfc2hlbGwKICAgIGlmIGFyZ3YgaXMgTm9uZToKICAgICAgICBh cmd2ID0gc3lzLmFyZ3YKIAogICAgb3B0cywgYXJncyA9IGdldG9wdC5nZXRv cHQoYXJndlsxOl0sICJodHNTIixbImhlbHAiLCAidGhyZWFkcyIsICJzdWJw cm9jZXNzIiwgInNoZWxsIl0pCiAgICBmb3Igb3B0LCB2YWx1ZSBpbiBvcHRz OgogICAgICAgIGlmIG9wdCBpbiAoJy1oJywgJy0taGVscCcpOgogICAgICAg ICAgICBwcmludCBfX2RvY19fICUgYXJndlswXQogICAgICAgICAgICByZXR1 cm4gMAogICAgICAgIGlmIG9wdCBpbiAoJy10JywgJy0tdGhyZWFkcycpOgog ICAgICAgICAgICBPUFRfdXNlX3RocmVhZHMgPSBUcnVlCiAgICAgICAgaWYg b3B0IGluICgnLXMnLCAnLS1zdWJwcm9jZXNzJyk6CiAgICAgICAgICAgIE9Q VF91c2Vfc3VicHJvY2VzcyA9IFRydWUKICAgICAgICBpZiBvcHQgaW4gKCct UycsICctLXNoZWxsJyk6CiAgICAgICAgICAgIE9QVF91c2Vfc2hlbGwgPSBU cnVlCgogICAgIyBOT1RFOiBpZiB1c2luZyBvcy5wb3BlbiAodGhhdCBpcywg T1BUX3VzZV9zdWJwcm9jZXNzIGlzIGZhbHNlKQogICAgIyAgICAgICB0aGVu IHdlICphbHdheXMqIHVzZSB0aGUgc2hlbGwKICAgIGlmIG5vdCBPUFRfdXNl X3N1YnByb2Nlc3M6CiAgICAgICAgT1BUX3VzZV9zaGVsbCA9IFRydWUKCiAg ICAjIGNvbXB1dGUgcmFuZ2Ugb2YgaXAgYWRkcmVzc2VzCiAgICBteWlwID0g bmFpdmVfZ2V0X2lwX2FkZHIoKQogICAgKG5ldHdrLCBoc3QpID0gaXBUb05l dEFuZEhvc3QobXlpcCwgOCkKICAgIGhzdGQgPSBkb3R0ZWRRdWFkVG9OdW0o aHN0KQogICAgbmV0d2tkID0gZG90dGVkUXVhZFRvTnVtKG5ldHdrKQogICAg aWYgaHN0ZCA+PSAyNTA6CiAgICAgICAgaG9zdHJhbmdlID0gcmFuZ2UoMjQ1 LDI1NCkKICAgIGVsc2U6CiAgICAgICAgaG9zdHJhbmdlID0gcmFuZ2UoaHN0 ZCAtIChoc3RkICUgMTApICsgMSwKICAgICAgICAgICAgICAgICAgICAgICAg ICBoc3RkIC0gKGhzdGQgJSAxMCkgKyAxMSkKCiAgICBob3N0cyA9IFtdIAog ICAgZm9yIGhvc3QgaW4gaG9zdHJhbmdlOgogICAgICAgaXAgPSBudW1Ub0Rv dHRlZFF1YWQobmV0d2tkICsgaG9zdCkKICAgICAgIGhvc3RzLmFwcGVuZChp cCkKIAogICAgcHJpbnQgdGltZS5jdGltZSgpCiAgICBpZiBPUFRfdXNlX3Ro cmVhZHM6CiAgICAgICAgbWFpbl90aHJlYWRzKGhvc3RzKQogICAgZWxzZToK ICAgICAgICBtYWluX25vdGhyZWFkcyhob3N0cykKICAgIHByaW50IHRpbWUu Y3RpbWUoKQoKCmlmIF9fbmFtZV9fID09ICJfX21haW5fXyI6CiAgICBzeXMu ZXhpdChtYWluKCkpCgo= --_----------=_1216915906174220 Content-Type: text/plain; charset=us-ascii -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ --_----------=_1216915906174220--