DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3DF2C3858D39
Subject: Re: Missing Python.h
To: Russell VT <russellvt AT gmail DOT com>
References: <56f03ba0-a90d-b2e1-2c7d-20865f717334 DOT ref AT bellsouth DOT net>
 <56f03ba0-a90d-b2e1-2c7d-20865f717334 AT bellsouth DOT net>
 <CANV2+nWk8-6V6PLkFQHhUUEo=FevZ9bDXco0P8B=RbKrSdf4HA AT mail DOT gmail DOT com>
From: Dennis Putnam <dap1 AT bellsouth DOT net>
Message-ID: <2bc4e96a-52c8-7791-0d9d-2f5baa95749e@bellsouth.net>
Date: Wed, 22 Sep 2021 10:19:06 -0400
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.14.0
MIME-Version: 1.0
In-Reply-To: <CANV2+nWk8-6V6PLkFQHhUUEo=FevZ9bDXco0P8B=RbKrSdf4HA@mail.gmail.com>
Content-Language: en-US
Precedence: list
Cc: cygwin <cygwin AT cygwin DOT com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit

Hi Russell,

Thanks for the reply. I program Python 2 and 3 on various Linux systems 
but not much on Cygwin. It normally is not a problem to have both so I 
didn't think it would be a problem on Cygwin. Since I have scripts for 2 
on Cygwin, I'm thinking I should toss 3,at least for now, and just stick 
with 2. I have a lot to think about at this point. Using  an environment 
layer adds more complexity when I want to launch a script from a Windows 
application or bat file.

On 9/22/2021 5:49 AM, Russell VT wrote:
> First off, this *probably* isn't a Cygwin problem ... but it looks 
> like your environment is confused as it's using BOTH Python2 and 
> Python3 modules to try to fulfill the requirements (including 
> resources that have already been cached).
>
> For the most part, pip and pip3 can differentiate, but there's a 
> "cart" and "horse" problem, as if you install things in a weird order 
> at the system level, it may or may not do the right thing. But, I'd 
> recommend "dumping" Python2, if you can at this point (it was EOL'd in 
> December of 2020 and WILL NOT receive more updates, except for 
> security ... and more-over, Python 3.7+ (approx) is going to demand 
> newer SSL libraries that will REALLY confuse earlier versions).
>
> For working with Python (as a Python devops type), I generally 
> recommend using 'pyenv' and 'pyenv-virtualenv' and trying to do *as 
> little as possible* to modify the system-level Python ...this gets 
> HARD with a system like Cygwin, where the generic user can generally 
> overwrite system binaries, without any real sort of warning (and NO, 
> UAC does NOT adequately fix anything).
>
> That also said, pyenv kinda really "fights" with Cygwin in some of the 
> library placement (specifically things like FFI, IIRC, which is stored 
> in a different library directory than it is, anywhere else I've found).
>
> Where I MIGHT start is to "Force Reinstall" the Python3 stuff from 
> Setup. Look to see if requests_html is part of the Cygwin-supported 
> modules, and use THAT... use the hell out of anything you see in the 
> actual Python packages list, as those will at least be done RIGHT, and 
> will leave you with more cycles to not worry about too much, except 
> your development.
>
> For "Advanced" handling in Python, you're going to want to use "pyenv" 
> or some other multi-python managers that are out there (virtualenv and 
> virtualenv-wrapper are good, but ONLY manage the library path). Pretty 
> much "pyenv" and "pipenv" are the top two, IIRC. I use pyenv, and 
> haven't dug too deep in to pipenv, at this point. But, like I said, 
> it's already tough enough to manage on older systems with older 
> libraries (SSL, specifically, throws wrenches in to *everything*).
>
> Feel free to hit me up for other ideas... I write too much Python 
> code, as it is, and on too many different environments (yes, some 
> still do Python2.3 through 2.6, and it makes me want to shoot myself, 
> sometimes... LOL).
>
> Hope that helps -
> Russell VT
>
>
> On Tue, Sep 21, 2021 at 11:38 AM Dennis Putnam <dap1 AT bellsouth DOT net 
> <mailto:dap1 AT bellsouth DOT net>> wrote:
>
>     I am trying to install 'requests_html' and when it tries to do a
>     compile
>     it fails because Python.h is missing. I have python2-devl
>     installed. I
>     notice that it is looking for it in /pub which apparently does not
>     exist. Can someone help? TIA.
>
>     Here is the entire 'pip' output:
>
>     $ pip install requests_html
>     Collecting requests_html
>        Using cached requests_html-0.10.0-py3-none-any.whl (13 kB)
>     Collecting requests
>        Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
>     Collecting w3lib
>        Using cached w3lib-1.22.0-py2.py3-none-any.whl (20 kB)
>     Collecting parse
>        Using cached parse-1.19.0.tar.gz (30 kB)
>     Collecting fake-useragent
>        Using cached fake-useragent-0.1.11.tar.gz (13 kB)
>     Collecting pyquery
>        Using cached pyquery-1.4.3-py3-none-any.whl (22 kB)
>     Collecting bs4
>        Using cached bs4-0.0.1.tar.gz (1.1 kB)
>     Collecting pyppeteer>=0.0.14
>        Using cached pyppeteer-0.2.6-py3-none-any.whl (83 kB)
>     Requirement already satisfied: tqdm<5.0.0,>=4.42.1 in
>     /usr/local/lib/python3.8/site-packages (from
>     pyppeteer>=0.0.14->requests_html) (4.62.3)
>     Requirement already satisfied: urllib3<2.0.0,>=1.25.8 in
>     /usr/local/lib/python3.8/site-packages (from
>     pyppeteer>=0.0.14->requests_html) (1.26.6)
>     Collecting appdirs<2.0.0,>=1.4.3
>        Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
>     Collecting importlib-metadata>=1.4
>        Using cached importlib_metadata-4.8.1-py3-none-any.whl (17 kB)
>     Requirement already satisfied: pyee<9.0.0,>=8.1.0 in
>     /usr/local/lib/python3.8/site-packages (from
>     pyppeteer>=0.0.14->requests_html) (8.2.2)
>     Requirement already satisfied: websockets<10.0,>=9.1 in
>     /usr/local/lib/python3.8/site-packages (from
>     pyppeteer>=0.0.14->requests_html) (9.1)
>     Requirement already satisfied: zipp>=0.5 in
>     /usr/local/lib/python3.8/site-packages (from
>     importlib-metadata>=1.4->pyppeteer>=0.0.14->requests_html) ( 3.5.0)
>     Requirement already satisfied: beautifulsoup4 in
>     /usr/local/lib/python3.8/site-packages (from bs4->requests_html)
>     (4.10.0)
>     Requirement already satisfied: soupsieve>1.2 in
>     /usr/local/lib/python3.8/site-packages (from
>     beautifulsoup4->bs4->requests_html) (2.2.1)
>     Collecting cssselect>0.7.9
>        Using cached cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
>     Collecting lxml>=2.1
>        Using cached lxml-4.6.3.tar.gz (3.2 MB)
>     Collecting idna<4,>=2.5
>        Using cached idna-3.2-py3-none-any.whl (59 kB)
>     Collecting certifi>=2017.4.17
>        Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
>     Collecting charset-normalizer~=2.0.0
>        Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
>     Requirement already satisfied: six>=1.4.1 in
>     /usr/local/lib/python3.8/site-packages (from w3lib->requests_html)
>     (1.16.0)
>     Using legacy 'setup.py install' for bs4, since package 'wheel' is not
>     installed.
>     Using legacy 'setup.py install' for fake-useragent, since package
>     'wheel' is not installed.
>     Using legacy 'setup.py install' for parse, since package 'wheel'
>     is not
>     installed.
>     Using legacy 'setup.py install' for lxml, since package 'wheel' is
>     not
>     installed.
>     Installing collected packages: lxml, importlib-metadata, idna,
>     cssselect, charset-normalizer, certifi, appdirs, w3lib, requests,
>     pyquery, pyppeteer,
>     parse, fake-useragent, bs4, requests-html
>          Running setup.py install for lxml ... error
>          ERROR: Command errored out with exit status 1:
>           command: /usr/bin/python -u -c 'import io, os, sys, setuptools,
>     tokenize; sys.argv[0] =
>     '"'"'/tmp/pip-install-mp64a75l/lxml_61e1128ed27f4063936a
>     7a21e44c273e/setup.py'"'"';
>     __file__='"'"'/tmp/pip-install-mp64a75l/lxml_61e1128ed27f4063936a7a21e44c273e/setup.py'"'"';f
>
>     = getattr(tokenize, '"'"'open'"'"', open)(__file__) if
>     os.path.exists(__file__) else io.StringIO('"'"'from setuptools import
>     setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"',
>     '"'"'\n'"'"');f.close();exec(compile(code, __file__,
>     '"'"'exec'"'"'))'
>     install --record /tmp/pip-record-qafwouur/install-record.txt
>     --single-version-externally-managed --compile --install-headers
>     /usr/local/include/python3.8/lxml
>               cwd:
>     /tmp/pip-install-mp64a75l/lxml_61e1128ed27f4063936a7a21e44c273e/
>          Complete output (86 lines):
>          Building lxml version 4.6.3.
>          Building without Cython.
>          Building against libxml2 2.9.10 and libxslt 1.1.29
>          running install
>          running build
>          running build_py
>          creating build
>          creating build/lib.cygwin-3.2.0-x86_64-3.8
>          creating build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/builder.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/cssselect.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/doctestcompare.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/ElementInclude.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/pyclasslookup.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/sax.py -> build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/usedoctest.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/_elementpath.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/__init__.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          creating build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/__init__.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          creating build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/builder.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/clean.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/defs.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/diff.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/ElementSoup.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/formfill.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/html5parser.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/soupparser.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/usedoctest.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/_diffcommand.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/_html5builder.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/_setmixin.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          copying src/lxml/html/__init__.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/html
>          creating build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron
>          copying src/lxml/isoschematron/__init__.py ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron
>          copying src/lxml/etree.h ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/etree_api.h ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/lxml.etree.h ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/lxml.etree_api.h ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>          copying src/lxml/includes/c14n.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/config.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/dtdvalid.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/etreepublic.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/htmlparser.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/relaxng.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/schematron.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/tree.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/uri.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/xinclude.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/xmlerror.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/xmlparser.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/xmlschema.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/xpath.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/xslt.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/__init__.pxd ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/etree_defs.h ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          copying src/lxml/includes/lxml-version.h ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/includes
>          creating
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/resources
>          creating
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/resources/rng
>          copying
>     src/lxml/isoschematron/resources/rng/iso-schematron.rng ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/resources/rng
>          creating
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/resources/xsl
>          copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/resources/xsl
>          copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/resources/xsl
>          creating
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
>
>
>          copying
>     src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl
>
>     -> build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron
>     /resources/xsl/iso-schematron-xslt1
>          copying
>     src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl
>
>     -> build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/re
>     sources/xsl/iso-schematron-xslt1
>          copying
>     src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl
>
>     -> build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschemat
>     ron/resources/xsl/iso-schematron-xslt1
>          copying
>     src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl
>
>     -> build/lib.cygwin-3.2.0-x86_64-3.8/lxml
>     /isoschematron/resources/xsl/iso-schematron-xslt1
>          copying
>     src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl
>
>     -> build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/
>     resources/xsl/iso-schematron-xslt1
>          copying
>     src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt
>     ->
>     build/lib.cygwin-3.2.0-x86_64-3.8/lxml/isoschematron/resources/xs
>     l/iso-schematron-xslt1
>          running build_ext
>          building 'lxml.etree' extension
>          creating build/temp.cygwin-3.2.0-x86_64-3.8
>          creating build/temp.cygwin-3.2.0-x86_64-3.8/src
>          creating build/temp.cygwin-3.2.0-x86_64-3.8/src/lxml
>          gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3
>     -Wall
>     -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
>     -fstack-protector-strong --param=ssp-buffer-size=4
>     -DOPENSSL_NO_SSL3=1
>     -fdebug-prefix-map=/pub/devel/python/python38/python38-3.8.10-1.x86_64/build=/usr/src/debug/python38-3.8.10-1
>
>     -fdebug-prefix-map=/pub/devel/python/python38/python38-3.8.10-1.x86_64/src/Python-3.8.10=/usr/src/debug/python38-3.8.10-1
>
>     -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
>     -fstack-protector-strong --param=ssp-buffer-size=4
>     -DOPENSSL_NO_SSL3=1
>     -fdebug-prefix-map=/pub/devel/python/python38/python38-3.8.10-1.x86_64/build=/usr/src/debug/python38-3.8.10-1
>
>     -fdebug-prefix-map=/pub/devel/python/python38/python38-3.8.10-1.x86_64/src/Python-3.8.10=/usr/src/debug/python38-3.8.10-1
>
>     -DCYTHON_CLINE_IN_TRACEBACK=0 -I/usr/include/libxml2 -Isrc
>     -Isrc/lxml/includes -I/usr/include/python3.8 -c src/lxml/etree.c -o
>     build/temp.cygwin-3.2.0-x86_64-3.8/src/lxml/etree.o -w
>          src/lxml/etree.c:97:10: fatal error: Python.h: No such file or
>     directory
>             97 | #include "Python.h"
>                |          ^~~~~~~~~~
>          compilation terminated.
>          Compile failed: command 'gcc' failed with exit status 1
>          creating tmp
>          cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c
>     /tmp/xmlXPathInit4efcky6t.c -o tmp/xmlXPathInit4efcky6t.o
>          cc tmp/xmlXPathInit4efcky6t.o -lxml2 -o a.out.exe
>          error: command 'gcc' failed with exit status 1
>          ----------------------------------------
>     ERROR: Command errored out with exit status 1: /usr/bin/python -u -c
>     'import io, os, sys, setuptools, tokenize; sys.argv[0] =
>     '"'"'/tmp/pip-install-mp64a75l/lxml_61e1128ed27f4063936a7a21e44c273e/setup.py'"'"';
>
>     __file__='"'"'/tmp/pip-install-mp64a75l/lxml_61e1128ed27f4063936a7a21e44c273e/setup.py'"'"';f
>
>     = getattr(tokenize, '"'"'open'"'"', open)(__file__) if
>     os.path.exists(__file__) else io.StringIO('"'"'from setuptools import
>     setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"',
>     '"'"'\n'"'"');f.close();exec(compile(code, __file__,
>     '"'"'exec'"'"'))'
>     install --record /tmp/pip-record-qafwouur/install-record.txt
>     --single-version-externally-managed --compile --install-headers
>     /usr/local/include/python3.8/lxml Check the logs for full command
>     output.
>
>
>     -- 
>     Problem reports: https://cygwin.com/problems.html
>     <https://cygwin.com/problems.html>
>     FAQ: https://cygwin.com/faq/ <https://cygwin.com/faq/>
>     Documentation: https://cygwin.com/docs.html
>     <https://cygwin.com/docs.html>
>     Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
>     <https://cygwin.com/ml/#unsubscribe-simple>
>
>
>
> -- 
> Russell M. Van Tassell <russellvt AT gmail DOT com <mailto:russellvt AT gmail DOT com>>


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple