X-Recipient: archive-cygwin AT delorie DOT com
X-SWARE-Spam-Status: No, hits=-1.6 required=5.0 	tests=AWL,BAYES_00,FB_WORD1_END_DOLLAR,J_CHICKENPOX_31,J_CHICKENPOX_33,J_CHICKENPOX_65,RCVD_IN_DNSWL_LOW,SPF_PASS,ZMIde_GENERICSPAM1
X-Spam-Check-By: sourceware.org
Message-ID: <498139CE.7070704@cwilson.fastmail.fm>
Date: Thu, 29 Jan 2009 00:08:30 -0500
From: Charles Wilson <cygwin AT cwilson DOT fastmail DOT fm>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.19) Gecko/20081209 Thunderbird/2.0.0.19 Mnenhy/0.7.5.666
MIME-Version: 1.0
To: cygwin AT cygwin DOT com
Subject: Re: RFD: cygwin + *native* MinGW compiler
References: <497FC147 DOT 306 AT cwilson DOT fastmail DOT fm> <497FE127 DOT 1010705 AT sbcglobal DOT net> <497FED17 DOT 4040901 AT cwilson DOT fastmail DOT fm> <4980B0EC DOT 3020705 AT sbcglobal DOT net>
In-Reply-To: <4980B0EC.3020705@sbcglobal.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com

Greg Chicares wrote:
> On 2009-01-28 05:28Z, Charles Wilson wrote:

First, thanks for your detailed response. It was very helpful.

>> Do you use gnu-style configured projects (autoconf, automake, libtool,
>> all that?) -- or some other build framework?
> 
> Yes. I use autotools to build "native" versions of libraries I need,
> in particular libxml2, libxslt, and wxwidgets. As an example, for
> libxml2, here's the crucial part:
...(reordered, below)...
> I can live with '--disable-dependency-tracking' because I rarely
> modify the sources; if I ever do, I can 'make clean' and rebuild
> them from scratch.
...
> I use only Cygwin's make-3.81:
...
[ stuff concerning -M*, including notations that you use "identity" mounts ]

Interesting. So you have to be quite careful about how your system is
configured: without identity mounts, it *will* break. If you (or any
package you build) uses -M* (except for -MMD) then it *will* break, and
-MMD only works if all paths in your build process are relative -- or
(again) if you have identity mounts. (And identity mounts are not
possible if you routinely use more than one drive).

This is already quite fragile.

>> What about creating static libraries?...
>
> I don't build third-party libraries as static. When I build my own
> static libraries, I use MinGW's 'ar', but the command line is just
>   /MinGW_/bin/ar -rus   liblmi.a convenience.o exception.o [...]
> where all the '.o' files are in the directory where I invoke 'ar'.

Once again -- a carefully managed situation where all paths are
explicitly relative.

> An incidental oddity is that the technique above produces
>   cygxml2-2.dll
>   cygxslt-1.dll
>   cygwxmsw28_gcc_344-0.dll
> with 'cyg-' instead of 'lib-'. AFAICT, this doesn't matter
> because the MinGW linker looks for both prefixes. I happen to
> have Cygwin's version of some dlls as well as my own, e.g.:
>   $where cygxml2-2.dll
>   /opt/lmi/local/bin/cygxml2-2.dll
>   /usr/bin/cygxml2-2.dll
>   /bin/cygxml2-2.dll
> but I specify my own '-L' path to the linker. Well, actually, I
> guess that doesn't matter: the MinGW linker wouldn't look here:
>   C:\cygwin\bin\cygxml2-2.dll
> by default anyway.

Oh, geez. That's really bad.  The whole POINT of cygwin using a special
prefix for its DLLs is so that they won't be found by "accident" when
the Windows Runtime Loader is loading a mingw app.  That is,

  mingw-foo.exe <-- (mingw) libz-1.dll
  cygwin-foo.exe <-- (cygwin) cygz-1.dll

So in the first case, the WRL sees that mingw-foo.exe needs a DLL named
"libz-1.dll" and searches (according to certain rules) for it. Because
all (ok, almost all) cygwin DLLs begin with 'cyg', none of them will be
named "libz-1.dll" -- so mingw-foo will work properly even if cygwin's
bin directory comes before (wherever libz-1.dll is) in the $PATH.

However, if instead you have:

  mingw-foo.exe <-- (mingw) cygz-1.dll
  cygwin-foo.exe <-- (cygwin) cygz-1.dll

Then either mingw-foo.exe or cygwin-foo.exe may break, depending on
which cygz-1.dll is found first in $PATH. (You might assume that if
mingw-foo.exe is in the same directory as the "mingw" cygz-1.dll, and
cygwin-foo.exe is in the same directory as the "cygwin" cygz-1.dll, that
the WRL would never be confused and things would always work, because
$PATH is never searched. You'd be wrong: what if mingw-foo.exe is a very
long-running process. So, it's in memory, along with a mapped copy of
the (mingw) cygz-1.dll. Now, you launch cygwin-foo.exe. The WRL is
smart: it sees that it needs a "cygz-1.dll" -- but wait! it already has
one loaded in memory. All it needs to do is map the readonly parts of
the (mingw) cygz-1.dll into cygwin-foo.exe's address space, and call the
dll initialization routines.

But that's the wrong cygz-1.dll. Bang. Dead.

Note that my concerns have little to do with what happens at link-time;
these are *run-time* problems.

[Aside: the mingw linker does NOT search for cyg*dll. It searches for:

          libxxx.dll.a
          xxx.dll.a
          libxxx.a
          xxx.lib
          libxxx.dll
          xxx.dll

because the mingw gcc spec file does NOT include --dll-search-prefix.
However, if the libxxx.dll.a that it DOES find specifies "I'm an import
library for cygxxx.dll" well, then, that's what the exe will tell the
Windows Runtime Loader that it needs.  cygwin's gcc spec file specifies
--dll-search-prefix=cyg, so its ld searches for:

          libxxx.dll.a
          xxx.dll.a
          libxxx.a
          xxx.lib
          cygxxx.dll
          libxxx.dll
          xxx.dll

But this usually doesn't matter, because you tell gcc -L/some/libdir/,
where it only finds *.dll.a and *.a, not -L/some/bindir where the
*.dll's live.]

What confuses me, tho, is this: if your package KNOWS (because you told
it) that $build and $host are mingw, then WHY is your libtool using
cygwin rules to generate the DLL name? But that's something for another
thread.

>> This led to a suggestion that "--build=cygwin --host=mingw32" should
>> always be interpreted as: mingw32-gcc is a cygwin-hosted cross compiler,
>> NOT the native MinGW-project supported gcc (and if it IS the native
>> MinGW one, expect breakage). I'm not sure such a sweeping statement is
>> accurate, or wise -- will that assumption break people's exising
>> (working) setups?
> 
> I use '--build=i686-pc-mingw32 --host=i686-pc-mingw32'. Here:
>   http://lists.gnu.org/archive/html/libtool-patches/2009-01/msg00193.html
> you say that's lying to 'configure', but you also observe that
> I'm in excellent company. I'd be dismayed if that broke.

Well, as I point out above: it's already very fragile, and requires a
great deal of careful setup to get it to work "correctly" (if generating
DLLs with the wrong name can be considered "correct").

I wouldn't want to gratuitously break this usage, but I am leaning
towards endorsing the suggestion that it's bad to lie to your toolchain.
However, if you REALLY want to do so, and assume all the risk of
breakage/failure/hairloss/heartburn, then...set these [*] environment
variables before you ./configure, and then cross your fingers...

> As for the build!=host case '--build=cygwin --host=mingw32',
> I could try rebuilding my stuff that way if you think an extra
> datapoint would help. I probably tried that at some time in the
> past, but can't remember whether it worked.

That would be a useful data point, but even if it worked, most testsuite
frameworks are smart enough to NOT try to run their tests in a
cross-build situation. One of Danny's (and TDM'a
http://www.tdragon.net/recentgcc/) reasons for "lying" the way they do
is that the build machinery "believes" it is a native build, so it goes
ahead and runs the testsuite.

> As for this sweeping statement:
>   http://lists.gnu.org/archive/html/libtool-patches/2009-01/msg00191.html
> | I don't think we should try to support the scenario where the MinGW gcc,
> | exactly as supplied by the MinGW project, is executed from within Cygwin.
> which I believe you're trying to counter: as evidence that at
> least some people in the real world care about this, you could
> cite the first paragraph here:
>   http://lists.nongnu.org/archive/html/lmi/2007-07/msg00008.html
> which echoes some of the things you said on libtool-patches,
> and the applicable part of this message:
>   http://lists.nongnu.org/archive/html/lmi/2007-11/msg00007.html
> | Here's my rationale for some of the technical decisions. First of
> | all, at least for now, I've chosen to use a MinGW toolchain in a
> | Cygwin environment. [...]

Thanks for those references; I will keep them in mind as we go forward.

[*] Here's what I'm thinking: right now (libtool-2.2.6+), the
--build=mingw --host=mingw situation when build is REALLY cygwin, is
broken, because (a) we no longer have the wrapper script when $host is
win32ish, only a wrapper exe (b) as it stands now, this wrapper exe will
ONLY have the correct ($host, not $build) path to the real exe when
mingw->mingw native, cygwin->cygwin native, or unix->mingw cross.  We
NEED to support a wider universe, because so many mingw and cygwin
clients use cross environments of some kind, beyond those currently
working: unix->cygwin, cygwin->mingw(non-lying), cygwin->mingw(lying?).

The most straightforward assumption is that cygwin->mingw is, well,
cygwin->mingw. That is, libtool should assume that the information it is
given about the $build environment (via config.guess or via --build) is
truthful. So, the proposed patch will support (in addition to c->c
native, m->m native, u->m cross), u->c cross, c->m (non-lying).

With a slight additional change, the c->m (lying) case can probably be
made to work as well, but it is already so fragile, requiring identity
mounts and avoid -M options, that I can't see it continuing as a
supported option. Which is not to say that it is officially supported
now -- it's just that several high-profile people do it, because it
(happens to) work...mostly. Kinda. Right now. With libtool older than
2.2.(4?)

So, I'm thinking we can kinda-sorta enable it to continue to mostly work
(caveat emptor, etc etc) by making the following libtool variables cached:

lt_cv_to_host_path_cmd
lt_cv_to_host_pathlist_cmd

Then, folks like you, TDM, Danny, etc, can do

cygwin$ export lt_cv_to_host_path_cmd=func_cygwin_to_mingw_path_convert
cygwin$ export
lt_cv_to_host_pathlist_cmd=func_cygwin_to_mingw_pathlist_convert

before

cygwin$ ../src/configure --build=mingw32 --host=mingw32 ...

Ordinarily, the above configure incantation would result in libtool (as
proposed) using func_msys_to_mingw_path[list]_convert -- which is
clearly wrong in this case, because func_msys_to_mingw_path_convert
ain't gonna work right in a $build environment that is actually cygwin.
 So, override it with the correct conversion functions -- and your
wrapper exe's will have the correct paths.

Setting autoconf cached variables is clearly an esoteric proposition,
requiring detailed knowledge of libtool internals (or the URL of this
post).  But so does "knowing" why you need identity mounts, or that you
have to use --disable-dependency-tracking for arcane path-handling
reasons.  I can't see that requiring to env vars is any worse than all that.

--
Chuck

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/