Date: Tue, 24 Feb 2009 12:36:55 -0800
From: rhubbell <Rhubbell AT iHubbell DOT com>
To: cygwin AT cygwin DOT com
Subject: Re: [CFT] libtool on nix->cygwin cross, with wine
Message-Id: <20090224123655.dd559d60.Rhubbell@iHubbell.com>
In-Reply-To: <49A37F05.8050900@cwilson.fastmail.fm>
References: <announce DOT 49A35E97 DOT 5090808 AT cwilson DOT fastmail DOT fm> 	<49A37F05 DOT 8050900 AT cwilson DOT fastmail DOT fm>
Reply-To: cygwin AT cygwin DOT com
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Precedence: bulk
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com

What's the [CFT] stand for?
Call For Test?

Are you requesting test volunteers?


On Tue, 24 Feb 2009 00:01:19 -0500
Charles Wilson wrote:

> The most recent release of libtool (2.2.7a-1 for cygwin-1.5, and
> 2.2.7a-10 for cygwin-1.7) ought to support cross builds at least as well
> as libtool-1.5 did.  Note that in *ordinary* cross builds (SomeBUILD ->
> SomeHOST) you can't run the $host executables on the $build machine --
> but you can still *build* them. That kind of thing has (hopefully)
> always worked, and still works, for ->cygwin crosses.  However, under
> certain conditions, it USED to be possible to run the $host (cygwin)
> executables on the $build machine, provided $build's CPU was x86, and
> $build had wine installed, and $build was running linux with the
> 'binfmt' kernel extension, and there was a cygwin "installation" in the
> wine "arena", etc, etc.
> 
> /That/ has been broken for about a year, due to a change in how
> "wrapper" executables are handled by libtool in libtool-2.2.2 and above.
> 
> ============ 1 ==============
> See, in the ANCIENT days, there was a wrapper script.  When you built an
> executable for $host=cygwin, libtool would create
> 
>    myprog (a wrapper script)
>    .libs/myprog.exe (the actual exe)
> 
> The wrapper script would set $PATH and various other environment
> variables so that the EXE would be able to "find" its (as yet
> uninstalled) DLLs, and then it would launch the EXE.  Obviously, scripts
> are not bound to any specific platform, so $build has no problem running
> the script.  So as long as $build could execute EXE (via wine, etc), the
> the wrapper/EXE combo worked fine.
> 
> Why'd we change it? Well, there IS one little problem with that scheme:
> the Makefile rule for building the EXE looks like this:
> 
> ./my_prog$EXEEXT: my_prog.o <stuff>
> 	... some libtool commands ...
> 
> But libtool doesn't create ./my_prog$EXEEXT -- it creates a wrapper
> named ./my_prog with NO $EXEEXT.  So make always believes that the
> executable must be rebuilt.  'make' -> go link the exe. 'make check' ->
> 'go link the exe again'. 'make install' -> go link the exe. Very annoying.
> 
> ============ 2 ==============
> So, in the OLD days, we had a wrapper executable AND a wrapper script:
> 
>   my_prog.exe (not the real exe; just a wrapper exe)
>   my_prog     (the wrapper script)
>   .libs/my_prog.exe (the real exe)
> 
> The way this scheme worked was: (a) the wrapper exe would launch the
> wrapper script, (b) the wrapper script would set $PATH and such, and
> then (c) launch the real exe.  This worked pretty well -- and was fairly
> transparent to cross builders: they'd try to run 'my_prog' (not
> my_prog.exe, because who types "blahblah.EXE" on unix?). But, my_prog is
> a shell script, and it Just Works(tm) like it did in the ANCIENT days.
> So cross-builders were happy -- they just ignored that wrapper exe (and,
> incidentally, never tested it...)
> 
> So, what was wrong with this?  Did we "fix" something that wasn't broken?
> 
> Well, not exactly. About three years ago, cygwin added a new feature:
> you could set the 'transparent_exe' option in the CYGWIN variable. Then,
> you could pretend you were even more "unixy": files which end in .exe to
> be used by appropriate functions when an input filename is specified
> with no extension.  That is, you say spawn("foo") and if foo.exe exists,
> then cygwin will turn that in to spawn("foo.exe").
> 
> So...what if I have foo.exe AND foo, and a really really want to
> spawn("foo") -- NOT spawn("foo.exe").  Say, for instance:
> 
>    my_foo.exe
>    my_foo
> 
> Uh...don't do that.
> 
> See, transparent_exe caused all KINDS of pain for libtool -- cygwin (and
> libtool) got really confused by the situation.  However, as long as
> transparent_exe was just an option, we were ok though: you just had to
> follow some rules:
>   1) if transparent_exe, then do not put files that differ only in
> .exe-extension in the same directory
>   2) since libtool does this, do not use libtool and transparent_exe
> together.
> Not the greatest situation, but we lived with it.
> 
> Backgrounders:
> http://cygwin.com/ml/cygwin/2006-03/msg00148.html
> http://cygwin.com/ml/cygwin-apps/2006-03/msg00028.html
> http://cygwin.com/ml/cygwin/2007-04/msg00543.html
> 
> 
> But then, along comes cygwin-1.7...in which 'transparent_exe' will be
> the default behavior. Oops.
> 
> "Don't use libtool and cygwin-1.7" is not a rule we can live with.
> 
> So, (and this is the part that broke the cross-builders), we changed
> libtool's behavior...
> 
> ============ 3 ==============
> About two years ago, for $host cygwin/mingw, libtool was modified to no
> longer put a wrapper *script* in the build dir. Instead, it put an
> uglify-named version of it in .libs:
> 
>   my_prog.exe (not the real exe; just a wrapper exe)
>   .libs/my_prog_ltshwrapper    (the wrapper script #1)
>   .libs/my_prog_ltshwrapperTMP (the wrapper script #2)
>   .libs/my_prog.exe (the real exe)
> 
> That way, no more 'transparent_exe' clashes. BUT, now the poor
> cross-builders have no wrapper script to execute. They have to find the
> uglified version in .libs, OR run the wrapper executable (which was
> borked anyway in cross-build situations. See below).
> 
> Now, why two copies of the wrapper script? Well, libtool actually uses
> the wrapper script for two different purposes. #1, it "saves"
> information between different executions of libtool -- say, from the
> link phase to the install phase. So, sometimes libtool actually
> 'sources' the wrapper script back in, to read certain variable values.
> That's copy #1.  Copy #2 is ever-so-slightly different...
> 
> The way this scheme worked was: (a) the wrapper exe would emit the #2
> copy of wrapper script in .libs/* ON THE FLY, (b) and execute that
> wrapper script.  Then, the wrapper script would set $PATH, etc, and
> launch the real exe.
> 
> That way, the #2 copy is ALWAYS up-to-date with the wrapper exe -- and,
> because it is generated by the wrapper exe, it can be modified slightly
> from copy #1 (which is used for communicating variable values from
> different executions of libtool).
> http://lists.gnu.org/archive/html/libtool-patches/2007-04/msg00052.html
> 
> This mostly worked, but was a bit awkward on cross-compile setups.  For
> mingw, say, you had to set the environment variable TARGETSHELL to the
> *dos* path to your *unix $build* shell, so that via the magic of wine,
> CreateProcess("C:/some/magic/path/sh /path/to/wrapper/script args"...)
> would cause the $build machine's /bin/sh to run the desired script.
> Also, there were problems with atomicity: if you're building with make
> -jN, you might launch the same wrapper exe multiple times
> simultaneously. They all clobber each other's "temporary" wrapper scripts.
> 
> ============ 4 ==============
> So, the obvious solution is to have the wrapper executable itself do all
> the work of setting $PATH, and then launching the real executable.
> That's what happened in libtool-2.2.2 (about a year ago).  But, the
> initial implementation worked great for native builds (cygwin, mingw) --
> but completely "broke" cross-builds.  That is, you could still *build*
> stuff in a nix->cygwin or nix->mingw environment, but the wrapper system
> was completely fubared if you wanted to actually *run* those programs
> via their wrappers, in a cross environment.  Even if you had wine installed.
> 
> The problem boils down to this (it's easier to explain by talking about
> unix->mingw crosses, but similar arguments apply for unix->cygwin):
> libtool itself is executing in $build (unix). It knows about
> $build-style paths.  But the wrapper executable is a mingw executable,
> and will use the win32 _spawnv("C:\I\Want\A\DOS\Path") function to
> launch the real executable.  libtool knows the unix path to the real
> executable -- when creating the "C" source code for the wrapper
> executable, libtool needs to convert the unix-style (that is, $build)
> path into the DOS-style (that is, $host) format.
> 
> So, recently (libtool-2.2.4ish) libtool learned how to do that -- for
> unix->mingw.  It doesn't yet -- officially -- know how to do that for
> unix->cygwin, because that is more complicated:
> 
> 1) you have to use winepath to convert the $build (unix) path to "win32"
> format.
> 2) THEN, you have to use cygpath -- running under wine -- to convert
> that "win32" path to cygwin format "inside" wine.
> 
> It gets even more complicated if your wine "arena" has, for instance,
> TWO different cygwin installations: a cygwin-1.5 one and a cygwin-1.7
> one.  Which cygpath do you use -- the cygwin-1.5 cygpath which use the
> "registry" in wine to find the cygwin mount table, and will convert the
> "win32" path using that mount mapping? Or the cygwin-1.7 cygpath, which
>  will find itself, locate its ../etc/fstab file, and use THAT for
> converting the "win32" path to cygwin format.
> 
> Yikes!
> 
> ======== CALL FOR TEST ========
> 
> This recent cygwin release of libtool has a mechanism for supporting
> these sorts of cross builds.  BUT: you have to set an environment
> variable in $build (that is, your unix environment) to tell libtool
> which cygpath program you want to use:
> 
> export LT_CYGPATH=/unix/path/to/cygpath
> 
> Something like:
> 
> export LT_CYGPATH=/opt/winestuff/cygwin-1.7/bin/cygpath.exe
> 
> You also have to be running linux -- not any other unix -- with the
> binfmt kernel extension turned on. It usually is, so you probably don't
> need to worry about that one. (I don't THINK other unices support
> anything like binfmt...)
> 
> Then, you can try to autoreconf (to get the new libtool stuff) and build
> your favorite package using your unix->cygwin cross tools.
> 
> Finally: try to actually *execute* a wrapper executable -- and see if it
> works.
> 
> I'm not set up to test that.  If you are, please try to exercise this
> new capability and let me know if it works. The easiest way to do so is:
> 
> 1) download libtool-2.2.7a-20090223.tar.lzma from here
> http://cygwin.cwilson.fastmail.fm/ITP/libtool-2.2.7a-20090223.tar.lzma
> 
> (you COULD get libtool-2.2.7a-1-src.tar.bz2 from the cygwin mirrors,
> BUT: it's a git checkout, with patches, so it's a bit of work. You have
> to apply the patches in order, then run the bootstrap script.  The
> .tar.lzma file above has already had all that done)
> 
> 2) unpack, and create separate build directory
>    $ ls
>    libtool-2.2.7a-20090223/
>    build/
> 3) export LT_CYGPATH=/unix/path/to/cygpath.exe
> 4) cd build/
> 5) ../libtool-2.2.7a-20090223/configure \
>        --srcdir=../libtool-2.2.7a-20090223 \
>        --build=i686-pc-linux-gnu
>        --host=i686-pc-cygwin
> 6) make
> 
> Assuming that all goes well, then try:
> 
> 7) make check TESTSUITEFLAGS="-V" TESTS='tests/mdemo-shared.test
> tests/mdemo-make.test tests/mdemo-exec.test' VERBOSE=t
> 
> and see what happens.  If it works, GREAT. Tell me.
> 
> If not, then the interesting things to look at:
>   a) did tests/mdemo-shared.test pass?
>   b) did tests/mdemo-make.test pass?
>   c) so, if tests/mdemo-exec.test failed, then what does
>      build/tests/mdemo/.libs/lt-mdemo.c look like?
>      are the following "C" variables correct:
>         const char * LIB_PATH_VALUE
>         const char * EXE_PATH_VALUE
>   d) try rebuilding the wrapper with debugging:
>      i)    cd build/tests/mdemo
>      ii)   rm -f mdemo.exe # the wrapper
>      iii)  make CFLAGS="-DLT_DEBUGWRAPPER" mdemo.exe
>      then run the wrapper manually, and see what it prints out.
> 
> --
> Chuck
> 
> 
> --
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
> Problem reports:       http://cygwin.com/problems.html
> Documentation:         http://cygwin.com/docs.html
> FAQ:                   http://cygwin.com/faq/
> 

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/