Message-ID: <49A37F05.8050900@cwilson.fastmail.fm>
Date: Tue, 24 Feb 2009 00:00:53 -0500
From: Charles Wilson <cygwin AT cwilson DOT fastmail DOT fm>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.19) Gecko/20081209 Thunderbird/2.0.0.19 Mnenhy/0.7.6.666
MIME-Version: 1.0
To: cygwin AT cygwin DOT com
Subject: [CFT] libtool on nix->cygwin cross, with wine
References: <announce DOT 49A35E97 DOT 5090808 AT cwilson DOT fastmail DOT fm>
In-Reply-To: <announce.49A35E97.5090808@cwilson.fastmail.fm>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com

The most recent release of libtool (2.2.7a-1 for cygwin-1.5, and
2.2.7a-10 for cygwin-1.7) ought to support cross builds at least as well
as libtool-1.5 did.  Note that in *ordinary* cross builds (SomeBUILD ->
SomeHOST) you can't run the $host executables on the $build machine --
but you can still *build* them. That kind of thing has (hopefully)
always worked, and still works, for ->cygwin crosses.  However, under
certain conditions, it USED to be possible to run the $host (cygwin)
executables on the $build machine, provided $build's CPU was x86, and
$build had wine installed, and $build was running linux with the
'binfmt' kernel extension, and there was a cygwin "installation" in the
wine "arena", etc, etc.

/That/ has been broken for about a year, due to a change in how
"wrapper" executables are handled by libtool in libtool-2.2.2 and above.

============ 1 ==============
See, in the ANCIENT days, there was a wrapper script.  When you built an
executable for $host=cygwin, libtool would create

   myprog (a wrapper script)
   .libs/myprog.exe (the actual exe)

The wrapper script would set $PATH and various other environment
variables so that the EXE would be able to "find" its (as yet
uninstalled) DLLs, and then it would launch the EXE.  Obviously, scripts
are not bound to any specific platform, so $build has no problem running
the script.  So as long as $build could execute EXE (via wine, etc), the
the wrapper/EXE combo worked fine.

Why'd we change it? Well, there IS one little problem with that scheme:
the Makefile rule for building the EXE looks like this:

./my_prog$EXEEXT: my_prog.o <stuff>
	... some libtool commands ...

But libtool doesn't create ./my_prog$EXEEXT -- it creates a wrapper
named ./my_prog with NO $EXEEXT.  So make always believes that the
executable must be rebuilt.  'make' -> go link the exe. 'make check' ->
'go link the exe again'. 'make install' -> go link the exe. Very annoying.

============ 2 ==============
So, in the OLD days, we had a wrapper executable AND a wrapper script:

  my_prog.exe (not the real exe; just a wrapper exe)
  my_prog     (the wrapper script)
  .libs/my_prog.exe (the real exe)

The way this scheme worked was: (a) the wrapper exe would launch the
wrapper script, (b) the wrapper script would set $PATH and such, and
then (c) launch the real exe.  This worked pretty well -- and was fairly
transparent to cross builders: they'd try to run 'my_prog' (not
my_prog.exe, because who types "blahblah.EXE" on unix?). But, my_prog is
a shell script, and it Just Works(tm) like it did in the ANCIENT days.
So cross-builders were happy -- they just ignored that wrapper exe (and,
incidentally, never tested it...)

So, what was wrong with this?  Did we "fix" something that wasn't broken?

Well, not exactly. About three years ago, cygwin added a new feature:
you could set the 'transparent_exe' option in the CYGWIN variable. Then,
you could pretend you were even more "unixy": files which end in .exe to
be used by appropriate functions when an input filename is specified
with no extension.  That is, you say spawn("foo") and if foo.exe exists,
then cygwin will turn that in to spawn("foo.exe").

So...what if I have foo.exe AND foo, and a really really want to
spawn("foo") -- NOT spawn("foo.exe").  Say, for instance:

   my_foo.exe
   my_foo

Uh...don't do that.

See, transparent_exe caused all KINDS of pain for libtool -- cygwin (and
libtool) got really confused by the situation.  However, as long as
transparent_exe was just an option, we were ok though: you just had to
follow some rules:
  1) if transparent_exe, then do not put files that differ only in
.exe-extension in the same directory
  2) since libtool does this, do not use libtool and transparent_exe
together.
Not the greatest situation, but we lived with it.

Backgrounders:
http://cygwin.com/ml/cygwin/2006-03/msg00148.html
http://cygwin.com/ml/cygwin-apps/2006-03/msg00028.html
http://cygwin.com/ml/cygwin/2007-04/msg00543.html


But then, along comes cygwin-1.7...in which 'transparent_exe' will be
the default behavior. Oops.

"Don't use libtool and cygwin-1.7" is not a rule we can live with.

So, (and this is the part that broke the cross-builders), we changed
libtool's behavior...

============ 3 ==============
About two years ago, for $host cygwin/mingw, libtool was modified to no
longer put a wrapper *script* in the build dir. Instead, it put an
uglify-named version of it in .libs:

  my_prog.exe (not the real exe; just a wrapper exe)
  .libs/my_prog_ltshwrapper    (the wrapper script #1)
  .libs/my_prog_ltshwrapperTMP (the wrapper script #2)
  .libs/my_prog.exe (the real exe)

That way, no more 'transparent_exe' clashes. BUT, now the poor
cross-builders have no wrapper script to execute. They have to find the
uglified version in .libs, OR run the wrapper executable (which was
borked anyway in cross-build situations. See below).

Now, why two copies of the wrapper script? Well, libtool actually uses
the wrapper script for two different purposes. #1, it "saves"
information between different executions of libtool -- say, from the
link phase to the install phase. So, sometimes libtool actually
'sources' the wrapper script back in, to read certain variable values.
That's copy #1.  Copy #2 is ever-so-slightly different...

The way this scheme worked was: (a) the wrapper exe would emit the #2
copy of wrapper script in .libs/* ON THE FLY, (b) and execute that
wrapper script.  Then, the wrapper script would set $PATH, etc, and
launch the real exe.

That way, the #2 copy is ALWAYS up-to-date with the wrapper exe -- and,
because it is generated by the wrapper exe, it can be modified slightly
from copy #1 (which is used for communicating variable values from
different executions of libtool).
http://lists.gnu.org/archive/html/libtool-patches/2007-04/msg00052.html

This mostly worked, but was a bit awkward on cross-compile setups.  For
mingw, say, you had to set the environment variable TARGETSHELL to the
*dos* path to your *unix $build* shell, so that via the magic of wine,
CreateProcess("C:/some/magic/path/sh /path/to/wrapper/script args"...)
would cause the $build machine's /bin/sh to run the desired script.
Also, there were problems with atomicity: if you're building with make
-jN, you might launch the same wrapper exe multiple times
simultaneously. They all clobber each other's "temporary" wrapper scripts.

============ 4 ==============
So, the obvious solution is to have the wrapper executable itself do all
the work of setting $PATH, and then launching the real executable.
That's what happened in libtool-2.2.2 (about a year ago).  But, the
initial implementation worked great for native builds (cygwin, mingw) --
but completely "broke" cross-builds.  That is, you could still *build*
stuff in a nix->cygwin or nix->mingw environment, but the wrapper system
was completely fubared if you wanted to actually *run* those programs
via their wrappers, in a cross environment.  Even if you had wine installed.

The problem boils down to this (it's easier to explain by talking about
unix->mingw crosses, but similar arguments apply for unix->cygwin):
libtool itself is executing in $build (unix). It knows about
$build-style paths.  But the wrapper executable is a mingw executable,
and will use the win32 _spawnv("C:\I\Want\A\DOS\Path") function to
launch the real executable.  libtool knows the unix path to the real
executable -- when creating the "C" source code for the wrapper
executable, libtool needs to convert the unix-style (that is, $build)
path into the DOS-style (that is, $host) format.

So, recently (libtool-2.2.4ish) libtool learned how to do that -- for
unix->mingw.  It doesn't yet -- officially -- know how to do that for
unix->cygwin, because that is more complicated:

1) you have to use winepath to convert the $build (unix) path to "win32"
format.
2) THEN, you have to use cygpath -- running under wine -- to convert
that "win32" path to cygwin format "inside" wine.

It gets even more complicated if your wine "arena" has, for instance,
TWO different cygwin installations: a cygwin-1.5 one and a cygwin-1.7
one.  Which cygpath do you use -- the cygwin-1.5 cygpath which use the
"registry" in wine to find the cygwin mount table, and will convert the
"win32" path using that mount mapping? Or the cygwin-1.7 cygpath, which
 will find itself, locate its ../etc/fstab file, and use THAT for
converting the "win32" path to cygwin format.

Yikes!

======== CALL FOR TEST ========

This recent cygwin release of libtool has a mechanism for supporting
these sorts of cross builds.  BUT: you have to set an environment
variable in $build (that is, your unix environment) to tell libtool
which cygpath program you want to use:

export LT_CYGPATH=/unix/path/to/cygpath

Something like:

export LT_CYGPATH=/opt/winestuff/cygwin-1.7/bin/cygpath.exe

You also have to be running linux -- not any other unix -- with the
binfmt kernel extension turned on. It usually is, so you probably don't
need to worry about that one. (I don't THINK other unices support
anything like binfmt...)

Then, you can try to autoreconf (to get the new libtool stuff) and build
your favorite package using your unix->cygwin cross tools.

Finally: try to actually *execute* a wrapper executable -- and see if it
works.

I'm not set up to test that.  If you are, please try to exercise this
new capability and let me know if it works. The easiest way to do so is:

1) download libtool-2.2.7a-20090223.tar.lzma from here
http://cygwin.cwilson.fastmail.fm/ITP/libtool-2.2.7a-20090223.tar.lzma

(you COULD get libtool-2.2.7a-1-src.tar.bz2 from the cygwin mirrors,
BUT: it's a git checkout, with patches, so it's a bit of work. You have
to apply the patches in order, then run the bootstrap script.  The
.tar.lzma file above has already had all that done)

2) unpack, and create separate build directory
   $ ls
   libtool-2.2.7a-20090223/
   build/
3) export LT_CYGPATH=/unix/path/to/cygpath.exe
4) cd build/
5) ../libtool-2.2.7a-20090223/configure \
       --srcdir=../libtool-2.2.7a-20090223 \
       --build=i686-pc-linux-gnu
       --host=i686-pc-cygwin
6) make

Assuming that all goes well, then try:

7) make check TESTSUITEFLAGS="-V" TESTS='tests/mdemo-shared.test
tests/mdemo-make.test tests/mdemo-exec.test' VERBOSE=t

and see what happens.  If it works, GREAT. Tell me.

If not, then the interesting things to look at:
  a) did tests/mdemo-shared.test pass?
  b) did tests/mdemo-make.test pass?
  c) so, if tests/mdemo-exec.test failed, then what does
     build/tests/mdemo/.libs/lt-mdemo.c look like?
     are the following "C" variables correct:
        const char * LIB_PATH_VALUE
        const char * EXE_PATH_VALUE
  d) try rebuilding the wrapper with debugging:
     i)    cd build/tests/mdemo
     ii)   rm -f mdemo.exe # the wrapper
     iii)  make CFLAGS="-DLT_DEBUGWRAPPER" mdemo.exe
     then run the wrapper manually, and see what it prints out.

--
Chuck


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/