Mailing-List: contact cygwin-help AT sourceware DOT cygnus DOT com; run by ezmlm
Sender: cygwin-owner AT sources DOT redhat DOT com
Message-ID: <39B1E2DF.2F8E9A7F@ece.gatech.edu>
Date: Sun, 03 Sep 2000 01:34:23 -0400
From: Charles Wilson <cwilson AT ece DOT gatech DOT edu>
MIME-Version: 1.0
To: "Gary V. Vaughan" <gvv AT techie DOT com>
CC: Tor Lillqvist <tml AT iki DOT fi>, cygwin AT sources DOT redhat DOT com, libtool AT gnu DOT org
Subject: Re: DLL naming conventions
References: <200009011148 DOT OAA23769 AT urkki DOT tellabs DOT fi> <39AFB333 DOT 69CE5F95 AT ece DOT gatech DOT edu> <20000902140958 DOT F7695 AT demon DOT co DOT uk> <39B14893 DOT A86AF3F9 AT ece DOT gatech DOT edu> <20000902231925 DOT Q7695 AT demon DOT co DOT uk>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

"Gary V. Vaughan" wrote:
> >
> > Also, there are two separate issues being discussed here: (1) how to
> > (compile-time) link against the 'right' version of a library, and (2)
> > how to insure that the 'right' dll is loaded at run-time.
> 
> Nah.  I'm talking about (2) only.  I *am* assuming that at link time
> the only version of a library you can link with is the one that
> corresponds with the header files you have installed.

Okay. I was just going by the example from
http://www.debian.org/Lists-Archives/lsb-spec-9905/msg00011.html, quoted
here:

-------------------
> The development links become a bit more complex. The temptation is to
> place both development sets in their own subdirectories, providing:
> 
>      /usr/include/gmp1
> and
>      /usr/include/gmp2
> 
> and the development symlinks:
> 
>      libgmp1.so
> and
>      libgmp2.so
> 
> This causes problems for packages which expect to build against gmp.
> Having to edit the make file to reflect the correct include and link names
> is not a clean solution.
> 
> Thus, for gmp-2.0.2, the includes are provided in /usr/include, and the
> development link is libgmp.so. Only gmp1 need be placed in the special
> locations above.
--------------------

But, you are right. The extra linktime libaries are only necessary if
you also have the extra header files in a special directory.  Since you
have to edit the makefiles (and possible the source code) for 'package
X' that depends on the old interface (gmp1 in this example), then you
might as well specify '-lgmp1' and you don't need the symlinks.

<snip>
 
> > Since windows-dll
> > loading is based on the executable path, and not 'LD_LIBRARY_PATH' or
> > similar, you've got a problem.
> 
> Most definitely.

Chris argues against this in his other message. I'll reply to that point
in a separt message.

> 
> > It would be nice if you could create a 'cygwin LD_LIBARY_PATH' and a 'PW
> > LD_LIBRARY_PATH' etc.  But that probably requires that you abandon dll's
> > altogether, create your own 'so' file format, write an ld.so (ld.dll?)
> > that will load .so's 'just like unix (tm)'.  And, for this to work
> > cleanly, such a beast would have to be written for ALL the major unix
> > emulation environments.  However, we're just concerned with cygwin,
> > here.  (I can't take credit for this idea; both Tor Lillqvist and Chris
> > Faylor have on occaision mentioned something similar)
> 
> In an ideal world.  I'd be surprised if anyone is keen enough to
> actually write it though.  Far more likely that we can find the time
> to make incremental improvements and adopt some conventions that make
> things work a bit better...

I think the ld.so stuff is very ambitious. IMO, it's something to keep
in mind for the next major revision of cygwin -- but then, I'm not an
official spokesman.

Again, IMO, 'adopting some conventions' is the way to go right now --
just like we adopted the convention that 'libfoo.dll.a' means an import
lib.

<snip>

> > You can (mostly) fix point (1: compiletime version) by careful naming
> > and use of import libs + careful naming and use of dll's.
> >
> > For instance:
> >
> >   libpng.dll.a  --> libpng3.dll.a
> >                 ^^
> >               symlink
> >
> >   libpng30.dll.a  (most recent API, corresponds to png-2.0.x)
> >     embeds the name "libpng3.dll"
> >   libpng20.dll.a  (older API, corresponds to png-1.0.x)
> >     embeds the name "libpng2.dll"
> >
> > Programs linked against '-lpng' will be dependent at runtime on
> > libpng3.dll. Programs linked against -lpng2 (or against '-lpng' before
> > version 2.0.x was installed -- e.g. when libpng.dll.a --> libpng2.dll.a)
> > will be dependent at runtime on libpng2.dll
> 
> Why bother with the symlinks?  You can't link against -lpng2 unless
> you have matching headers, and presumably you upgraded those when you
> installed libpng3.dll.
> 
> My proposal was to start with this:
> 
>     libpng.dll.a embeds libpng2.dll
> 
>     and we keep libpng1.dll around for the applications that were
>     linked against interface 1.
> 
> when we install png-2.0, which implements a new interface that is not
> compatible with libpng2.dll, we install:
> 
>     a replacement libpng.dll.a that embeds libpng3.dll
>     libpng3.dll
> 
>     and we keep libpng2.dll around for the applications that were
>     linked against interface 2.
> 
>     and we keep libpng1.dll around for the applications that were
>     linked against interface 1.

You are (mostly) right. The symlinks are not really necessary -- but
keeping libpng20.dll.a is a good idea (assuming you also kept the 2.0.x
headers around, and put them in /usr/include/png2/.  

The symlink buys you ease of installation for the implib.  Suppose you
had 1.0.x installed, so libpng.dll.a embeds libpng2.dll.  Then, you
install 2.0.x.  First, you need to rename libpng.dll.a to libpng2.dll.a,
and then install the new implib as libpng.dll.a.  With symlinks, none of
that is necessary -- just install the new implib as libpng30.dll.a and
replace the existing symlink with one that points to the new implib.
Easy.

BUT -- you still need smarts create the appropriate /usr/include/png2/
directory, move the header files into it, and then install the new
headers into /usr/include.

Yuck yuck and double yuck. Okay, forget the symlinks -- they help with
the implibs, but nothin' helps with the header files.

The "Right Thing To Do", if you want development stuff (compile time
headers, link time import libs) for multiple incompatible versions of a
lib, is for the maintainer (the cygwin person who maintains the port of
a particular package, on sourceware -- for libpng, me) to do the
following:

 Oh, look: libpng-2.0.x is out.  That's not compatible with
libpng-1.0.x.  I need to create a new libpng-1.0.x package that installs
thusly:
  /usr/include/png2/*.h
  /usr/bin/libpng2.dll
  /usr/lib/libpng2.dll.a (libpng20.dll.a?)

Then, I can create a new libpng-2.0.x package that installs like so:
  /usr/include/*.h
  /usr/bin/libpng3.dll
  /usr/lib/libpng3.dll.a (libpng30.dll.a?)

And tell folks to install both the updated 1.0.x package and the new
2.0.x package.  Just like you see done in the linux distributions:
especially gmp1 and gmp2.

> 
> > Now, suppose that the png folks release a newer version, say png-2.1.x
> > that exposes some additional features, but is backwards compatible with
> > png-2.0.x.  Great.  You build it, but create:
> >
> >   libpng31.dll.a (embeds the name libpng31.dll.a)
> >   libpng.dll.a -> libpng31.dll.a
> >   libpng3.dll (replaces the old version (*))
> 
> I don't think we need anything more than
> 
>     a replacement libpng3.dll

Not so.  Remember, libpng-2.1.x can include additional functions that
were not included by libpng-2.0.x, as long as it includes all of the
'old' functions/vars without modification.  So, you need to update the
import lib as well as the dll, so that new packages that use the
additional functions will work.

> 
> now applications that were linked against the libpng3.dll from png-2.0
> will pick up the bugfixes from png-2.1, but new application can be
> written for the header files installed with png-2.1 and take advantage
> of the new features.

Unless you link directly against the dll (and don't use the import lib)
this new application won't link -- because the new features (additional
functions) added in the new version of the png library won't be exposed
by the old implib.

> 
> > Now, /usr/lib contains:
> >   libpng.dll.a -> libpng31.dll.a
> >   libpng31.dll.a
> >   libpng30.dll.a
> >   libpng20.dll.a
> > and /usr/bin contains:
> >   libpng3.dll
> >   libpng2.dll
> >
> > (*) you can't play simlink games with the dll's themselves, because the
> > windows loader doesn't grok cygwin's symlinks.  You *can* play simlink
> > games, as I've shown here, with the import libs, because the cygwin ld
> > and cygwin gcc do grok symlinks.
> >
> > This scheme will work without the help of libtool (e.g. you can still
> > build & link dependent packages without being *forced* to use libtool)
> > but it's similar enough to the 'unix way' that libtool should be able to
> > handle it.
> 
> Except that I think you have an extra level of indirection that we can
> safely do away with.  Otherwise we are still in violent agreement.

I agree. The symlinks are unnecessary, because of the difficulties
imposed by keeping track of the header files for the old version.  You
just need to create a new package for the old library, that relocates
and renames the implib and headers appropriately.  

> 
> > > Libtool doesn't version dll's on Windows correctly right now, it uses
> > > the linux versioning scheme to calculate the numbers.  It should
> > > probably name them after the oldest ``interface'' (see the document
> > > quoted above) that the library fully supports.  That is, if you build
> > > a dll using libtool's --version 5:4:3, you would get library2.dll
> >                                                        ^^^^^^^^^^^^
> >                    I think you mean library3.dll (supports v5, v4, & v3)
> 
> Nope.  5:4:3 is revision 4 of the implementation of interface 5, which
> is backwards compatible with the 3 previous interface definitions
> (i.e. it is safe for applications linked against interfaces 5, 4, 3
> and 2 to load the 5:4:3 dll at runtime).   Libtool translates the
> 5:4:3 into a system specific version number for the soname to help the
> runtime loadee choose the best available library at runtime.  As I
> said before, currently this mapping is wrong on Windows, and I think
> the correct mapping is to always use the oldest supported interface
> number -- in this case library2.dll -- when generating the soname.
> This is explained fully in the version node of the libtool manual link
> that was quoted earlier in the thread.

Doh!  0-based numbering. Not 1-based numbering.

> 
> > > (this version is at interface 5, but fully supports the last 3
> > > interfaces).  This gives the windows loader the best chance of finding
> > > the newest binary compatible dll.
> >
> > In my scheme, --version 5:4:3 will give you library3.dll (assuming my
> > correction above is right), but you'd also get:
> >
> >  library.dll.a --> library5.dll.a
> >  library5.dll.a (embeds the name library2.dll)
> >
> > And these are still hanging around from previous builds
> >  library4.dll.a (embeds the name library2.dll)
> >  library3.dll.a (embeds the name library2.dll)
> >  library2.dll.a (embeds the name library1.dll)
> >  library1.dll
> 
> Okay we're diverging a little here.  I think that all you need is
> library.dll.a, which always embeds the name of the dll that matches
> the header files you have installed right now.  The point about
> library1.dll and library2.dll is spot on though.

You are correct, unless one also installs the appropriate header files
in 
  /usr/include/library4/
  /usr/include/library3/
  /usr/include/library2/

> 
> > > Unfortunately Windows makes no distinction between the application
> > > search path and the library search path, so the scheme I propose is
> > > still flawed =(O|  You might have a newer bugfixed binary compatible
> > > version of the dll you need, but if it is further down the application
> > > search path than the buggy version you originally linked with, you get
> > > the broken version.  Nice one Bill...
> >
> > Yeah, but you're still assuming that the link lib and the runtime lib
> > are the same file.
> 
> Nope.  If we maintain the versions of library.dll according to either
> of our schemes (which are broadly the same), it all breaks down if
> some bozo installs a library2.dll in a location higher up our
> *application* search path that the directory we are using, since our
> applications will load their dll instead.  Maybe ours is the
> library5.dll.a version with 2 releases worth of bugfixes and
> enhancements over their library3.dll.a version they installed.

This is true.

> 
> > > Maybe Cygwin should put all of it's libtool built libraries (or any
> > > others that are correctly versioned w.r.t the runtime loader) into a
> > > single directory that is near the front of the default path.  /usr/lib
> > > seems like a good place.
> >
> > That's part of my 'easy' solution above.  It may fix the cygwin problem,
> > but the requirement is not nice to native apps or other unix-on-win
> > emulations that the user may have.
> >
> > Relying on 'get dll's from same dir as .exe' works -- but only if EVERY
> > cygwin exe and EVERY cygwin dll are piled into the same dir.
> 
> Can't we assume that the native dll's we rely on are outside cygwin's
> control and can be relied upon to be managed by the operating system
> (I'm thinking of stuff in the SYSTEM directory), and that non-cygwin
> libraries (such as Pauls pw dlls) will not be installed into the
> cygwin binary search path?  In which case I think it is okay to
> install all cygwin dlls and import libs to /usr/lib, making sure that
> cygwin.bat (or any other cygwin startup methods) put /usr/lib first in
> $PATH.  

this works for the 'sandbox user' -- Michael Ring's 'user 1' in this
message:
   http://sources.redhat.com/ml/cygwin/2000-08/msg01241.html

It doesn't work for Michael's 'user 2' -- the guy who normally runs in
cmd.exe/command.com, but relies on cygwin commands every once in a
while.  He doesn't use bash, but likes the cygwin-perl or grep every
once and a while.  User 2 will have the cygwin directories somewhere in
her path -- but not necessarily first.

> Does cygwin ld use -rpath yet?  

I don't think so.  -rpath is something that ld.so uses; Windows doesn't
have ld.so. Windows *always* loads dll's according to the following
search order:
   current directory
   app's load directory
   global executable search path

The only two exceptions I know of are:

  1) In Win2K, if there is a file called 'app.exe.local' in the same
directory as app.exe, then all dll's will be loaded from the app's load
directory -- even if explicitly dlopened() with an absolute path that
points elsewhere.  the .local. may also override the 'current directory'
part of the search order listed above, but I'm not sure.

  2) You can put something called 'AppPath' in the registry, which will
influence the directories that are searched. I don't know where in the
list above that the directories listed in the 'AppPath' key are
inserted.

> If so we could encode
> /usr/lib into each application as the first place to *always* look
> when searching for a dll. (I guess I am asking whether the Windows
> runtime loader will honor whatever rpath directory cygwin ld may
> encode into an application -- as happens with all of the Unix ports of
> libtool).

Nope. Don't think so.  That's why the suggestion to implement an
ld.so-like loader specifically for cygwin is an attractive, but
extremely difficult, option.

> 
> > I *think* the combination of implib/dll versioning described above, plus
> > (sigh) unique prefixes for different platforms -- e.g. 'cygfoo.dll' --
> > will solve most of the problems brought up w.r.t compiletime/linktime
> > versioning, and runtime loading.  Without interfering too much with
> > native apps or other unix-emul packages.  So what do you lose?
> >
> > -1) Can't link directly to a dll; you must use an import lib.  (why? #1,
> > ld only hunts for libfoo.dll & foo.dll, not cygfoo.dll. #2, if you link
> > directly to dll's without using import libs, you lose the clean
> > separation between 'runtime links' and 'development links' -- see
> > http://www.debian.org/Lists-Archives/lsb-spec-9905/msg00011.html.  I
> > know, there aren't really any 'runtime' symlinks, but at least you can
> > separate the major so-names...)
> 
> The cygfoo point is well taken.  I'm not sure I would accept that
> patch anymore...  infact I think that it is not a necessary part of
> this proposal (either of our proposals infact) if we have some way to
> control the runtime dll search path (by always putting /usr/lib first
> in the PATH for example).

Again, works for Michael's 'user 1', but not for 'user 2'.  And there
are a lot of 'user 2's out there.

> 
> > -2) It's a helluva lot of work to change everything on a platform (say,
> > cygwin) so that all dll's start with 'cyg' instead of 'lib'
> 
> This may be a non issue if you agree with my last paragraph.
> 
> > -3) It mostly solves a lot of the issues, but not completely. Not
> > ideally. Not perfectly.
> 
> Hey.  This is Windows we're talking about =)O|
> 
> > -4) Still requires significant changes to libtool to handle dll's and
> > importlibs.
> 
> Not that significant.  And this is all stuff that is on libtool's TODO
> list anyway.  It will get done eventually, just as soon as we can nail
> down a proposal that everyone (well most of us at least) likes.
> 
> > What do you gain?
> >
> > +1) mostly solves the problem
> > +2) you can still link dynamically or statically, without the use of
> > libtool if you so choose
> 
> That is important, and I am prepared to accept a patch (nay, write a
> patch in the fullness of time) to have libtool deal with import libs
> properly.
> 
> > +3) you don't have to develop an ld.so 'smart' runtime loader
> >
> > Is +3) a bigger win than work involved in -2) ?
> 
> I think so.  Unless someone steps up to the plate and offers to
> implement an ld.so-a-like, I am prepared to work on having libtool do
> the right thing as far as possible.

I think libtool should do the right thing anyway; even if an ld.so-clone
is proposed and work begins, it'll be a LONNNGGG time before a beast
like that is ready for prime time.

What was the right thing, again?  :-)

> 
> > Will Chris, DJ, Corinna, et. al. even go for -2) ?
> 
> Guys?  Stop me now if you violently disagree with libtool taking this
> direction =)O|
> 
> > Developing an ld.so runtime loader requires far fewer changes to
> > libtool, and since you *can* link directly to dll's (*) it'll work 'just
> > like unix (tm). But developing ld.so will be a tough job.
> >
> > (*) DJ has pushed a feature into ld that generates a 'virtual' import
> > lib on-the-fly if you put a dll in the link line.
> 
> Yup. I picked his brains while he was doing it, and wrote something
> similar for libtool.  By default libtool always searches for a dll to
> link against and generates the implib on the fly if a suitable one is
> found.

There are occaisions where you want to link to an import lib in
preference to a dll -- for instance, libcygwin.a is an import lib, but
contains initializer code and actual function implementations for some
functions that are not included in the dll itself.  If you attempt to
link directly to cygwin1.dll, the link fails because those things are
missing from the virtual on-the-fly implib.

These cases are rare.

It's possible that we can do away completely with import libs for most
libraries: based on my tests, all of the following four scenarios work:

libfoo.dll & libfoo.dll.a compiled with __declspec(dllexport)
decorations on all functions and data
use-libfoo.o compiled with __declspec(dllimport) decorations on all
functions and data

link use-libfoo.o to libpfoo.dll.a
  -- generates a working executable

link use-libfoo.o to libfoo.dll
  -- generates a working executable

libfoo.dll & libfoo.dll.a compiled with __declspec(dllexport)
decorations *ONLY* on data, not functions
use-libfoo.o compiled with __declspec(dllimport) decorations *ONLY* on
data, not functions

link use-libfoo.o to libpfoo.dll.a
  -- generates a working executable

link use-libfoo.o to libfoo.dll
  -- generates a working executable

So, it seems that there is no case in which the import lib gives you
better behavior than the dll itself.

But where do you put the dll?  It has to go into the executable PATH so
that the windows loader can find it.  Do you copy it into /usr/lib, so
that the default ld search path will find it?  Do you add /usr/bin to
the linktime library search path (-L/usr/bin)?  Perhaps a symlink in
/usr/lib, pointing to /usr/bin/libfoo.dll is all you need. 

However!!! Ld uses the following library name search order when hunting
for -lfoo:

  libfoo.dll.a
  foo.dll.a
  libfoo.a  <<<< NOTE!!
  libfoo.dll
  foo.dll

So, if both the static lib (libfoo.a) and the symlink/copy of the dll
(libfoo.dll) exist in /usr/lib, the link will grab the static lib in
preference to linking directly to the dll.  (this was a necessary
compromise for backwards compatibility; if libfoo.dll can override
libfoo.a, many things break. Don't go there).  Therefore, in order for
dynamic linking to take precedence over static linking, you have to have
an import lib in /usr/lib, not the dll.  Naturally, since libtool is
smart, it could explicitly link the dll in preference to the static lib
by listing the actual pathname of the library on the link line, instead
of relying on '-lfoo', but I thought we're trying to come up with a
solution that allows both libtool-ization and normal, unassisted
compiles.

Therefore:
  /usr/lib/libfoo.dll.a
  /usr/lib/libfoo.a
  /usr/bin/libfoo.dll  (maybe libfoo3.dll, whatever)

--Chuck

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe AT sourceware DOT cygnus DOT com