Subject: Re: DLL naming conventions
From: "Gary V. Vaughan" <[email-masked]>
To: "Charles S. Wilson" <[email-masked]>
Cc: Tor Lillqvist <[email-masked]>, [email-masked], [email-masked]
Date: Sat, 2 Sep 2000 23:19:25 +0100

On Sat, Sep 02, 2000 at 02:36:03PM -0400, Charles S. Wilson wrote:
> "Gary V. Vaughan" wrote:
> > 
> > Charles S. Wilson wrote:
> > >
> > > There is *something* in the windows system that gets notified that
> > > myapp.exe depends on mydll.dll whenever you execute myapp.exe.  This
> > > *something* searches the file system according to the search path
> > > described earlier in this thread, loads the dll into the executable
> > > process's memeory space (rebasing the memory location if necessary), and
> > > fixes up all the internal pointers to match the in-process virtual
> > > memory location.
> > >
> > > Isn't that considered a dynamic linker? (Whether you call that
> > > *something* a dynamic linker or not is really immaterial.  The
> > > *something* is a runtime loader, and that's what the reference above is
> > > discussing)
> > 
> > Nope.  Tor is correct.  The libtool versioning system assumes that the
> > runtime loader is smart enough to choose the most recent compatible
> > version of each dll that an application was linked with.  Many of the
> > operating systems that libtool supports have different rules for how
> > to name these versions so that the runtime loader can do it's stuff.
> > Libtool maps the unified version number scheme it exposes to the user
> > onto the versioning rules for the host platform.  That is why it is
> > okay to have libc5 and libc6 in the same directory on your Linux box.
> > Infact I could have several incompatible versions of libc6 in my lib
> > directory, and the runtime loader would choose the newest one that is
> > binary compatible with the version that the application was originally
> > linked with.
> 
> Hmmm....you can do this sort of thing -- in a limited way, of course --
> with appropriate naming of the dll and importlib.  Recall that import
> libs contain the hardcoded name of the dll for which they were built.
> 
> Also, there are two separate issues being discussed here: (1) how to
> (compile-time) link against the 'right' version of a library, and (2)
> how to insure that the 'right' dll is loaded at run-time.

Nah.  I'm talking about (2) only.  I *am* assuming that at link time
the only version of a library you can link with is the one that
corresponds with the header files you have installed.

> In the first
> case, 'right' just means the correct version that supports the necessary
> interface; in the second case, 'right' means the correct version
> compiled for the right environment -- since you can have windows-native
> dlls, Paul's unix-emulation-environment-based dll's (PW), cygwin dll's,
> UWin dll's -- all with (potentially) the same name.

Okay.  I probably wasn't thinking too hard about that...

> Since windows-dll
> loading is based on the executable path, and not 'LD_LIBRARY_PATH' or
> similar, you've got a problem.

Most definitely.

> It would be nice if you could create a 'cygwin LD_LIBARY_PATH' and a 'PW
> LD_LIBRARY_PATH' etc.  But that probably requires that you abandon dll's
> altogether, create your own 'so' file format, write an ld.so (ld.dll?)
> that will load .so's 'just like unix (tm)'.  And, for this to work
> cleanly, such a beast would have to be written for ALL the major unix
> emulation environments.  However, we're just concerned with cygwin,
> here.  (I can't take credit for this idea; both Tor Lillqvist and Chris
> Faylor have on occaision mentioned something similar)

In an ideal world.  I'd be surprised if anyone is keen enough to
actually write it though.  Far more likely that we can find the time
to make incremental improvements and adopt some conventions that make
things work a bit better...

> While not abandoning the preceding idea, it IS a lot of work (and there
> are a lot of implementation details to work out).  Although, it *will*
> fix point (1) *and* point (2) above.  
> 
> Now, you can (mostly) fix point (2: runtime loading) by the simple
> expedient of insuring that all dll's and exe's go into the same
> directory. Ugly, but much easier that the above semi-proposal.  An
> alternative solution to point (2) is to choose a unique prefix for dll's
> built on each platform -- e.g. Paul's suggestion of 'cygfoo.dll' for
> cygwin and 'pwfoo.dll' for his PW environment.  NOTE: you only need to
> munge the prefix of the dll itself; the import lib and/or static lib can
> still be 'libfoo.dll.a' or 'libfoo.a' -- because at
> compiletime/linktime, we *control* the library search path and are not
> limited by the global windows executable path.

I with you all the way so far...

> You can (mostly) fix point (1: compiletime version) by careful naming
> and use of import libs + careful naming and use of dll's.
> 
> For instance:
> 
>   libpng.dll.a  --> libpng3.dll.a  
>                 ^^
>               symlink
> 
>   libpng30.dll.a  (most recent API, corresponds to png-2.0.x)
>     embeds the name "libpng3.dll"
>   libpng20.dll.a  (older API, corresponds to png-1.0.x)
>     embeds the name "libpng2.dll"
> 
> Programs linked against '-lpng' will be dependent at runtime on
> libpng3.dll. Programs linked against -lpng2 (or against '-lpng' before
> version 2.0.x was installed -- e.g. when libpng.dll.a --> libpng2.dll.a)
> will be dependent at runtime on libpng2.dll

Why bother with the symlinks?  You can't link against -lpng2 unless
you have matching headers, and presumably you upgraded those when you
installed libpng3.dll.

My proposal was to start with this:

    libpng.dll.a embeds libpng2.dll
    
    and we keep libpng1.dll around for the applications that were
    linked against interface 1.
    
when we install png-2.0, which implements a new interface that is not
compatible with libpng2.dll, we install:

    a replacement libpng.dll.a that embeds libpng3.dll
    libpng3.dll
    
    and we keep libpng2.dll around for the applications that were
    linked against interface 2.

    and we keep libpng1.dll around for the applications that were
    linked against interface 1.

> Now, suppose that the png folks release a newer version, say png-2.1.x
> that exposes some additional features, but is backwards compatible with
> png-2.0.x.  Great.  You build it, but create:
> 
>   libpng31.dll.a (embeds the name libpng31.dll.a)
>   libpng.dll.a -> libpng31.dll.a
>   libpng3.dll (replaces the old version (*))

I don't think we need anything more than

    a replacement libpng3.dll
    
now applications that were linked against the libpng3.dll from png-2.0
will pick up the bugfixes from png-2.1, but new application can be
written for the header files installed with png-2.1 and take advantage
of the new features.

> Now, /usr/lib contains:
>   libpng.dll.a -> libpng31.dll.a
>   libpng31.dll.a
>   libpng30.dll.a
>   libpng20.dll.a
> and /usr/bin contains:
>   libpng3.dll
>   libpng2.dll
> 
> (*) you can't play simlink games with the dll's themselves, because the
> windows loader doesn't grok cygwin's symlinks.  You *can* play simlink
> games, as I've shown here, with the import libs, because the cygwin ld
> and cygwin gcc do grok symlinks.
> 
> This scheme will work without the help of libtool (e.g. you can still
> build & link dependent packages without being *forced* to use libtool)
> but it's similar enough to the 'unix way' that libtool should be able to
> handle it.

Except that I think you have an extra level of indirection that we can
safely do away with.  Otherwise we are still in violent agreement.

> > Libtool doesn't version dll's on Windows correctly right now, it uses
> > the linux versioning scheme to calculate the numbers.  It should
> > probably name them after the oldest ``interface'' (see the document
> > quoted above) that the library fully supports.  That is, if you build
> > a dll using libtool's --version 5:4:3, you would get library2.dll
>                                                        ^^^^^^^^^^^^
>                    I think you mean library3.dll (supports v5, v4, & v3)

Nope.  5:4:3 is revision 4 of the implementation of interface 5, which
is backwards compatible with the 3 previous interface definitions
(i.e. it is safe for applications linked against interfaces 5, 4, 3
and 2 to load the 5:4:3 dll at runtime).   Libtool translates the
5:4:3 into a system specific version number for the soname to help the
runtime loadee choose the best available library at runtime.  As I
said before, currently this mapping is wrong on Windows, and I think
the correct mapping is to always use the oldest supported interface
number -- in this case library2.dll -- when generating the soname.
This is explained fully in the version node of the libtool manual link
that was quoted earlier in the thread. 

> > (this version is at interface 5, but fully supports the last 3
> > interfaces).  This gives the windows loader the best chance of finding
> > the newest binary compatible dll.
> 
> In my scheme, --version 5:4:3 will give you library3.dll (assuming my
> correction above is right), but you'd also get:
> 
>  library.dll.a --> library5.dll.a
>  library5.dll.a (embeds the name library2.dll)
> 
> And these are still hanging around from previous builds
>  library4.dll.a (embeds the name library2.dll)
>  library3.dll.a (embeds the name library2.dll)
>  library2.dll.a (embeds the name library1.dll)
>  library1.dll

Okay we're diverging a little here.  I think that all you need is
library.dll.a, which always embeds the name of the dll that matches
the header files you have installed right now.  The point about
library1.dll and library2.dll is spot on though.

> > Unfortunately Windows makes no distinction between the application
> > search path and the library search path, so the scheme I propose is
> > still flawed =(O|  You might have a newer bugfixed binary compatible
> > version of the dll you need, but if it is further down the application
> > search path than the buggy version you originally linked with, you get
> > the broken version.  Nice one Bill...
> 
> Yeah, but you're still assuming that the link lib and the runtime lib
> are the same file.

Nope.  If we maintain the versions of library.dll according to either
of our schemes (which are broadly the same), it all breaks down if
some bozo installs a library2.dll in a location higher up our
*application* search path that the directory we are using, since our
applications will load their dll instead.  Maybe ours is the
library5.dll.a version with 2 releases worth of bugfixes and
enhancements over their library3.dll.a version they installed.

> > Maybe Cygwin should put all of it's libtool built libraries (or any
> > others that are correctly versioned w.r.t the runtime loader) into a
> > single directory that is near the front of the default path.  /usr/lib
> > seems like a good place.
> 
> That's part of my 'easy' solution above.  It may fix the cygwin problem,
> but the requirement is not nice to native apps or other unix-on-win
> emulations that the user may have.
> 
> Relying on 'get dll's from same dir as .exe' works -- but only if EVERY
> cygwin exe and EVERY cygwin dll are piled into the same dir.

Can't we assume that the native dll's we rely on are outside cygwin's
control and can be relied upon to be managed by the operating system
(I'm thinking of stuff in the SYSTEM directory), and that non-cygwin
libraries (such as Pauls pw dlls) will not be installed into the
cygwin binary search path?  In which case I think it is okay to
install all cygwin dlls and import libs to /usr/lib, making sure that
cygwin.bat (or any other cygwin startup methods) put /usr/lib first in
$PATH.  Does cygwin ld use -rpath yet?  If so we could encode
/usr/lib into each application as the first place to *always* look
when searching for a dll. (I guess I am asking whether the Windows
runtime loader will honor whatever rpath directory cygwin ld may
encode into an application -- as happens with all of the Unix ports of
libtool).

> I *think* the combination of implib/dll versioning described above, plus
> (sigh) unique prefixes for different platforms -- e.g. 'cygfoo.dll' --
> will solve most of the problems brought up w.r.t compiletime/linktime
> versioning, and runtime loading.  Without interfering too much with
> native apps or other unix-emul packages.  So what do you lose?
> 
> -1) Can't link directly to a dll; you must use an import lib.  (why? #1,
> ld only hunts for libfoo.dll & foo.dll, not cygfoo.dll. #2, if you link
> directly to dll's without using import libs, you lose the clean
> separation between 'runtime links' and 'development links' -- see
> http://www.debian.org/Lists-Archives/lsb-spec-9905/msg00011.html.  I
> know, there aren't really any 'runtime' symlinks, but at least you can
> separate the major so-names...)

The cygfoo point is well taken.  I'm not sure I would accept that
patch anymore...  infact I think that it is not a necessary part of
this proposal (either of our proposals infact) if we have some way to
control the runtime dll search path (by always putting /usr/lib first
in the PATH for example).

> -2) It's a helluva lot of work to change everything on a platform (say,
> cygwin) so that all dll's start with 'cyg' instead of 'lib'

This may be a non issue if you agree with my last paragraph.

> -3) It mostly solves a lot of the issues, but not completely. Not
> ideally. Not perfectly.

Hey.  This is Windows we're talking about =)O|

> -4) Still requires significant changes to libtool to handle dll's and
> importlibs.

Not that significant.  And this is all stuff that is on libtool's TODO
list anyway.  It will get done eventually, just as soon as we can nail
down a proposal that everyone (well most of us at least) likes.

> What do you gain?
> 
> +1) mostly solves the problem
> +2) you can still link dynamically or statically, without the use of
> libtool if you so choose

That is important, and I am prepared to accept a patch (nay, write a
patch in the fullness of time) to have libtool deal with import libs
properly.

> +3) you don't have to develop an ld.so 'smart' runtime loader
> 
> Is +3) a bigger win than work involved in -2) ?

I think so.  Unless someone steps up to the plate and offers to
implement an ld.so-a-like, I am prepared to work on having libtool do
the right thing as far as possible.

> Will Chris, DJ, Corinna, et. al. even go for -2) ?

Guys?  Stop me now if you violently disagree with libtool taking this
direction =)O|

> Developing an ld.so runtime loader requires far fewer changes to
> libtool, and since you *can* link directly to dll's (*) it'll work 'just
> like unix (tm). But developing ld.so will be a tough job.
> 
> (*) DJ has pushed a feature into ld that generates a 'virtual' import
> lib on-the-fly if you put a dll in the link line.

Yup. I picked his brains while he was doing it, and wrote something
similar for libtool.  By default libtool always searches for a dll to
link against and generates the implib on the fly if a suitable one is
found.

Cheers,
	Gary.
-- 
  ___              _   ___   __              _         mailto: [email-masked]
 / __|__ _ _ ___ _| | / / | / /_ _ _  _ __ _| |_  __ _ ___       [email-masked] 
| (_ / _` | '_|// / |/ /| |/ / _` | || / _` | ' \/ _` | _ \
 \___\__,_|_|\_, /|___(_)___/\__,_|\_,_\__, |_||_\__,_|//_/
home page:  /___/                      /___/                  gpg public key:
http://www.oranda.demon.co.uk           http://www.oranda.demon.co.uk/key.asc

--
Want to unsubscribe from this list?
Send a message to [email-masked]