Mailing-List: contact cygwin-help AT sourceware DOT cygnus DOT com; run by ezmlm List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT sources DOT redhat DOT com Delivered-To: mailing list cygwin AT sources DOT redhat DOT com Date: Sat, 2 Sep 2000 23:19:25 +0100 From: "Gary V. Vaughan" To: "Charles S. Wilson" Cc: Tor Lillqvist , cygwin AT sources DOT redhat DOT com, libtool AT gnu DOT org Subject: Re: DLL naming conventions Message-ID: <20000902231925.Q7695@demon.co.uk> References: <200009011148 DOT OAA23769 AT urkki DOT tellabs DOT fi> <39AFB333 DOT 69CE5F95 AT ece DOT gatech DOT edu> <20000902140958 DOT F7695 AT demon DOT co DOT uk> <39B14893 DOT A86AF3F9 AT ece DOT gatech DOT edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <39B14893.A86AF3F9@ece.gatech.edu>; from cwilson@ece.gatech.edu on Sat, Sep 02, 2000 at 02:36:03PM -0400 X-Operating-System: Linux oranda 2.2.13 On Sat, Sep 02, 2000 at 02:36:03PM -0400, Charles S. Wilson wrote: > "Gary V. Vaughan" wrote: > > > > Charles S. Wilson wrote: > > > > > > There is *something* in the windows system that gets notified that > > > myapp.exe depends on mydll.dll whenever you execute myapp.exe. This > > > *something* searches the file system according to the search path > > > described earlier in this thread, loads the dll into the executable > > > process's memeory space (rebasing the memory location if necessary), and > > > fixes up all the internal pointers to match the in-process virtual > > > memory location. > > > > > > Isn't that considered a dynamic linker? (Whether you call that > > > *something* a dynamic linker or not is really immaterial. The > > > *something* is a runtime loader, and that's what the reference above is > > > discussing) > > > > Nope. Tor is correct. The libtool versioning system assumes that the > > runtime loader is smart enough to choose the most recent compatible > > version of each dll that an application was linked with. Many of the > > operating systems that libtool supports have different rules for how > > to name these versions so that the runtime loader can do it's stuff. > > Libtool maps the unified version number scheme it exposes to the user > > onto the versioning rules for the host platform. That is why it is > > okay to have libc5 and libc6 in the same directory on your Linux box. > > Infact I could have several incompatible versions of libc6 in my lib > > directory, and the runtime loader would choose the newest one that is > > binary compatible with the version that the application was originally > > linked with. > > Hmmm....you can do this sort of thing -- in a limited way, of course -- > with appropriate naming of the dll and importlib. Recall that import > libs contain the hardcoded name of the dll for which they were built. > > Also, there are two separate issues being discussed here: (1) how to > (compile-time) link against the 'right' version of a library, and (2) > how to insure that the 'right' dll is loaded at run-time. Nah. I'm talking about (2) only. I *am* assuming that at link time the only version of a library you can link with is the one that corresponds with the header files you have installed. > In the first > case, 'right' just means the correct version that supports the necessary > interface; in the second case, 'right' means the correct version > compiled for the right environment -- since you can have windows-native > dlls, Paul's unix-emulation-environment-based dll's (PW), cygwin dll's, > UWin dll's -- all with (potentially) the same name. Okay. I probably wasn't thinking too hard about that... > Since windows-dll > loading is based on the executable path, and not 'LD_LIBRARY_PATH' or > similar, you've got a problem. Most definitely. > It would be nice if you could create a 'cygwin LD_LIBARY_PATH' and a 'PW > LD_LIBRARY_PATH' etc. But that probably requires that you abandon dll's > altogether, create your own 'so' file format, write an ld.so (ld.dll?) > that will load .so's 'just like unix (tm)'. And, for this to work > cleanly, such a beast would have to be written for ALL the major unix > emulation environments. However, we're just concerned with cygwin, > here. (I can't take credit for this idea; both Tor Lillqvist and Chris > Faylor have on occaision mentioned something similar) In an ideal world. I'd be surprised if anyone is keen enough to actually write it though. Far more likely that we can find the time to make incremental improvements and adopt some conventions that make things work a bit better... > While not abandoning the preceding idea, it IS a lot of work (and there > are a lot of implementation details to work out). Although, it *will* > fix point (1) *and* point (2) above. > > Now, you can (mostly) fix point (2: runtime loading) by the simple > expedient of insuring that all dll's and exe's go into the same > directory. Ugly, but much easier that the above semi-proposal. An > alternative solution to point (2) is to choose a unique prefix for dll's > built on each platform -- e.g. Paul's suggestion of 'cygfoo.dll' for > cygwin and 'pwfoo.dll' for his PW environment. NOTE: you only need to > munge the prefix of the dll itself; the import lib and/or static lib can > still be 'libfoo.dll.a' or 'libfoo.a' -- because at > compiletime/linktime, we *control* the library search path and are not > limited by the global windows executable path. I with you all the way so far... > You can (mostly) fix point (1: compiletime version) by careful naming > and use of import libs + careful naming and use of dll's. > > For instance: > > libpng.dll.a --> libpng3.dll.a > ^^ > symlink > > libpng30.dll.a (most recent API, corresponds to png-2.0.x) > embeds the name "libpng3.dll" > libpng20.dll.a (older API, corresponds to png-1.0.x) > embeds the name "libpng2.dll" > > Programs linked against '-lpng' will be dependent at runtime on > libpng3.dll. Programs linked against -lpng2 (or against '-lpng' before > version 2.0.x was installed -- e.g. when libpng.dll.a --> libpng2.dll.a) > will be dependent at runtime on libpng2.dll Why bother with the symlinks? You can't link against -lpng2 unless you have matching headers, and presumably you upgraded those when you installed libpng3.dll. My proposal was to start with this: libpng.dll.a embeds libpng2.dll and we keep libpng1.dll around for the applications that were linked against interface 1. when we install png-2.0, which implements a new interface that is not compatible with libpng2.dll, we install: a replacement libpng.dll.a that embeds libpng3.dll libpng3.dll and we keep libpng2.dll around for the applications that were linked against interface 2. and we keep libpng1.dll around for the applications that were linked against interface 1. > Now, suppose that the png folks release a newer version, say png-2.1.x > that exposes some additional features, but is backwards compatible with > png-2.0.x. Great. You build it, but create: > > libpng31.dll.a (embeds the name libpng31.dll.a) > libpng.dll.a -> libpng31.dll.a > libpng3.dll (replaces the old version (*)) I don't think we need anything more than a replacement libpng3.dll now applications that were linked against the libpng3.dll from png-2.0 will pick up the bugfixes from png-2.1, but new application can be written for the header files installed with png-2.1 and take advantage of the new features. > Now, /usr/lib contains: > libpng.dll.a -> libpng31.dll.a > libpng31.dll.a > libpng30.dll.a > libpng20.dll.a > and /usr/bin contains: > libpng3.dll > libpng2.dll > > (*) you can't play simlink games with the dll's themselves, because the > windows loader doesn't grok cygwin's symlinks. You *can* play simlink > games, as I've shown here, with the import libs, because the cygwin ld > and cygwin gcc do grok symlinks. > > This scheme will work without the help of libtool (e.g. you can still > build & link dependent packages without being *forced* to use libtool) > but it's similar enough to the 'unix way' that libtool should be able to > handle it. Except that I think you have an extra level of indirection that we can safely do away with. Otherwise we are still in violent agreement. > > Libtool doesn't version dll's on Windows correctly right now, it uses > > the linux versioning scheme to calculate the numbers. It should > > probably name them after the oldest ``interface'' (see the document > > quoted above) that the library fully supports. That is, if you build > > a dll using libtool's --version 5:4:3, you would get library2.dll > ^^^^^^^^^^^^ > I think you mean library3.dll (supports v5, v4, & v3) Nope. 5:4:3 is revision 4 of the implementation of interface 5, which is backwards compatible with the 3 previous interface definitions (i.e. it is safe for applications linked against interfaces 5, 4, 3 and 2 to load the 5:4:3 dll at runtime). Libtool translates the 5:4:3 into a system specific version number for the soname to help the runtime loadee choose the best available library at runtime. As I said before, currently this mapping is wrong on Windows, and I think the correct mapping is to always use the oldest supported interface number -- in this case library2.dll -- when generating the soname. This is explained fully in the version node of the libtool manual link that was quoted earlier in the thread. > > (this version is at interface 5, but fully supports the last 3 > > interfaces). This gives the windows loader the best chance of finding > > the newest binary compatible dll. > > In my scheme, --version 5:4:3 will give you library3.dll (assuming my > correction above is right), but you'd also get: > > library.dll.a --> library5.dll.a > library5.dll.a (embeds the name library2.dll) > > And these are still hanging around from previous builds > library4.dll.a (embeds the name library2.dll) > library3.dll.a (embeds the name library2.dll) > library2.dll.a (embeds the name library1.dll) > library1.dll Okay we're diverging a little here. I think that all you need is library.dll.a, which always embeds the name of the dll that matches the header files you have installed right now. The point about library1.dll and library2.dll is spot on though. > > Unfortunately Windows makes no distinction between the application > > search path and the library search path, so the scheme I propose is > > still flawed =(O| You might have a newer bugfixed binary compatible > > version of the dll you need, but if it is further down the application > > search path than the buggy version you originally linked with, you get > > the broken version. Nice one Bill... > > Yeah, but you're still assuming that the link lib and the runtime lib > are the same file. Nope. If we maintain the versions of library.dll according to either of our schemes (which are broadly the same), it all breaks down if some bozo installs a library2.dll in a location higher up our *application* search path that the directory we are using, since our applications will load their dll instead. Maybe ours is the library5.dll.a version with 2 releases worth of bugfixes and enhancements over their library3.dll.a version they installed. > > Maybe Cygwin should put all of it's libtool built libraries (or any > > others that are correctly versioned w.r.t the runtime loader) into a > > single directory that is near the front of the default path. /usr/lib > > seems like a good place. > > That's part of my 'easy' solution above. It may fix the cygwin problem, > but the requirement is not nice to native apps or other unix-on-win > emulations that the user may have. > > Relying on 'get dll's from same dir as .exe' works -- but only if EVERY > cygwin exe and EVERY cygwin dll are piled into the same dir. Can't we assume that the native dll's we rely on are outside cygwin's control and can be relied upon to be managed by the operating system (I'm thinking of stuff in the SYSTEM directory), and that non-cygwin libraries (such as Pauls pw dlls) will not be installed into the cygwin binary search path? In which case I think it is okay to install all cygwin dlls and import libs to /usr/lib, making sure that cygwin.bat (or any other cygwin startup methods) put /usr/lib first in $PATH. Does cygwin ld use -rpath yet? If so we could encode /usr/lib into each application as the first place to *always* look when searching for a dll. (I guess I am asking whether the Windows runtime loader will honor whatever rpath directory cygwin ld may encode into an application -- as happens with all of the Unix ports of libtool). > I *think* the combination of implib/dll versioning described above, plus > (sigh) unique prefixes for different platforms -- e.g. 'cygfoo.dll' -- > will solve most of the problems brought up w.r.t compiletime/linktime > versioning, and runtime loading. Without interfering too much with > native apps or other unix-emul packages. So what do you lose? > > -1) Can't link directly to a dll; you must use an import lib. (why? #1, > ld only hunts for libfoo.dll & foo.dll, not cygfoo.dll. #2, if you link > directly to dll's without using import libs, you lose the clean > separation between 'runtime links' and 'development links' -- see > http://www.debian.org/Lists-Archives/lsb-spec-9905/msg00011.html. I > know, there aren't really any 'runtime' symlinks, but at least you can > separate the major so-names...) The cygfoo point is well taken. I'm not sure I would accept that patch anymore... infact I think that it is not a necessary part of this proposal (either of our proposals infact) if we have some way to control the runtime dll search path (by always putting /usr/lib first in the PATH for example). > -2) It's a helluva lot of work to change everything on a platform (say, > cygwin) so that all dll's start with 'cyg' instead of 'lib' This may be a non issue if you agree with my last paragraph. > -3) It mostly solves a lot of the issues, but not completely. Not > ideally. Not perfectly. Hey. This is Windows we're talking about =)O| > -4) Still requires significant changes to libtool to handle dll's and > importlibs. Not that significant. And this is all stuff that is on libtool's TODO list anyway. It will get done eventually, just as soon as we can nail down a proposal that everyone (well most of us at least) likes. > What do you gain? > > +1) mostly solves the problem > +2) you can still link dynamically or statically, without the use of > libtool if you so choose That is important, and I am prepared to accept a patch (nay, write a patch in the fullness of time) to have libtool deal with import libs properly. > +3) you don't have to develop an ld.so 'smart' runtime loader > > Is +3) a bigger win than work involved in -2) ? I think so. Unless someone steps up to the plate and offers to implement an ld.so-a-like, I am prepared to work on having libtool do the right thing as far as possible. > Will Chris, DJ, Corinna, et. al. even go for -2) ? Guys? Stop me now if you violently disagree with libtool taking this direction =)O| > Developing an ld.so runtime loader requires far fewer changes to > libtool, and since you *can* link directly to dll's (*) it'll work 'just > like unix (tm). But developing ld.so will be a tough job. > > (*) DJ has pushed a feature into ld that generates a 'virtual' import > lib on-the-fly if you put a dll in the link line. Yup. I picked his brains while he was doing it, and wrote something similar for libtool. By default libtool always searches for a dll to link against and generates the implib on the fly if a suitable one is found. Cheers, Gary. -- ___ _ ___ __ _ mailto: gvv AT techie DOT com / __|__ _ _ ___ _| | / / | / /_ _ _ _ __ _| |_ __ _ ___ gary AT gnu DOT org | (_ / _` | '_|// / |/ /| |/ / _` | || / _` | ' \/ _` | _ \ \___\__,_|_|\_, /|___(_)___/\__,_|\_,_\__, |_||_\__,_|//_/ home page: /___/ /___/ gpg public key: http://www.oranda.demon.co.uk http://www.oranda.demon.co.uk/key.asc -- Want to unsubscribe from this list? Send a message to cygwin-unsubscribe AT sourceware DOT cygnus DOT com