Mailing-List: contact cygwin-apps-help AT sourceware DOT cygnus DOT com; run by ezmlm Sender: cygwin-apps-owner AT sourceware DOT cygnus DOT com List-Subscribe: List-Archive: List-Post: List-Help: , Delivered-To: mailing list cygwin-apps AT sources DOT redhat DOT com Message-ID: <3BE702B5.7090706@ece.gatech.edu> Date: Mon, 05 Nov 2001 16:20:53 -0500 From: Charles Wilson User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1 X-Accept-Language: en-us MIME-Version: 1.0 To: Robert Collins CC: cygwin-apps AT cygwin DOT com Subject: Re: patches to vendor source trees - discussion References: <3BE4D4A7 DOT 2070900 AT ece DOT gatech DOT edu> <1004867254 DOT 5388 DOT 51 DOT camel AT lifelesswks> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit [read my reply in Corinna's subthread as well as this one before replying] Robert Collins wrote: >>*PERHAPS* -- if it doesn't already -- setup could record the fact in >>/etc/setup/foo-src.list.gz that "foo-pristine-ver.tar.gz" was put into >>/home/rcollins/src/ORIG/foo/ (or /usr/src/ORIG/foo). Likewise the other >>primary files in the foo-ver-rel-src.tar.bz2 archive. >> But this really isn't a big deal. >> > > As I mentioned, I don't think it should record source stuff at all - > with the specific exception of packages whose binary package *is* a > source package (a specific example of which is the kernel-source > packages on linux systems). Okay. >>>I'd love too be able to do that - with with the rpm or deb inspiration, >>>but frankly setup ain't ready for that. >>> >>Ready for what? All I'm saying is "maintainers, make your src package look >>like this". All setup has to do is unpack it. Under /usr/src (or ~/src if >>you like) >> > > rpm/deb have a bunch of support infrastructure for package maintainers - > if setup could install rpms or debs then we could use those tools. Later. >>Where does stuff ACTUALLY get built? in /cygwin1.3/* ? Ick! I'd imagine >>your '/' directory (wherever it happens to be) gets awful cluttered with >>oodles of foo2.4* bar4.6* and baz1.3* files and directories... >> > > Most things build in the source tree, but the package build can be done > in a .build subdir or equivalent trivially. True. And I guess rpm does the same thing. Unpacked source goes in BUILD, and it is *built* there as well (mebbe in a .build subdir, I forget). >>>Nope. IMO it shouldn't. The cygbuild script or whatever we call it, that >>>can call setup to grab the source, >>> >>Okay, you're assuming setup has the argv handling stuff already... >> > > Well, it does - it's got getopt now, just no options :]. More seriously, > it doesn't have the flexability to do this regardless of cmd line > parameters yet (because of the hard coded /usr/src) K. >>Yes! But I wasn't going that far. Cygbuild is in the future (say, version >>3 in my scheme above). Version 1 -- packages have their own specific build >>script. Later, that build script is converted into a .spec-like file or >>rules-like file for cygbuild to parse. Or something. >> > > FYI debian 'rules' is a make file. The first line reads > #! /bin/make -f > with the obvious consequence. Oohhh...that's interesting. Neat. >>Anyway, "my" proposed -src package only contains: >> >>/cygwin/ORIG/zlib/zlib-113.tgz >>/cygwin/ORIG/zlib/zlib-1.1.3-9.patch >>/cygwin/ORIG/zlib/zlib-1.1.3-9.postinstall >>/cygwin/ORIG/zlib/* >>/cygwin/DOCS/zlib/zlib-1.1.3.README >> >>/cygwin/SCRIPTS/zlib-build >> > > Right, what I object to here is the pathing: > cygwin/orig|docs+|scripts/zlib > > to me that means I cannot separate out building separate packages unless > I descend into the cygwin/ORIG directory. Also the zlib dir for ORIG and > DOCS is redundant, as all the files are package-version and > package-version-foo in naming. That's a minor point. Since we can't rely on setup.exe to unintall for us, what's easier? 'rm -rf zlib' or 'rm -rf zlib*' Not much difference, I suppose. It boils down to personal style. I like deep paths with lots of structure; others like shallow paths with 'everything at their fingertips'. I'm not a zealot about it, tho. >>setup-uninstall should just remove those files. Not the BUILD/zlib or INST/zlib or other "created" files >> > > Again, for source packages, IMO setup should not do anything uninstall wise. Thats the users call. Okay. See above. [single patch vs. multiple patches] > I don't see how I'm making it hard in my suggestion... I'm suggesting > that the source tarball be *the vendors tarball* (given we offer > pristine source). > > And that we have a *single* additional file created via. > diff -Nrup origsrcdir package-vendor > package-vendor-suffix.patch > > In my original suggestion I was suggesting that the patch be added to > the vendor tarball, but that suggestion also included pre-patched > source. If the source isn't pre-patched having the patch separate is > easier for the maintainer, I think. True. >>Your idea makes it difficult for *me* to keep track of all the various >>patch files -- or to explain to a new maintainer. I'd have to maintain a >>*separate* build script that handled the *separated* patches, but creates a >>src tarball with a unified patch and a build script that uses it. (I'm >>thinking along the lines of "how do I ensure that someone else can take >>over maintaining my 'readline' or 'libtiff' package if I get hit by a bus". >> With the "all-patches-in-one" method, it would be practically impossible >>for a NEW maintainer to, say, remove the lossless jpeg stuff from the jpeg >>package, but keep the dllization patches.) >> > > *sigh*. But I'm not suggesting that you elimintate your separate > patches. > > I'm suggesting that the user have a consolidated patch for them to use, > and that your working patches can be contained therin should you need > the, - preapplied or not at your choice. I'm considering anybody who'd actually download the src to a package and build it themselves as a developer, not a user. Further, as a potential new maintainer for that package. I suppose that's taking things a bit far -- raising the participation bar for 'ordinary' developers in order to make things easier for the corner case (taking over maintainership) is kinda silly. Sorry. >>I'd like for anybody to be able to take over maintainance of ANY package if >>the original maintainer goes kaflooey, using ONLY the stuff in the -src >>package. >> > > Thats a worthy goal. I'd hate to make it *hard* to do this. It's a corner case. It's rare that we NEED to have someone takeover maintaining a package. It's even more rare when the current port includes more than just "cygwinization" patches -- I can think of only two examples right now (both mine). Forget it. >>>-ONLY if your autotools version is different- (or for old autoconfs :[). >>> >> >>automake 1.3 still sees use, and there are six (count 'em, six) subversions >>of 1.4 that have different Makefile.in signatures: 1.4, 1.4p1,... 1.4p5. >>And don't forget 1.5. >> > > Yes but!. Automakte 1.4p5 should run automake 1.3 stuff just fine, and > 1.5 is the fly in the ointment. Oh, sure, it'll WORK. I'm talking about generating changes to many files unnecessarily, and uselessly increasing the .diff size(*). Using 1.4p5 will change the signature of all Makefile.in's in the tree, if the original ones were created using 1.3. (*) Actually, my concern here is pushing changes back to the upstream maintainers. Most maintainers don't want "the rabble" updating their auto* files to new autotool versions, and will reject patches that do so (even if the patch contains other stuff). OTOH, probably the only reason we'd mess with .am files is to enable installation of cygwin-specific documentation files -- which shouldn't go upstream anyway. So perhaps this is just a red herring. >>>then the patch only alters Makefile.am and configure.in. 'nuff said. >>> >>You are assuming that people who want to build from (our) -src packages >>MUST run autoreconf and/or automake themselves, before configuring. That's >>contrary to the autotools philosophy: distributed packages should be >>buildable even if the builder does NOT have autotools installed. (That's >>why libtoolize puts the libtool scripts into the target package itself, >>instead of assuming that libtool itself will be installed on the builder's >>machine) >> > > No, I'm assuming that for *your* packages, you wanted to trade patch > size for complexity. Personally, I'd patch the whole kit and kaboodle, > and trust gz to make it managable size wise. But that contradicts your earlier statement "the patch only alters Makefile.am and configure.in". Either the patch is small -- and the user must run auto*, or the patch is large and touches Makefile.in's and configure as well as Makefile.am and configure.in. But this is no longer a big deal to me. See red herring above. >>>yees, I see what you mean. However you have an environment where >>>building the package as it gets installed requires a fixed diretory tree >>>- information not in the src dir. >>> >>Okay, but so does debian, right? >> > > No, debian package building is self contained. There is a new directory > called debian sitting under the top src dir. In that there is (letsee, > automake 1.5 is a good example, > changelog > control > copyright > doc-base > lintian.overrides > postinst > prerm > rules > > But thats debian, and I intended to simply talk about what setup > extracts - what the user expects to see, and we are now talking about > how the packages are created by the user. But the two are linked...sorta. besides, you left out the upper dir stuff: automake-1.5-X.dsc automake-1.5-X.tar.gz automake-1.5-X.diff.gz automake-1.5/ debian/ your list of stuff >>>You're also putting the src dir at a fixed level in the tree relative to >>>the package data - which might play havoc with nested source trees like >>>winsup+newlib. >>> >>Err...only if those are ever downloaded and built independently of each >>other. I'm assuming they will not be. >> > > Could be risky assumption :]. Setup.exe will build quite happily out of > the tree AFAIK. w32api certainly will.... Dunno. Never tried. >>>If we make the extra work easy, then is it HARD? Seriously, I'll give >>>you a ssh account here, and you can have a play with the debian package >>>building environment. Nothing is easier - for the maintainer - IMO. >>> >> >>Well, it's already pretty easy -- for an autotool'ed project -- if you >>understand .am files and .m4 files. But ordinary porters should NOT have >>to learn the intricacies of m4 scripting just to insure that package foo >>installs its cygwin-specific README file. Hell, they shouldn't have to >> > > Cygwin packagers have to learn (this is from experience - watching you, > doing squid..) > * text vs binmode coding. (Can anyone show automode doing the job > completely, ever?) > * .dll building for packages with libraries. (libtool isn't complete > yet). > * Makefile twitching. (make foo.exe fails - oh, it only has a rule for > make foo). > > And thats a minimum. Frankly, adding a automake target for installing a > README is trivial. Sure -- but it annoyed the hell out of me. You can't just add it to the DOCFILES variable (or whatever) -- because then it gets installed in the normal docdir -- usually /usr/doc//). We want it to go into /usr/doc/Cygwin/. Well, that's harder, you have to add a special rule for install-data-local: (I think). Note to self: new documentation for packagers ought to include this as a hint...I had no hints and it took forever to figure this out... >>this is madness (to borrow a phrase). >> > > Perhaps :}. I'm really not intending to push the barrier up, but I do > think that README.Cygwin is really just the first little bit to be a > maintainer. well if the rules file (or spec file or .sh file or whatever) has the explicit rule to install into /usr/doc/Cygwin/.README that's one thing. To have the /Makefile do it is slightly harder. >>>So how many build scripts do we need? And how do they determine which >>>patches to apply? >>> >>At *first* (step 1, but not a baby step ), one build script for each >>package. So, zlib-build. libpng-build. automake-build. etc. (versioning >>aside -- see below) Later, those get converted into, what?, zlib.cyg, >>libpng.cyg, and automake.cyg. cygbuild parses them, etc. Just like >>dpkg/rules or rpm/spec. >> > > See my comment on 'rules' before, but yes. I have no issue with this. (I > *still* maintain it's orthogonal to having a global patch or not though > :}) correct. >>UUGGHH! You want to muck with the Makefile.am, Makefile.in, Makefile's to >>support building *our* split-up packages directly via make? I *really* >> > > No! I don't want that at all. I want to let *you* choose, just as you do > now. I didn't want to touch the package creation stuff *at all* other > than specify that a global patch be provided. > > As for building via make, I think that when we start to address > automated package creation, thats one of the things to address. I happen > to like the debian way, but please note: It doesn't require many > autotool file fiddling, because it uses a separate makefile, that calls > the pacakge makefile, and then does stuff on top of it - just like a > script, but more flexible for the user. Yeah, that's kinda cool. >>tar cvjf ../cgywin-new.tar.bz2 usr --exclude='usr/bin/mingwm10.dll' \ > Sure. So to do this as a makefile: > but *not* in the packages makefile. *That's* the part I was missing. The debian 'rules' file is like my .sh file in that it does MORE stuff on top of the package's Makefile. >>>* What about complex packages? >>>Well, thats the beauty, by including the scripts within the source tree >>>they can't get lost. So it becomes orthogonal to the 'standard'. >>> >> >>Ah, but I'm assuming that *THIS* is the source tree: >> >>cygwin/ORIG/foo.tgz >>cygwin/ORIG/foo.patch1 >>cygwin/ORIG/foo.patch2 >>cygwin/SCRIPTS/foo-build >>cygwin/DOCS/foo.README >> >>You're assuming that the "unpacked" foo/ directory is the source tree. But >>in either case, the patches won't get lost. >> > > But! the user cannot grab the foo tree and mv it elsewhere in one step. > Not without grabbing *all* the source trees. URGGH. That's a point. However, as with SRPMS, I assume that the source package contains all the necessary stuff, in the appropriate tree structure, all in one easy-to-unpack archive. You want to 'move' to a different build tree? Unpack the src archive somewhere else. (Granted, if setup doesn't keep track of files installed from -src packages, then deleting the oringal -src install is tricky). >>>2) We'll end up with ??how many ?? different scripts in that directory >>>tree? (remember, as each -suffix will have a different script (think on >>>this if you need to) installing the source to three revisions of package >>>foo ends up with three foo-build's, three foo-README's etc. >>> >>Erm...and what's wrong with that? debian has how many dsc files for >>different versions of the same package? I don't see the problem. >> > > One for each, and because they are in the same directory as the src tree > is rooted at, there is no need to fossack around finding all the related > files. Right. Using '%' as a shortcut for "the dir that the process was started from - not necessarily /usr/src" (You were using '/' to represent that. Too confusing). You'll have: %/cygwin1.3-2.diff.gz %/cygwin1.3-2.dsc %/cygwin1.3.tar.gz %/cygwin1.3 %/cygwin1.3-2.deb %/cygwin1.3-2-doc.deb %/cygwin1.3-2-src.deb As well as %/cygwin1.3-3.diff.gz %/cygwin1.3-3.dsc %/cygwin1.3-3.deb %/cygwin1.3-3-doc.deb %/cygwin1.3-3-src.deb and %/bash2.05-2.diff.gz %/bash2.05-2.dsc %/bash2.05.tar.gz %/bash2.05 %/bash2.05-2.deb %/bash2.05-2-doc.deb %/bash2.05-2-src.deb and tons of other packages, presumably all in the same (default) '%' dir. You're using ls's penchant for alphabetizing files to help orgainize your % dir. I'm using the dir structure under '%' to organize mine. >>As I see it, the debian method eliminates my: >> >>cgywin/PACKAGES/ >>cgywin/ORIG/ >>cgywin/SRC/ >> >>directories and just uses the top level (call it cgywin/) for the files >>that would go into them. It also collapses the cygwin/SRC up to the >>cygwin/ level, and unpacks the tarball there. >> >>the debian scheme doesn't seem to have any provision for building *outside* >>the unpacked source directory (unless it uses some dynmically created dir >>in /var/tmp ?). ditto the temporary installation directory. So that's >> >> > >>cygwin/BUILD >>cygwin/INST >> >>finally, the >> >>cygwin/SCRIPTS/ >>cygwin/DOCS/ >> >>are kinda replaced by the cygwin//debian/ directory (created by the >>patch) coupled with the .dsc file at the cgywin/ level. >> > > whats the "cygwin" dir? think '%' == /usr/src/cygwin/ (or /usr/src/. What the hell). > >>do I have that right? If so, then the primary difference between the >>proposals is >> >>a) splitting things up into multiple toplevel directories vs. everything >>goes in one toplevel directory (not counting the unpacked source archive) >> > > Uhmm possibly, if I uderstand you right. You seem to be using 1 top > level dir, and ~3 1st level dirs. I'm suggsting 1 top level dir per > package. Plus 5 or 6 files in the toplevel, parallel with that 1 top level dir. I want to squirrel those files down a level or two. It's the whole shallow-but-wide vs. deep-but-narrow distinction. >>b) script handles applying (possibly) multiple patches; some support files >>NOT created by a patch but are included directly in the primary -src >>archive vs. single patch, which creates SOME support files within the >>patchedNunpacked src tree, one other support file (.dsc) is NOT created by >>that unified patch. >> >>I gotta say, this (b) is one area where I think RPM-style wins out over >>DEB...ever look at a linux kernel-src RPM? *Much* more informative than if >>everything were all in a single patch. IMO. >> > > kernel source is a different beast. Ever look at debian for this? All > the kernel patches *are* separate. They come as separate .debs and are > automatically applied by make-kpkg. So kernel packages have to be handled specially by a different tool than dpkg? That sounds as if the debian scheme is not flexible enough for the general case, requiring an additional tool for complicated packages. >>However, in both cases the ultimate goal is to transition to a "process >>description file" that is interpreted by a custom tool (cygbuild? debbuild? >>etc) >> > > Uhmm yes. > > > Thining about this: > > Here are my revised immediate goals (forget everything I wrote above): > 1) ship a single -src tarball that contains everything needed to build a > package. Yep. > 2) Ship unaltered vendor src in that tarball for maintenance ease. Yep. > 3) Not impose any new rules on the package maintainers. > > To this end I proposed that we include a single unified patch, and the > src tree. Err...that's a new rule. I'll have to redo (slowly, as they require updating in the normal course of events) all of my source packages and patching schemes -- *ESPECIALLY* the icky dll-ized ones like readline, gettext, and ncurses. Except...I *cannot* do it your way, with current tools. I will have to delay ALL updates of the icky dll-ized packages until auto* and libtool support autoimport properly (or your profiling scheme, whatever). Unless I "get a waiver" and keep doing my packages the old way -- which leads us back to not having any standard at all, (see item (3) from my first email in this thread, the item that "scared you"). > The reason I don't want to alter the rules for package maintainers is > that I agree with the spirit of all you've said, and I percieve the > creation of these scripts, and the directory structure, and teh tools to > be a big task. We have a working status quo. The various patches go into > a CYGWIN_PATCHES or wahtever dir in the source tree, and the README > tells how to create teh package. That works. Lets *leave it in place*. But I can't. The build procedure for some of my packages is really ugly, and requires patching after building the stat libs, and after re-running configure, but before running make the 2nd time... Now that I think about it, a Makefile-like rules file won't allow me to get away with this either, because the patch-in-the-middle will screw up the dependency checking that make performs... Now perhaps you might argue (and I might even agree with you) that ports whose build proceedure for cygiwn is that ugly don't belong in the cygwin distribution. Mebbe so -- but there are a few that are THERE *now*. And we can't just throw them out, because other apps depend on them (ncurses, gettext). > Why? because any change now that is not a direct step towards deb/rpm is > simply bit twiddling, IMO. It won't make your life easier, and it will > raise the bar for maintainers. Well, yes and no. IMO, moving directly to one-patch-and-one-patch-only is bit twiddling -- because rpm supports multiple patches (counterargument: rpm does NOT support half-building a package, and THEN applying a patch before completing the build. So *I've* still got problems :-) One-patchism is a prerequisite for deb packages, and the "throw everything in '%'" is debian-like, but let's be honest: Given the political realities (Cygnus == Red Hat, RPM == Red Hat Package Manager), if we go to any autobuild system it will be rpm, first. deb may come later, but... Granted, there are mucho problems with rpm. It needs db, which hasn't *really* been ported yet. There is no *native* port of rpm (nor dpkg) --which complicates setup.exe (See Dario's attempt at an rpm-based installer). The whole replace DLL/EXE while in use problem. etc. Somebody famous (I forget who. RMS?) wrote a long critique expounding the shortcomings of rpm as a package management system. Yeah, it's got problems. But cygwin is a Red Hat product. (Anyway, many of the aforementioned problems are shared by dpkg) > When we are able to install an rpm or a deb, *then* I believe we should > start the change process. And then the steps will be pretty clear. Truthfully, I don't think either of us is advocating a process change right now. *I* want to be able to keep doing things my way (perhaps with some directory structure adjustments). My way involves shell scripts to control the build process, uses separate src, build, and inst directories for each package, etc. *You* want to enable doing things the debian way (perhaps with some adjustments). Both are fair arguments, and we should be able to get there. > So there it is, I wasn't actually trying to change anything signficiant, > other than making the ability to go (to originally, now from ) pristine > source to cygwin ready source a standard step. This missing piece of the puzzle: some (of my) packages aren't standardized, and will require more than any conceivable standard would allow, until other tools are improved. > The rest of the process > is just fine until the infrastructure to do more is in place iMO. See next message.... --chuck