X-Spam-Check-By: sourceware.org To: cygwin AT cygwin DOT com From: Cj2k5 Subject: Re: download cygwin packages in multiple session Date: Tue, 20 Dec 2005 19:58:40 +0000 (UTC) Lines: 538 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit User-Agent: Loom/3.14 (http://gmane.org/) X-IsSubscribed: yes Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT cygwin DOT com Mail-Followup-To: cygwin AT cygwin DOT com Delivered-To: mailing list cygwin AT cygwin DOT com Igor Pechtchanski cs.nyu.edu> writes: > > On Thu, 15 Dec 2005, Cj2k5 wrote: > > > Hi all > > > > I have created a shell script because I dont have an internet connection > > and this script allow me to do a dowload session limited in n Mbytes > > using wget. I then fill a usb flash memory key and go home to update my > > local cygwin miror. Then repeat the session. > > > > The script compare a latest setup.ini with a local miror of cygwin > > packages and generate multiple text file that will be use by wget. > > > > I did not have my script when sending this message, but tomorrow I will > > look for reply to this message and if there are people interesting to > > it, I will post the script to a new message. > > This may not be apparent from that FAQ entry, but Michael Chase's > clean_setup.pl, referenced from [1], does that and more. It would be good > to know how your script differs from it... > Igor > [1] Hi Igor I submit a second version of the shell script. It correct some bugs and have new features. I did not yet look at the clean_setup.pl script, but I will today. If it's not the appropriate place to post this shell script, let me know. I will try to find an other place. Regards, Cj2k5 file: dl-cygwin.doc -------------------------------------------------------------------- Why? This script is for people that do not have internet access and have to download cygwin in multiple session. The reason because setup.exe cannot help in this situation, is that it must have access to a local miror of the packages if a user want to make an update to new packages. To get these files, a user will have to go at a Public Internet Access (internet cafe, library etc) and use a USB flash memory key to transport packages files. Some PIA manager would not allow a user to monopolise a station if there are more then one transport of package files to do, or they wont let them leave temporary file on hardisk (not secure). Using this shell script, 1. user can download a fresh setup.ini, 2. generate a list of missing packages, 3. download the files at a PIA using wget, 4. go to home and update local miror, 5. repeat 3 and 4 if still missing files. What is it? This is a shell script that generate listing of package files that are missing in a cygwin local release directory, base on a new setup.ini file. It can also do a md5 checksum on downloaded files. The shell script depend on: - sh, gawk, sort, uniq, cat and sed commands - a new setup.ini file - and optionaly an relatively old local package repository. The script do not manage cygwin instalation nor manage local packages miror, it only generate text files. These text files can be use to manage a local packages miror in a shell session, or use with wget as a list of files to download. The generated list are splited base on a total of packages bytes size limit given as argument. It default to 60000000 bytes. To generate only one file, use '-s 0' in the argument list. There are 7 type of listing that can be genrated: 1. listing for all the packages bin (_DL_new_all_bin-nnn.txt) 2. listing for all the packages sources (_DL_new_all_src-nnn.txt) 3. partial listing from existing bin packages (_DL_new_bin-nnn.txt) 4. partial listing from existing sources packages (_DL_new_src-nnn.txt) 5. listing for all the packages to [re]move (_mov.txt) 6. report on total files and total size per file list (_00report_all.txt, _00report.txt) 7. 3 listing of checksum on packages for: - files that dont move and pass the checksum (_md5_ok.txt), - files that should be move and cannot be check (_md5_mov.txt), - and files that fail the checksum (_md5_no.txt). Availables options: --help -h print this help and exit --version -v print version information and exit -ab listing for all the packages bin -as listing for all the packages sources -pb partial listing from existing bin packages -ps partial listing from existing sources packages -d debug (will not erase temp files) -m create a [re]move list (release dir against setup.ini) -c compare with md5 checksum (-r dir against setup.ini) -i file new setup.ini file -s bytes file group size limit in byte (default 60000000), ('-s 0' mean generate only one file) -r dir location of the local 'release' miror directory (default ./release) -t dir target directory you want the files generated (default ./results)" Todo - better doc and examples - generate new setup.ini files for use with setup.exe - more test - check if it's useseful for me (done) - check if it's realy useseful for the others Cj2k5 -------------------------------------------------------------------- file: dl-cygwin.sh -------------------------------------------------------------------- #!/bin/sh # # Copyright (C) 2005 Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2, or (at your option) # any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # # partial download from setup.ini # depend on sort, uniq and gawk progname=$0 package=cygwin-etc version=0.0.3 # for creating groups listing for all the packages bin grpdlallbin=no # for creating groups listing for all the packages sources grpdlallsrc=no # create a [re]move list movelist=no # create a md5 check list checklist=no # for creating partial listing for binaris from already existing packages grpdlpartbin=no # for creating partial listing for sources from already existing packages grpdlpartsrc=no # for debug do not erase temp files debug=no # get setup.ini file from the script argument inifile=setup.ini # file group size listing in byte grpsise="60000000" # location of the local 'release' miror directory releasedir=./release # target directory you want the files generated targetdir=./results # func_usage # outputs to stdout the --help usage message. func_usage () { echo "\ Usage: $progname [OPTION]... Create miscelanous list of files to help manage or update cygwin packages. Generated files are splited base on group of packages files size limit. Options: --help -h print this help and exit --version -v print version information and exit -ab listing for all the packages bin -as listing for all the packages sources -pb partial listing from existing bin packages -ps partial listing from existing sources packages -d debug will not erase temp files -m create a [re]move list (release dir against setup.ini) -c compare a md5check list (release dir against setup.ini) -i file new setup.ini file -s bytes file group size limit in byte (default 60000000), ('-s 0' mean generate only one file) -r dir location of the local 'release' miror directory (default $releasedir) -t dir target directory you want the files generated (default $targetdir)" echo " Written by Jacques Champagne Report bugs to ." } # func_version # outputs to stdout the --version message. func_version () { echo "$progname (GNU $package) $version" echo "Copyright (C) 2005 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE." echo "Written by" "Jacques Champagne" } # Command-line option processing. # Removes the OPTIONS from the arguments. Sets the variables: # I do not fully understand this arg opts mecanism, # I use it anyway. Cj2k5 { while test $# -gt 0; do case "$1" in -ab ) shift grpdlallbin=yes ;; -as ) shift grpdlallsrc=yes ;; -pb ) shift grpdlpartbin=yes ;; -ps ) shift grpdlpartsrc=yes ;; -d ) shift debug=yes ;; -c ) shift checklist=yes ;; -m ) shift movelist=yes ;; -i ) shift inifile=$1 shift ;; -r ) shift releasedir=$1 shift ;; -s ) shift grpsise=$1 shift ;; -t ) shift targetdir=$1 shift ;; --help | --hel | --he | --h | -h ) func_usage ; exit 0 ;; --version | --versio | --versi | --vers | --ver | --ve | --v | -v ) func_version ; exit 0 ;; -- ) # Stop option prcessing shift; break ;; -* ) echo "$progname: unknown option $1" 1>&2 echo "Try '$progname --help' for more information." 1>&2 exit 1 ;; * ) break ;; esac done } # Command-line argument processing. # Analyzes the remaining arguments. { if test $# -gt 0; then func_usage 1>&2 exit 1 fi } releasedir=`echo $releasedir|sed -e 's,/$,,'` # test if release dir exist echo "$releasedir" if ! test -d $releasedir ; then echo "$releasedir does not exist" exit 1 fi targetdir=`echo $targetdir|sed -e 's,/$,,'` # test if target dir exist echo "$targetdir" if ! test -d $targetdir ; then echo "$targetdir does not exist" exit 1 fi # test if setup.ini file exist if ! test -f $inifile ; then echo "$inifile not found" exit 1 fi # needed by gawk export grpsise targetdir releasedir # clean up old list files rm $targetdir/_*txt # generate a global download list from setup.ini file for binary download if test $grpdlallbin != no ; then gawk '/^install:/ {fn=$2;s=$3;m=$4;\ line= s " " fn " " m;\ t=sub("[^/]+$","",fn);\ if (fn!=lfn) {line= line " " fn "setup.hint " fn "md5.sum"; lfn=fn};\ print line;}' $inifile | gawk '\ BEGIN{\ ofileA= ENVIRON["targetdir"] "/_DL_new_all_bin-";\ ofileB= ".txt";\ ofile= ofileA "000" ofileB;\ result= ENVIRON["targetdir"] "/_00report_all.txt"};\ /^/ {fn=$2; setup=$4; md5sum=$5; t=$1; tt+=t; u+=1;\ if (tt>ENVIRON["grpsise"] && ENVIRON["grpsise"]+0!=0) {\ fflush(ofile);\ print "file " ofile >> result;\ print "total files " u-1 >> result;\ print "total size " tt-t >> result;\ print "" >> result;\ tt=t; u=1; ttt+=1; ttf="" ttt;\ if (ttt<10) {ttf="00" ttt};\ if (ttt>9 && ttt<100) {ttf="0" ttt};\ ofile=ofileA ttf ofileB;}\ print fn >> ofile;\ if (setup) {u+=1; print setup >> ofile}\ if (md5sum) {u+=1; print md5sum >> ofile}}\ END{\ print "file " ofile >> result;\ print "total files " u >> result;\ print "total size " tt >> result;\ print "" >> result;\ print "number of bin listing group " ttt+1 >> result;\ print "" >> result}' fi # generate a global download list from setup.ini file for sources download if test $grpdlallsrc != no ; then gawk '/^source:/ {print $2 " " $3;}' $inifile \ | sort | uniq | gawk '\ BEGIN{\ ofileA= ENVIRON["targetdir"] "/_DL_new_all_src-";\ ofileB= ".txt";\ ofile= ofileA "000" ofileB;\ result= ENVIRON["targetdir"] "/_00report_all.txt"};\ /^/ {fn=$1; t=$2;\ if (fn!=lfn) {tt+=t; u+=1;\ if (tt>ENVIRON["grpsise"] && ENVIRON["grpsise"]+0!=0) {\ fflush(ofile);\ print "file " ofile >> result;\ print "total files " u-1 >> result;\ print "total size " tt-t >> result;\ print "" >> result;\ tt=t; u=1; ttt+=1; ttf="" ttt;\ if (ttt<10) {ttf="00" ttt};\ if (ttt>9 && ttt<100) {ttf="0" ttt};\ ofile=ofileA ttf ofileB};\ print fn >> ofile;\ lfn=fn}};\ END{\ print "file " ofile >> result;\ print "total files " u >> result;\ print "total size " tt >> result;\ print "" >> result;\ print "number of src listing group " ttt+1 >> result;\ print "" >> result};' fi # generate a list of files to [re]move if test $movelist != no ; then find $releasedir -iname '*' -type f -print > $targetdir/_mov_all_old.txt gawk '/^install:/ {fn=$2;\ line= fn;\ t=sub("[^/]+$","",fn);\ if (fn!=lfn) {line= line "\n" fn "setup.hint\n" fn "md5.sum"; lfn=fn};\ print line;}' $inifile > $targetdir/_mov_all_new.tmp gawk '/^source:/ {print $2;}' $inifile >> $targetdir/_mov_all_new.tmp cat $targetdir/_mov_all_new.tmp |sort|uniq|\ sed -e 's,release/,'$releasedir'/,' > $targetdir/_mov_all_new.txt cat $targetdir/_mov_all_old.txt |gawk '\ BEGIN {i=0;\ c = "cat " ENVIRON["targetdir"] "/_mov_all_new.txt" ;\ while ((c | getline) > 0) {i++; f[i] = $0}; close(c)}\ /^/ {fd=0; s=$0; fn=$0; t=sub("[^/]+$","",fn);\ for (n in f) {if (f[n] == s) {fd=-1}};\ if (!fd) {print s;\ if(fn!=lfn) {print fn "setup.hint";\ print fn "md5.sum"; lfn=fn}}}' |sort|uniq > $targetdir/_mov.txt if test $debug = no ; then rm $targetdir/_mov_all_new.tmp $targetdir/_mov_all_new.txt $targetdir/_mov_all_old.txt fi fi # generate a list of md5 checksum of package files and compare with setup.ini md5 if test $checklist != no ; then gawk '/^install:/ {print $4 " " $2;}' $inifile > $targetdir/_md5_all_new.tmp gawk '/^source:/ {print $4 " " $2;}' $inifile >> $targetdir/_md5_all_new.tmp cat $targetdir/_md5_all_new.tmp |\ sed -e 's, release/, '$releasedir'/,' |sort > $targetdir/_md5_all_new.txt find $releasedir -iname '*bz2' -type f -exec md5sum '{}' ';' \ |sed -e 's, [*], ,' > $targetdir/_md5_all_old.txt cat $targetdir/_md5_all_old.txt |gawk '\ BEGIN {i=0;j=0;k=0;\ c = "cat " ENVIRON["targetdir"] "/_md5_all_new.txt" ;\ while ((c | getline) > 0) {i++; f[i] = $0;}; close(c)};\ /^/ {fd=0; m=$1; s=$2; t=$0; ok=""; no="";\ for (n in f) {if (index(f[n],s)) {fd=-1;\ if (index(f[n],m)) {ok=t} else {no=t} }};\ if (!fd) {print m " " s >> ENVIRON["targetdir"] "/_md5_mov.txt"};\ if (ok) {print ok >> ENVIRON["targetdir"] "/_md5_ok.txt"};\ if (no) {print no >> ENVIRON["targetdir"] "/_md5_no.txt"}}'\ > $targetdir/_md5_errors.txt 2>&1 if test $debug = no ; then rm $targetdir/_md5_all_new.tmp $targetdir/_md5_all_new.txt rm $targetdir/_md5_all_old.txt $targetdir/_md5_errors.txt fi fi # generate a partial download list from existing local packages files # to create group list files for binary download if test $grpdlpartbin != no ; then gawk '/^install:/ {fn=$2;s=$3;m=$4;\ line= s " " fn " " m;\ t=sub("[^/]+$","",fn);\ if (fn!=lfn) {\ line= line " " fn "setup.hint " fn "md5.sum";\ lfn=fn}; print line;}' \ $inifile |sed -e 's, release/, '$releasedir'/,' |gawk '\ BEGIN{\ ofileA= ENVIRON["targetdir"] "/_DL_new_bin-";\ ofileB= ".txt";\ ofile= ofileA "000" ofileB;\ result= ENVIRON["targetdir"] "/_00report.txt"};\ /^/ {md5=$3; fn=$2; setup=$4; md5sum=$5; c="md5sum " fn;\ while ((c|getline)>0) {r=$1; continue}; close(c);\ if (match(md5,r)){}else{\ t=$1; tt+=t; u+=1;\ if (tt > ENVIRON["grpsise"] && ENVIRON["grpsise"]+0!=0) {\ fflush(ofile);\ print "file " ofile >> result;\ print "total files " u-1 >> result;\ print "total size " tt-t >> result;\ print "" >> result;\ tt=t; u=1; ttt+=1; ttf="" ttt;\ if (ttt<10) {ttf="00" ttt};\ if (ttt>9 && ttt<100) {ttf="0" ttt};\ ofile=ofileA ttf ofileB;};\ t=sub(ENVIRON["releasedir"] "/","release/",fn);\ print fn >> ofile;\ if (setup) {u+=1; print setup >> ofile;}\ if (md5sum) {u+=1; print md5sum >> ofile;};};\ }; END{\ print "file " ofile >> result;\ print "total files " u >> result;\ print "total size " tt >> result;\ print "" >> result;\ print "number of bin listing group " ttt+1 >> result;\ print "" >> result};' > $targetdir/_DL_new_bin_errors.txt 2>&1 if test $debug = no ; then rm $targetdir/_DL_new_bin_errors.txt fi fi # generate a partial download list from existing local packages files # to create group list file for sources download if test $grpdlpartsrc != no ; then gawk '/^source:/ {print $2 " " $3 " " $4;}' $inifile \ | sort | uniq | sed -e 's,^release/, '$releasedir'/,' | gawk '\ BEGIN{\ ofileA= ENVIRON["targetdir"] "/_DL_new_src-";\ ofileB= ".txt";\ ofile= ofileA "000" ofileB;\ result= ENVIRON["targetdir"] "/_00report.txt"};\ /^/ {fn=$1; t=$2; md5=$3; c="md5sum " fn;\ while ((c|getline)>0) {r=$1; continue;}; close(c);\ if (match(md5,r)){}else{tt+=t; u+=1;\ if (tt>ENVIRON["grpsise"] && ENVIRON["grpsise"]+0!=0) {\ fflush(ofile);\ print "file " ofile >> result;\ print "total files " u-1 >> result;\ print "total size " tt-t >> result;\ print "" >> result;\ tt=t; u=1; ttt+=1; ttf="" ttt;\ if (ttt<10) {ttf="00" ttt};\ if (ttt>9 && ttt<100) {ttf="0" ttt};\ ofile=ofileA ttf ofileB;};\ t=sub(ENVIRON["releasedir"] "/","release/",fn);\ print fn >> ofile; };\ }; END{\ print "file " ofile >> result;\ print "total files " u >> result;\ print "total size " tt >> result;\ print "" >> result;\ print "number of src listing group " ttt+1 >> result;\ print "" >> result};' > $targetdir/_DL_new_src_errors.txt 2>&1 if test $debug = no ; then rm $targetdir/_DL_new_src_errors.txt fi fi ---------------------------------------------------------------------- -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/