X-Spam-Check-By: sourceware.org To: cygwin AT cygwin DOT com From: Cj2k5 Subject: Re: download cygwin packages in multiple session Date: Fri, 16 Dec 2005 19:09:51 +0000 (UTC) Lines: 430 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit User-Agent: Loom/3.14 (http://gmane.org/) X-IsSubscribed: yes Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner AT cygwin DOT com Mail-Followup-To: cygwin AT cygwin DOT com Delivered-To: mailing list cygwin AT cygwin DOT com Hi Igor and all. Why? The script is for people that do not have internet access. The reason because setup.exe cannot help in this situation, is that it must have access to a local miror of the packages if a user want to make a update to new packages. Also, to get these file, a person will have to go to a Public Internet Access. The situation is clear that the PIA manager will not allow that person to monopolise a station if there are more then one transport of package file to do or to leave temporary file on hardisk (not secure for file integrity). So in that case, 1. user can download a fresh setup.ini, 2. generate a list of missing packages at home, 3. go to a public internet access and download the files, 4. go back to home and update local miror, 5. repeat 3 and 4 if still missing files. What is it? This is a shell script that generate listing of package file to download and listing of package file to move or remove from local miror. The shell script depend on: - gawk, sort, uniq, cat commands - a latess setup.ini file - and optionaly an relatively old local package repository. The script do not manage cygwin instalation nor manage local packages miror, it only generate text files. These text files can be use to manage a local packages miror in a shell session, or use with wget as a list of files to download. The generated list are splited base on a total of packages bytes size limit given as argument. It default to 60000000 bytes. There are 5 type of listing that can be genrated: 1. listing for all the packages bin 2. listing for all the packages sources 3. partial listing from existing bin packages 4. partial listing from existing sources packages 5. listing for all the packages to [re]move Availables options: --help -h print this help and exit --version -v print version information and exit -ab listing for all the packages bin -as listing for all the packages sources -pb partial listing from existing bin packages -ps partial listing from existing sources packages -m create a [re]move list -i file new setup.ini file -s bytes file group size limit in byte (default 60000000) -r dir location of the local 'release' miror directory -t dir target directory you want the files generated Well. I submit this script as a GNU software and if someone want to modify it for a better usage or else, go for it. 1. I do not fully understand the arg opts mecanism, I copied codes from an othr shell script (GNU) and use it anyway. 2. I may be hard to reach, so if someone want to take care of this script, I allow, if nobody I will do what I can. Hope this help, Regards, Cj2k5 This message is composed offline, copied and paste to gmane interface at a public internet access. Sorry for file attachment, I did not found a way to make it work. file: dl-cygwin.sh ---------------------------------------------------------------- #!/bin/sh # # Copyright (C) 2005 Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2, or (at your option) # any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # # partial download from setup.ini # depend on sort, uniq and gawk progname=$0 package=cygwin-etc version=0.0.1 # for creating groups listing for all the packages bin grpdlallbin=no # for creating groups listing for all the packages sources grpdlallsrc=no # create a [re]move list movelist=no # for creating partial listing for binaris from already existing packages grpdlpartbin=no # for creating partial listing for sources from already existing packages grpdlpartsrc=no # get setup.ini file from the script argument inifile=setup.ini # file group size listing in byte grpsise="60000000" # location of the local 'release' miror directory releasedir=./release # target directory you want the files generated targetdir=./results # func_usage # outputs to stdout the --help usage message. func_usage () { echo "\ Usage: $progname [OPTION]... Create miscelanous list of files to help manage or update cygwin packages. Generated files are splited base on group of packages files size limit. Options: --help -h print this help and exit --version -v print version information and exit -ab listing for all the packages bin -as listing for all the packages sources -pb partial listing from existing bin packages -ps partial listing from existing sources packages -m create a [re]move list -i file new setup.ini file -s bytes file group size limit in byte (default 60000000) -r dir location of the local 'release' miror directory (default $releasedir) -t dir target directory you want the files generated (default $targetdir)" echo " Written by Jacques Champagne Report bugs to ." } # func_version # outputs to stdout the --version message. func_version () { echo "$progname (GNU $package) $version" echo "Copyright (C) 2002-2004 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE." echo "Written by" "Jacques Champagne" } # Command-line option processing. # Removes the OPTIONS from the arguments. Sets the variables: # I do not fully understand this arg opts mecanism, # I use it anyway. Cj2k5 { while test $# -gt 0; do case "$1" in -ab ) shift grpdlallbin=yes ;; -as ) shift grpdlallsrc=yes ;; -pb ) shift grpdlpartbin=yes ;; -ps ) shift grpdlpartsrc=yes ;; -m ) shift movelist=yes ;; -i ) shift inifile=$1 shift ;; -r ) shift releasedir=$1 shift ;; -s ) shift grpsise=$1 shift ;; -t ) shift targetdir=$1 shift ;; --help | --hel | --he | --h | -h ) func_usage ; exit 0 ;; --version | --versio | --versi | --vers | --ver | --ve | --v | -v ) func_version ; exit 0 ;; -- ) # Stop option prcessing shift; break ;; -* ) echo "$progname: unknown option $1" 1>&2 echo "Try '$progname --help' for more information." 1>&2 exit 1 ;; * ) break ;; esac done } # Command-line argument processing. # Analyzes the remaining arguments. { if test $# -gt 0; then func_usage 1>&2 exit 1 fi } # needed by gawk export grpsise grpdlallbin grpdlallsrc targetdir if ! test -d $releasedir ; then echo "$releasedir does not exist" exit 1 fi if ! test -d $targetdir ; then echo "$targetdir does not exist" exit 1 fi if ! test -f $inifile ; then echo "$inifile not found" exit 1 fi rm $targetdir/*txt # generate a global download list from setup.ini file # to create group list files for binary download # generate a file needed for [re]move listing generator gawk '/^install:/ {fn=$2;s=$3;m=$4;\ line= s " " fn " " m;\ t=sub("[^/]+$","",fn);\ if (fn!=lfn) {line= line " " fn "setup.hint " fn "md5.sum"; lfn=fn};\ print line;}' $inifile | gawk '\ BEGIN{\ ofileA= ENVIRON["targetdir"] "/_DL_new_all_bin-";\ ofileB= ".txt";\ ofile= ofileA "000" ofileB;\ result= ENVIRON["targetdir"] "/_00report_all.txt"};\ /^/ {fn=$2; setup=$4; md5sum=$5; t=$1; tt+=t; u+=1;\ if (tt>ENVIRON["grpsise"] && ENVIRON["grpdlallbin"]!="no") {\ fflush(ofile);\ print "file " ofile >> result;\ print "total files " u-1 >> result;\ print "total size " tt-t >> result;\ print "" >> result;\ tt=t; u=1; ttt+=1; ttf="" ttt;\ if (ttt<10) {ttf="00" ttt};\ if (ttt>9 && ttt<100) {ttf="0" ttt};\ ofile=ofileA ttf ofileB;}\ print fn; if (ENVIRON["grpdlallbin"]!="no") {print fn >> ofile};\ if (setup) {u+=1; print setup;\ if (ENVIRON["grpdlallbin"]!="no") {print setup >> ofile}}\ if (md5sum) {u+=1; print md5sum;\ if (ENVIRON["grpdlallbin"]!="no") {print md5sum >> ofile};}}\ END{if (ENVIRON["grpdlallbin"]!="no") {\ print "file " ofile >> result;\ print "total files " u >> result;\ print "total size " tt >> result;\ print "" >> result;\ print "number of bin listing group " ttt+1 >> result;\ print "" >> result}};' > $targetdir/_all_new.tmp # generate a global download list from setup.ini file # to create group list files for sources download # generate a file needed for [re]move listing generator gawk '/^source:/ {print $2 " " $3;}' $inifile \ | sort | uniq | gawk '\ BEGIN{\ ofileA= ENVIRON["targetdir"] "/_DL_new_all_src-";\ ofileB= ".txt";\ ofile= ofileA "000" ofileB;\ result= ENVIRON["targetdir"] "/_00report_all.txt"};\ /^/ {fn=$1; t=$2;\ if (fn!=lfn) {tt+=t; u+=1; print fn;\ if (tt>ENVIRON["grpsise"] && ENVIRON["grpdlallsrc"]!="no") {\ fflush(ofile);\ print "file " ofile >> result;\ print "total files " u-1 >> result;\ print "total size " tt-t >> result;\ print "" >> result;\ tt=t; u=1; ttt+=1; ttf="" ttt;\ if (ttt<10) {ttf="00" ttt};\ if (ttt>9 && ttt<100) {ttf="0" ttt};\ ofile=ofileA ttf ofileB};\ if (ENVIRON["grpdlallsrc"]!="no") {print fn >> ofile};\ lfn=fn}};\ END{if (ENVIRON["grpdlallsrc"]!="no") {\ print "file " ofile >> result;\ print "total files " u >> result;\ print "total size " tt >> result;\ print "" >> result;\ print "number of src listing group " ttt+1 >> result;\ print "" >> result}};' >> $targetdir/_all_new.tmp # generate a list of files to [re]move if test $movelist != no ; then find $releasedir -iname '*' -type f -print \ |sed -e s,^./,, > $targetdir/_all_old.txt cat $targetdir/_all_new.tmp |sort > $targetdir/_all_new.txt cat $targetdir/_all_old.txt |gawk '\ BEGIN {i=0;\ c = "cat " ENVIRON["targetdir"] "/_all_new.txt" ;\ while ((c | getline) > 0) {i++; f[i] = $1;}; close(c)};\ /^release/ {fd=0; s=$0; fn=$0; t=sub("[^/]+$","",fn);\ for(n in f){if (match(f[n],s)) {fd=-1}};\ if (!fd) {print s;\ if(fn!=lfn){print fn "setup.hint";\ print fn "md5.sum";\ lfn=fn}}}' |sort |uniq > $targetdir/_2mov.txt fi rm $targetdir/_all_new.tmp # generate a partial download list from existing local packages files # to create group list files for binary download if test $grpdlpartbin != no ; then gawk '/^install:/ {fn=$2;s=$3;m=$4;\ line= s " " fn " " m;\ t=sub("[^/]+$","",fn);\ if (fn!=lfn) {\ line= line " " fn "setup.hint " fn "md5.sum";\ lfn=fn}; print line;}' \ $inifile |gawk '\ BEGIN{\ ofileA= ENVIRON["targetdir"] "/_DL_new_bin-";\ ofileB= ".txt";\ ofile= ofileA "000" ofileB;\ result= ENVIRON["targetdir"] "/_00report.txt"};\ /^/ {md5=$3; fn=$2; setup=$4; md5sum=$5; c="md5sum " fn;\ while ((c|getline)>0) {r=$1; continue}; close(c);\ if (match(md5,r)){}else{\ t=$1; tt+=t; u+=1;\ if (tt > ENVIRON["grpsise"]) {\ fflush(ofile);\ print "file " ofile >> result;\ print "total files " u-1 >> result;\ print "total size " tt-t >> result;\ print "" >> result;\ tt=t; u=1; ttt+=1; ttf="" ttt;\ if (ttt<10) {ttf="00" ttt};\ if (ttt>9 && ttt<100) {ttf="0" ttt};\ ofile=ofileA ttf ofileB;};\ print fn >> ofile;\ if (setup) {u+=1; print setup >> ofile;}\ if (md5sum) {u+=1; print md5sum >> ofile;};};\ }; END{\ print "file " ofile >> result;\ print "total files " u >> result;\ print "total size " tt >> result;\ print "" >> result;\ print "number of bin listing group " ttt+1 >> result;\ print "" >> result};' fi # generate a partial download list from existing local packages files # to create group list file for sources download if test $grpdlpartsrc != no ; then gawk '/^source:/ {print $2 " " $3 " " $4;}' $inifile \ | sort | uniq | gawk '\ BEGIN{\ ofileA= ENVIRON["targetdir"] "/_DL_new_src-";\ ofileB= ".txt";\ ofile= ofileA "000" ofileB;\ result= ENVIRON["targetdir"] "/_00report.txt"};\ /^/ {fn=$1; t=$2; md5=$3; c="md5sum " fn;\ while ((c|getline)>0) {r=$1; continue;}; close(c);\ if (match(md5,r)){}else{\ if (fn!=lfn) {tt+=t; u+=1;\ if (tt>ENVIRON["grpsise"]) {\ fflush(ofile);\ print "file " ofile >> result;\ print "total files " u-1 >> result;\ print "total size " tt-t >> result;\ print "" >> result;\ tt=t; u=1; ttt+=1; ttf="" ttt;\ if (ttt<10) {ttf="00" ttt};\ if (ttt>9 && ttt<100) {ttf="0" ttt};\ ofile=ofileA ttf ofileB;};\ print fn >> ofile; lfn=fn}};\ }; END{\ print "file " ofile >> result;\ print "total files " u >> result;\ print "total size " tt >> result;\ print "" >> result;\ print "number of src listing group " ttt+1 >> result;\ print "" >> result};' fi ----------------------------------------------------------------- -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/