Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Date: Fri, 1 Oct 2004 15:51:09 +1000 (EST)
From: luke DOT kendall AT cisra DOT canon DOT com DOT au
Subject: Mirror site goodness
To: cygwin AT cygwin DOT com
MIME-Version: 1.0
Content-Type: TEXT/plain; CHARSET=US-ASCII
Message-Id: <20041001055109.3BF0884C99@pessard.research.canon.com.au>

I've made some improvements to my md5cygchk script, to solve the
problem of knowing when to trust our local mirrors.

Basically, we have a "stable" mirror - tried and tested, and known to
be complete and good.  And we have a "latest" mirror, which is updated
nightly via rsync from a single mirror site.

Nightly, after the rsync, we run "md5cygchk --report", and that writes a
report file at the top of the Cygwin mirror, and removes any old report
 files. So anyone can look at the mirror, and if they see the
"cygwin-archive-incomplete.txt", they know it'd be unwise to run setup
pointing there.  OTOH, if the file cygwin-archive-okay.txt, they know
the mirror is complete.

(cygwin-archive-incomplete.txt also lists the packages with problems, as
reported by md5.)

So now our local scripts (some of them dumb old .bat scripts), can tell
at a glance whether or not to trust the mirror.

It occurred to me that if the mirror sites chose to run the script,
with the --report option so that it wrote/deleted the appropriate
report file(s), then everyone around the world would be able to look at
a Cygwin mirror site and know whether or not it was complete.

I'm happy to pass on the updated script (even though I still haven't
gotten around to incorporating Fergus's 1st two good suggestions yet:

>  There are 3 parts to what the script does: 
>   
>  1. checks the timestamp on your local setup.ini with the one currently on 
>  the mirror; 
>  2. checks that the files you've got match the ones you should have (this 
>  just checks the names); 
>  3. checks the md5sum's. 

But, how much contact do you have with the people running the mirrors?
If you made it available to them, would they use it?

Anyway, I'm sharing the updated script (appended below), in case it's of
use to others.  Perhaps it should really be called cygmd5chk

BTW, the reason for creating two files is because I believe cmd.exe is
so dumb it has no way to distinguish between an empty file and one
that's not empty.

luke
------------------------- md5cygchk ------------------------------
#!/bin/sh
#
# Check all the md5 signatures for a Cygwin mirror.
#
# Author: Luke Kendall
#
CYG_BAD=cygwin-archive-incomplete.txt
CYG_GOOD=cygwin-archive-okay.txt
ECHO=echo
MD5ARG=
MIRROR=/u/mirror/cygwin/release
MYNAME=`basename "$0"`
RECORD=false
TMP=/tmp/md5cyg$$
USAGE="Usage: $MYNAME [-q] [--record] cygwin-mirror-directory
Where:
    -q   	means quiet.
    --record	means record a success or failure indicator in the
		the directory above the mirror directory, by making
		$CYG_GOOD & $CYG_BAD files
		appear or disappear (so even DOS batch scripts can test
		the archive's goodness).
		You have to have write permission for this to work.
    cygwin-mirror-directory
		this should be the directory where you find setup.bz2,
		setup.exe, and setup.ini.
		Success/failure files are created in the directory
		above this, if the --record option is used.

    E.g.:

	$MYNAME --record -q /u/mirror/cygwin/release

    NOTE: running md5 over 2GB of files is best done on the host
    machine, to avoid hammering our network.
    (Use 'df -k \$MIRROR' to discover what machine
    hosts \$MIRROR, that you should slogin to,
    to run this $MYNAME.)"

trap 'rm -f $TMP; exit 1'

usage()
{
    # sed, so that if they specified a mirror, it's used in the usage message.
    echo "$USAGE" | sed "s|\$MIRROR|$MIRROR|" >&2
    exit 1
}

while [ $# -ge 1 ]
do
    case $1 in
	-q)
	    MD5ARG="--status"
	    ECHO=":"
	    ;;
	--record)
	    RECORD=true
	    ;;
	-*)
	    usage
	    ;;
	*)
	    MIRROR="$1"
	    shift
	    break
	    ;;
    esac
    shift
done
if [ $# != 0 ]
then
    usage
fi
cd $MIRROR || exit 1
if [ ! -s ../setup.bz2 ]
then
    echo "$MYNAME: Not a cygwin download (no file setup.bz2 in $MIRROR parent)" >&2
    exit 1
fi
WHERE=`pwd`

find . -type d -print | \
    while read dir
    do
    (
	cd "$dir"
	if [ -s md5.sum ]
	then
	    $ECHO "$dir:"
	    if md5sum $MD5ARG --check md5.sum
	    then
		:
	    else
		{
		    echo
		    echo "In $WHERE/$dir:"
		    md5sum --check md5.sum 2>&1 | grep -v OK
		} >> $TMP
	    fi
	elif [ `ls -l | grep "^-" | wc -l` = 0 ]
	then
	    :
	else
	    echo "Worrying: $dir has no md5.sum file" >&2
	    echo "Worrying: $dir has no md5.sum file" >> $TMP
	fi
	cd $WHERE # Really, the shell popping via (...) does this.
    )
    done

if $RECORD
then
    if [ ! -w $WHERE/.. ]
    then
	echo "$MYNAME: you don't have write permission on $WHERE/.." >&2
	RECORD=false
    else
	# Remove them both, temporarily, in case the previous creator didn't
	# give you file write permissions.
	rm -f $WHERE/../$CYG_BAD $WHERE/../$CYG_GOOD
    fi
fi

if [ -s $TMP ]
then
    #
    # The check is bad.
    #
    if $RECORD
    then
	{
	    echo "Summary of files with broken checksums:"
	    cat $TMP
	} > $WHERE/../$CYG_BAD
	chmod 664 $WHERE/../$CYG_BAD
    fi
    echo "Summary of files with broken checksums:"
    cat $TMP
    rm $TMP
    [ -s $WHERE/../$CYG_GOOD ]  && echo "$MYNAME: impossible: good and bad!" >&2
    exit 1
else
    #
    # The check is good.
    #
    if $RECORD
    then
	echo "--- All files checked okay. ---" > $WHERE/../$CYG_GOOD
	chmod 664 $WHERE/../$CYG_GOOD
    fi
    $ECHO "--- All files checked okay. ---"
    [ -s $WHERE/../$CYG_BAD ]  && echo "$MYNAME: impossible: bad and good!" >&2
    exit 0
fi
trap '' 0


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/