X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:message-id:date:from:mime-version:to:cc
	:subject:references:in-reply-to:content-type
	:content-transfer-encoding; q=dns; s=default; b=bSJ+P/U6V6YT2ZFw
	NQw8J2u6j8uHM4mm2/0LSJ70OtrrvkTyQqJcwCkvAMIOzFhcGysUB4RzsuSadGWx
	Ftqdn5Rad0/BRblNNOBUlYzLCz6zdd3eLu17kjoNasou9s6onXo307hYdaeW04Sz
	cDQTfB0WXx0S1+ALxYZW2GXENmQ=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:message-id:date:from:mime-version:to:cc
	:subject:references:in-reply-to:content-type
	:content-transfer-encoding; s=default; bh=LU8Gtz3DVTNyCm9xeOOhRV
	KIVI0=; b=nXZ7+v1uwFB7WRZooWazZiAG2gquWtFKR57kqAlO11aBfq40UgzRvf
	Jy52tybMMWFOc+c/U/+DG3b8nmukXT3q7USyYn0hsCViJL2s0eNHKcpalgNre260
	1aEpUqypN8tUn6zeyMMUwrgdzQ+Fx7i2mld/RecmBTDIco1ct0RyI=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Virus-Found:	No
X-Spam-SWARE-Status:	No, score=-1.5 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.3.2
X-HELO:	EXE01-WPP.cisra.canon.com.au
Message-ID:	<53BB6183.9000007@cisra.canon.com.au>
Date:	Tue, 8 Jul 2014 13:12:03 +1000
From:	Luke Kendall <luke DOT kendall AT cisra DOT canon DOT com DOT au>
User-Agent:	Mozilla/5.0 (X11; Linux i686 on x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version:	1.0
To:	<cygwin AT cygwin DOT com>
CC:	audit <audit-mail-disclaimer AT cisra DOT canon DOT com DOT au>
Subject:	Re: Observations about Cygwin's md5 checksums
References:	<53BA242D DOT 1040101 AT cisra DOT canon DOT com DOT au> <53BA70CC DOT 7010809 AT gmail DOT com>
In-Reply-To:	<53BA70CC.7010809@gmail.com>
X-IsSubscribed:	yes

On 07/07/14 20:05, Marco Atzeri wrote:
> On 07/07/2014 06:38, Luke Kendall wrote:
>> Here are five observations about md5 checksums in Cygwin.  I share it in
>> case it may be of some small interest to a few people.  Please note that
>> I may be wrong; if so, I'm happy to be corrected.
>>
>> 1) For each package, Cygwin stores the md5sum for the components of the
>> main parts of the package in the setup.ini file.  The exception is the
>> setup.hint file: its md5 sum is not recorded in setup.ini.
>
> setup.ini is built using the setup.hint's of the several packages.
> No further usage outside the www.cygwin.com server.

Thanks for the explanation, Marco.  Can I check what you mean?
I think you're saying that setup.hint has no function once setup.ini has 
been created.
(You're not saying "setup.ini shouldn't be used for other purposes.")

>> 2) In each zip file for each package, an md5.sum file is almost always
>> provided.  But not always. (*)
>>
>> 3) These md5.sum files list all the components of the package (including
>> setup.hint), but these md5 sums are not reliable: they often don't match
>> the actual md5 checksum (of the file itself, or of course the md5 stored
>> for it in setup.ini).(**)
>
> during upload of new files the creation of md5.sum is out of sync
> with the directory content. Md5.sum is updated 1 time per hour
> If the mirror sync before the creation of the md5sum it has still the 
> old version.
>

Does that mean that if the md5.sum file were created at the same time 
that the package contents were updated, there would be no possible way 
for out-of-sync md5.sum files to be provided?  Do you think the current 
process could be improved?

I have observed that the errors in the mirrors persist for weeks or more.

Anyway, by changing our checking process to use the information in 
setup.ini instead of the md5.sum files which can be wrong (for many 
weeks), we can bypass the incorrect md5.sum files.

>> 4) The most common file to have the wrong md5 checksum is setup.hint
>>
>> 5) It's not rare for files to be mentioned in a package's md5.sum which
>> are be absent from the package itself.(***)
>>
>> I'm curious about the purpose of having the md5.sum file in each
>> package.  Is it a relic of a previous system?
>>
>> The above observations are based on a few weeks of mirroring and
>> automatically checking the md5 sums of what we downloaded.  The main
>> site we used was aarnet.edu.au (IIRC); recently we changed to
>> mirrors.kernel.org, but from my ad hoc checks there wasn't much
>> difference between the two).
>>
>> Regards,
>>
>> luke
>>
>> (*)
>> For mirrors.kernel.org last night:
>>
>> Worrying: X11/khronos-opengl-registry has no md5.sum file
>> Worrying: X11/xlaunch has no md5.sum file
>> Worrying: cygwin64-gcc/cygwin64-gcc-debuginfo has no md5.sum file
>> Worrying: git/git-oodiff has no md5.sum file
>> Worrying: git/stgit has no md5.sum file
>> Worrying: git/tig has no md5.sum file
>> Worrying: man has no md5.sum file
>> Worrying: python/python-paramiko has no md5.sum file
>
> some of this directory does not exist anymore on www.cygwin.com

Fair enough.  But from what you explained above, it seems strange that 
no md5.sum file exists.  Is it another problem caused by not providing 
the updated md5.sum file at the same time that a package is updated?  I 
wonder if there's a race condition: files in the package are updated, 
one by one, the md5.sum file removed, the new md5 sums computed and the 
file written - and in the meantime, someone may have mirrored the 
partially-updated package?

>
>>
>> (**)
>> $ grep FAILED [path-omitted]/x86/cygwin-archive-incomplete.txt | wc
>>       55     110    1463
>> $ grep "^setup.hint: FAILED"
>> [path-omitted]/x86/cygwin-archive-incomplete.txt | wc
>>       28      56     560
>>
>> (***)
>> $ wc -l < [path-omitted]/x86/cygwin-archive-all-missing-files.txt
>> 406
>>
>>
>
> Regards
> MArco
>
>
> -- 
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>

Thanks for taking the time to explain, Marco.

regards,

luke

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -