X-Recipient: archive-cygwin AT delorie DOT com
DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:date:message-id:subject:from:to
	:content-type; q=dns; s=default; b=g3RwWyKPCsEsU7Pv3vGD0UWj3OHIM
	N0n0dqldW42T/zC3MrBrlh02EgpcM/aH8uZA4MRxRDsnbFflkcgLD7N8vvcDimCO
	Q94/kWHIG5JxfKapC6C3jlisijKRDN9Ffe+QEytc1QCeLThINYEeMqbG8WrQUBuA
	JQW+lAOtIMWAQQ=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:date:message-id:subject:from:to
	:content-type; s=default; bh=mzU4OjLgS/w7L43OKfxmpXX6Epg=; b=G3Q
	V4wB8eI/WiNj8FD8Cpfg9XjYOnAUxkjs1OzC+2jjdB1mO7X4PX7dLyvwSapRpHTP
	w0e5nmmi7kUg/oUUheshtDZwqs2RTMWKwonS6JF/wW0hVcxoewTRUpMCc9oUJmYA
	ApnYs2b1i2LYOlCllXani0TcJK1G6xh0p3NzzJ3E=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Authentication-Results: sourceware.org; auth=none
X-Virus-Found: No
X-Spam-SWARE-Status: No, score=0.7 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL autolearn=ham version=3.3.2
X-HELO: mail-oa0-f53.google.com
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=1e100.net; s=20130820;        h=x-gm-message-state:mime-version:date:message-id:subject:from:to         :content-type;        bh=OE35vpsa+qs3DMzh8r0oT3QXMBiZf4JPwURmIC/mg5g=;        b=e3ktQePf/ddKTP6YE4h+0ffgC1GeUmIOcom6GesQswthTjaesE4LydaMoBjkV1wyaq         fCBlC8UPEUREr0M/Qco2H9IY9+CLovZleQzhxUnGlxnureGxh7RcBQupVKcEL0I9hOtg         T9Mcr3/DDxgrYvIO9rT7XHLuwIuN7oojV2mDuKJEoaHJV7LVDzvzu4S8rAwKlmzW/Swj         VvNm//rupwu6MOyKvJ8shrwy8Qb51UZwEiQLYNSu9zSlacY+QdD0TeLa8DZ0yQaAlnz5         akFY21dkGYPsa5eLim3RpWL6cUYoX5cK5eDfVBhf3W6zA3IHt3KM7CHHKogf5ibRegWM         xL1g==
X-Gm-Message-State: ALoCoQmqklRYv/cUgE5medRmLMN8Gg27otVJtDIBYHmBOF6ry27lWOEQriiROnLsU841j1L04B8v
MIME-Version: 1.0
X-Received: by 10.182.33.33 with SMTP id o1mr43100751obi.24.1410314025140; Tue, 09 Sep 2014 18:53:45 -0700 (PDT)
Date: Tue, 9 Sep 2014 18:53:45 -0700
Message-ID: <CACK47LMM-jPL5a5qhuJhp902YeinbhPPj+SSetSj=kzOeKVPkw@mail.gmail.com>
Subject: mirror synchronization woes
From: Pierre Demartines <pierre AT demartines DOT com>
To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset=UTF-8

Hello,

I am questioning whether it is possible to make a local mirror that is
consistent, i.e. that doesn't contain a mixup of versions due to
non-atomic rsync (mirrors trying to sync from what I imagine is a
master, or between each other, and my server trying to sync from one
of the mirrors).  At the moment, my local copy seems corrupted
(setup-x86.exe croaks on signature error and refuses to go ahead), and
I haven't found another mirror to rsync from that would repair the
situation. Several rsyncs from different mirrors lead to the same
situation: an apparently corrupted local repo.


Details
-------
Being behind a (slowish) firewall, I set up a local cygwin mirror on a
linux server.
I let an rsync-based script on crontab eat the slow connection, once
daily, and most of the time rsync does its thing with minimal impact
and all is well.

However, I've bumped today into what seems to be a mismatch of
signatures and versions.  I've tried several mirrors as the source for
my local copy, but I've been unable to find one that let's the client
cygwin installer work.
Instead, when running setup-x86.exe (the latest one, always downloaded
by the same script as the one running rsync), I get a "Mirror error:
Setup.ini signature (...) setup.bz2.sig from (...) failed to verify".

Thinking a bit more about it, I wonder what mechanisms, if any, are in
place to provide consistency and atomicity of the process?  Is there a
special time of the day when the "master copy" is frozen, the
signatures and md5sum are calculated, and a new version is released
for the mirrors to pick up?
Also, if the mirrors are simply updated via rsync, how do I make sure
that I am not myself rsyncing when they are in the midst of it?

I was curious about timestamps and md5sums on various mirrors (I
looked only in the US ones --I put the one-liner to do that as
post-scriptum). And indeed, I have found that different mirrors have
different signature files, and with various timestamps.

For example, as of 20140910-001532UTC, I find the following timestamps
for various setup.bz2.sig:
- 20140908-231037UTC mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.bz2.sig
- 20140909-191037UTC mirrors.syringanetworks.net_cygwin/x86/setup.bz2.sig
- 20140909-191037UTC mirrors.xmission.com_cygwin/x86/setup.bz2.sig
- 20140909-201044UTC mirror.team-cymru.org_cygwin/x86/setup.bz2.sig
- 20140909-231038UTC rsync.osuosl.org_cygwin/x86/setup.bz2.sig
- 20140910-001035UTC mirrors.kernel.org_sourceware_cygwin/x86/setup.bz2.sig

Worse, their md5sum don't match:

# grep -n -E 'bz2.sig$; */x86/md5.sum | perl -ne 'chomp;s/:/
/g;@a=split;print join("\t",@a[2,0])."\n"'
f88c73ee59cf59f7f32ca4bad26376c1
mirrors.kernel.org_sourceware_cygwin/x86/md5.sum
15a0cca932d75390e01b54a46b2558e2
mirrors.syringanetworks.net_cygwin/x86/md5.sum
f449fdb08944b8563c0ad273069669f1
mirrors-usa.go-parts.com_mirrors_cygwin/x86/md5.sum
15a0cca932d75390e01b54a46b2558e2        mirrors.xmission.com_cygwin/x86/md5.sum
413c8532019a41a1b310482cad0d25bb        mirror.team-cymru.org_cygwin/x86/md5.sum
6616150da358594eaa613032b66d2681        rsync.osuosl.org_cygwin/x86/md5.sum


Any suggestion or explanation?

Thanks!

~Pierre



PS: test all the signatures of the US-based rsync:// servers:

# mkcd ~/tmp/cygwin_signatures

# for i ($(perl -ne 's,/",",g;print "$1\n" while
(s,"rsync://(.*?)",,)' us_mirror_list.html)); do j=$i:gs,/,_; echo $i;
rsync -vaz --exclude 'release/' --progress --timeout 3600 rsync://$i/
$j/; done

# head -20 */x86/md5.sum
==> mirrors.kernel.org_sourceware_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb  setup.bz2
f88c73ee59cf59f7f32ca4bad26376c1  setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  setup.ini
4b0a55de2687c537124139bd569328ae  setup.ini.sig

==> mirrors.syringanetworks.net_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb  setup.bz2
15a0cca932d75390e01b54a46b2558e2  setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  setup.ini
d2d94f4e1734e896af4efa44e4c6654f  setup.ini.sig

==> mirrors-usa.go-parts.com_mirrors_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb  setup.bz2
f449fdb08944b8563c0ad273069669f1  setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  setup.ini
9ff8b9cfb0e29b9960ae918bb68a4dd6  setup.ini.sig

==> mirrors.xmission.com_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb  setup.bz2
15a0cca932d75390e01b54a46b2558e2  setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  setup.ini
d2d94f4e1734e896af4efa44e4c6654f  setup.ini.sig

==> mirror.team-cymru.org_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb  setup.bz2
413c8532019a41a1b310482cad0d25bb  setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  setup.ini
212b298441fdf11ee45ef3841dd38b62  setup.ini.sig

==> rsync.osuosl.org_cygwin/x86/md5.sum <==
a7f25f8d488a30b68cf1c08f66ca35cb  setup.bz2
6616150da358594eaa613032b66d2681  setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  setup.ini
75dd5f39d40c255951daf0ab5b865333  setup.ini.sig

# md5sum */x86/setup*
a7f25f8d488a30b68cf1c08f66ca35cb
mirrors.kernel.org_sourceware_cygwin/x86/setup.bz2
f88c73ee59cf59f7f32ca4bad26376c1
mirrors.kernel.org_sourceware_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936
mirrors.kernel.org_sourceware_cygwin/x86/setup.ini
4b0a55de2687c537124139bd569328ae
mirrors.kernel.org_sourceware_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb
mirrors.syringanetworks.net_cygwin/x86/setup.bz2
15a0cca932d75390e01b54a46b2558e2
mirrors.syringanetworks.net_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936
mirrors.syringanetworks.net_cygwin/x86/setup.ini
d2d94f4e1734e896af4efa44e4c6654f
mirrors.syringanetworks.net_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb
mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.bz2
f449fdb08944b8563c0ad273069669f1
mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936
mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.ini
9ff8b9cfb0e29b9960ae918bb68a4dd6
mirrors-usa.go-parts.com_mirrors_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb  mirrors.xmission.com_cygwin/x86/setup.bz2
15a0cca932d75390e01b54a46b2558e2  mirrors.xmission.com_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  mirrors.xmission.com_cygwin/x86/setup.ini
d2d94f4e1734e896af4efa44e4c6654f  mirrors.xmission.com_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb  mirror.team-cymru.org_cygwin/x86/setup.bz2
413c8532019a41a1b310482cad0d25bb  mirror.team-cymru.org_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  mirror.team-cymru.org_cygwin/x86/setup.ini
212b298441fdf11ee45ef3841dd38b62  mirror.team-cymru.org_cygwin/x86/setup.ini.sig
a7f25f8d488a30b68cf1c08f66ca35cb  rsync.osuosl.org_cygwin/x86/setup.bz2
6616150da358594eaa613032b66d2681  rsync.osuosl.org_cygwin/x86/setup.bz2.sig
2728b16b23c2c6d697b5ae1fb3a4d936  rsync.osuosl.org_cygwin/x86/setup.ini
75dd5f39d40c255951daf0ab5b865333  rsync.osuosl.org_cygwin/x86/setup.ini.sig

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple