Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sources.redhat.com/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sources.redhat.com/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Message-ID: <00ac01c2d3e2$c1d4bed0$ab7886d9@webdev>
Reply-To: "Elfyn McBratney" <cygwin AT cygwin DOT com>
From: "Elfyn McBratney" <elfyn-cygwin AT exposure DOT org DOT uk>
To: "cygwin" <cygwin AT cygwin DOT com>
References: <5 DOT 2 DOT 0 DOT 9 DOT 2 DOT 20030213182750 DOT 01e97e98 AT pop3 DOT cris DOT com> <5 DOT 2 DOT 0 DOT 9 DOT 2 DOT 20030213185143 DOT 01da0ef0 AT pop3 DOT cris DOT com> <5 DOT 2 DOT 0 DOT 9 DOT 2 DOT 20030213190807 DOT 03359eb0 AT pop3 DOT cris DOT com>
Subject: Re: Wget  ignores robot.txt entry
Date: Fri, 14 Feb 2003 04:37:17 -0000
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106

> Max,
>
> No, I don't think cURL does recursive retrieval. I don't think it does
> Web page dependency retrieval, either. Both of these are a big deal for
> me. How could a tool of wget's versatility be replaced by something
> inferior? Whatever happened to technological meritocracy? (Please, no
> laughing.)
>
> I was actually hoping to get some time to work on an extension to wget
> of my own. I wanted to add an option that would cause wget to look in
> one hierarchy to determine file existence and modification times
> relative to the set of files and mod times on the server and download
> new or newer files to a different location. That way I can easily
> maintain mirror copies on a CD-ROM. I'd tell wget to use the CD's
> contents as the file and mod-time reference and to download to a
> location on my hard drive (of course). Then I could incrementally
> update the ROM with whatever was downloaded.

That's a real good idea! :-)

> Of course I can still do that and I may yet. Does that sound like a
> desirable feature to anyone? I don't know how many people share my
> mania for keeping local archives of content from the Internet.

I seem to end-up doing this quite a lot when on a hunt for new concepts and
ways of doing things. A 'uge web suck, most stuff I never glance a quarter
of my eye over, and I got a whole archive of stuff where i can just grep out
the crap.

> What happens to an open source project when it devolves to this state?
> Who, for example, could hand out writable access to the wget CVS
> repository? Surely this isn't an unrecoverable state of affairs, is it?
>
> Randall Schulz

Wasn't a patch applied to CVS HEAD of the wget repos only a few weeks ago.
Thats what it looks like anyway.


Regards,

Elfyn McBratney
elfyn AT exposure DOT org DOT uk
www.exposure.org.uk



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/