delorie.com/archives/browse.cgi | search |
X-Recipient: | archive-cygwin AT delorie DOT com |
X-Spam-Check-By: | sourceware.org |
From: | "Gary R. Van Sickle" <g DOT r DOT vansickle AT worldnet DOT att DOT net> |
To: | <cygwin AT cygwin DOT com> |
References: | <6a42eec70710311438s273b63dcxc6c741dde4593afc AT mail DOT gmail DOT com> <47294BE7 DOT E195E110 AT dessent DOT net> |
Subject: | RE: copying a million tiny files? |
Date: | Wed, 31 Oct 2007 23:13:45 -0500 |
Message-ID: | <001c01c81c3d$992990c0$020aa8c0@DFW5RB41> |
MIME-Version: | 1.0 |
X-Mailer: | Microsoft Office Outlook 11 |
In-Reply-To: | <47294BE7.E195E110@dessent.net> |
X-IsSubscribed: | yes |
Mailing-List: | contact cygwin-help AT cygwin DOT com; run by ezmlm |
List-Id: | <cygwin.cygwin.com> |
List-Unsubscribe: | <mailto:cygwin-unsubscribe-archive-cygwin=delorie DOT com AT cygwin DOT com> |
List-Subscribe: | <mailto:cygwin-subscribe AT cygwin DOT com> |
List-Archive: | <http://sourceware.org/ml/cygwin/> |
List-Post: | <mailto:cygwin AT cygwin DOT com> |
List-Help: | <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs> |
Sender: | cygwin-owner AT cygwin DOT com |
Mail-Followup-To: | cygwin AT cygwin DOT com |
Delivered-To: | mailing list cygwin AT cygwin DOT com |
> From: Brian Dessent > > sam reckoner wrote: > > > I'm not exaggerating. I have over one million small files > that like to > > move between disks. The problem is that even getting a directory > > listing takes forever. > > > > Is there a best practice for this? > > I know it's heresy but if you just want to copy files why not > use the native XCOPY? It will not suffer the performance > degredation of having to emulate the POSIX stat semantics on > every file, just like the native DIR command in a large > directory does not take ages because it simply uses > FindFirstFile/FindNextFile which are fairly efficient (but do > not provide sufficient information to emulate POSIX.) > > Brian > I have a similar situation to the OP (copying many thousands of small files over a fairly slow link), and actually timed using XCOPY vs. Cygwin methods (cp in my case). It didn't make a significant difference. Ultimately what I think you run into in these sorts of situations is that you bump up against the slowness of the link (or physical disk) because, POSIX emulation or not, all your caches do is thrash. -- Gary R. Van Sickle -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
webmaster | delorie software privacy |
Copyright © 2019 by DJ Delorie | Updated Jul 2019 |