DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:from:date:message-id:subject:to
	:content-type; q=dns; s=default; b=EX7heSEaUXoOrTtOagwwfmYQsowUN
	So+ABkiMbuSrDgDembYqONyiWqhM668Y4CPBcHPoiOhwiDSUUZzJ3keQQKuQ0EQ6
	JvxZWzeLUQnsJEqu1DrK+gfQKFO+OnmuVm0vx26jSJVttAjG2/ftcjRCi1TsUU2z
	rN6Oa1IxUxdDwA=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
MIME-Version: 1.0
From: Marco Mason <marco DOT mason AT gmail DOT com>
Date: Sun, 30 Sep 2018 14:41:07 -0400
Message-ID: <CANNqMjAGEm64Z4ULhbe4KtcmT1Y7njOYoJCG6V_KbrFuj1dj=A@mail.gmail.com>
Subject: Filesystem enumeration performance improvement
To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset="UTF-8"

I recently upgraded from cygwin v2.10 to v2.11.1 and noticed that one of my
programs got a tremendous speed boost.  It's a custom filesystem
enumeration program whose output I feed to frcode to update the
/var/locatedb database.  It used to take quite a bit of time (15-20
minutes?), and now runs in about a minute.  Since the program seems to work
well, just many times faster, I'm rather happy with the changes.

The reason I'm writing is that I don't see *why* I should have any timing
changes at all!  The reason I have my own file enumerator for locatedb is
that the original went through the POSIX layer and was pretty slow,
especially for remote-mounts.  As I only needed enough for locate, I wrote
my own enumerator against the Windows API for speed.  Since my loop is
essentially just using FindFirstFile/FindNextFile and printf(), I don't
know why file gathering would be any faster.

So either printf() has gotten remarkably faster, or there are some
interactions between Cygwin and windows in the file enumeration area that
are surprising me.  Can someone please clue me in to what might be causing
the speed increases?

Looking at the git log and mailing list history, my best guess would be
that it's related to the EMail threads  "Why does readdir() open files ?"
(Ben Rubson 2018-03-28) and "Why does (stat() ?) open files ?" (Ben Rubson
2018-04-09).  However, I can't seem to pin down which git commits are
relevent to those threads.  If anyone can provide a little insight, I'd
really appreciate it.

--marco

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple