DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 6E2F13858C3B
Subject: Re: updatedb broken as of findutils 4.8.0-1 due to bigram.exe no
 longer being provided
To: "cygwin@cygwin.com" <cygwin@cygwin.com>
References: <986736274.144968.1630167325057.ref@mail.yahoo.com>
 <986736274.144968.1630167325057@mail.yahoo.com>
 <a60ffa68-274a-5072-c90a-0dce7bc93431@harkless.org>
 <3457cee1-18b5-2916-adee-afdfaf9769ea@t-online.de>
From: Dan Harkless <cygwin-list21@harkless.org>
Message-ID: <525a832a-78fd-5a32-e195-5747120da922@harkless.org>
Date: Sun, 29 Aug 2021 05:06:37 -0700
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
MIME-Version: 1.0
In-Reply-To: <3457cee1-18b5-2916-adee-afdfaf9769ea@t-online.de>
Content-Language: en-US
Precedence: list
Cc: bug-findutils@gnu.org
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces+archive-cygwin=delorie.com@cygwin.com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie.com@cygwin.com>
Content-Transfer-Encoding: 8bit

On 8/29/2021 4:02 AM, Hans-Bernhard Bröker wrote:
> Am 28.08.2021 um 18:23 schrieb Dan Harkless:
>> Looks like it's because in findutils 4.8.0-1, the bigram.exe program 
>> is no longer provided, but the /usr/bin/updatedb script (still) 
>> depends on it being there:
>      [...]
>>      + for binary in $find $frcode $bigram $code
>>      + checkbinary /usr/libexec/frcode
>
> The version of updatedb in the 4.8.0-1 package does not actually 
> contain those lines.  Mention of both $bigram and $code has been 
> removed from the loop construct (and from everywhere else in the script).
>
> That's because the old format of find databases, which is the only one 
> actually using bigram and code, was removed from updatedb as of 
> findutils version 4.7, so there really cannot be a need for the bigram 
> tool any more.

Argh!  So sorry for the false report!  I completely forgot that years 
back I had made a locally patched version (which is earlier in my path) 
of Cygwin updatedb 4.6.0-1 to troubleshoot and work around problems I 
was having with the tool.

I have 12M+ pathnames on my main Windows system, and I suddenly started 
having issues with the updatedb going from taking less than an hour, to 
taking more than 24 hours, and running into the next job.

It was very awkward to try to troubleshoot what was going on without a 
'find' log to 'tail', so I patched my  local copy of updatedb to write 
to an intermediate file, rather than going direct to 'sort' over a pipe.

Another problem I was having was that though I have 24 GB of RAM on my 
system, I would get low-memory popup warnings from the OS when the sort 
would go off.  (The warnings mislay the blame on Firefox, because I 
usually have big sessions running that take even more RAM than the sort.)

I was hoping running sort on a _file_ rather than stdin might allow it 
to reduce the RAM use enough to not get the warning, but unfortunately 
(and unsurprisingly) I still get it with the intermediate file.  This is 
just a warning, though — I haven't had it actually run out of RAM so 
far, I don't think.

The final problem I was addressing in my patched version was some 
missing error-checking, which was causing me to be left with _no_ 
filename DB, when the update would fail, rather than at least being left 
with the one from last time.

I could send along my patches, but I don't know that I've solved these 
issues in a general enough way.  For instance, my 12 million+ pathnames 
come out to about 1.4 GiB of UNIX-linefeed-separated UTF-8 strings.  
Writing that much to my HD is not a concern, but obviously some people 
might not want to write that much every time to, say, a small 
flash-based device.

Thoughts?

-- 
Dan Harkless


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple