X-Recipient: archive-cygwin AT delorie DOT com
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 0327E3994813
Authentication-Results: sourceware.org; dmarc=none (p=none dis=none)
 header.from=SystematicSw.ab.ca
Authentication-Results: sourceware.org;
 spf=none smtp.mailfrom=systematicsw.ab.ca
X-Authority-Analysis: v=2.4 cv=Ac10o1bG c=1 sm=1 tr=0 ts=60f7013a
 a=T+ovY1NZ+FAi/xYICV7Bgg==:117 a=T+ovY1NZ+FAi/xYICV7Bgg==:17
 a=IkcTkHD0fZMA:10 a=CCpqsmhAAAAA:8 a=e5mUnYsNAAAA:8 a=FhwSYsPFeF_qToIT26AA:9
 a=QEXdDO2ut3YA:10 a=o1yDpi_DOQcA:10 a=ul9cdbp4aOFLsgKbc677:22
 a=Vxmtnl_E_bksehYqCbjh:22
To: cygwin AT cygwin DOT com
References: <DB6PR03MB3013DDDB803E3E4B8EF8E935A4290 AT DB6PR03MB3013 DOT eurprd03 DOT prod DOT outlook DOT com>
 <DB6PR03MB301312724F0DFF66AF9E796DA4270 AT DB6PR03MB3013 DOT eurprd03 DOT prod DOT outlook DOT com>
 <782011494 DOT 20200910083521 AT yandex DOT ru>
 <DB6PR03MB3013D1FD4D925B12A4022E98A4270 AT DB6PR03MB3013 DOT eurprd03 DOT prod DOT outlook DOT com>
 <5a2fdf46-93c8-048b-cadb-cb9d9212c716 AT SystematicSw DOT ab DOT ca>
 <20200910230426 DOT 5811f3e8 AT quarternote>
 <VI1PR03MB3022EBE68767183BCA4889ECA4240 AT VI1PR03MB3022 DOT eurprd03 DOT prod DOT outlook DOT com>
 <DB7PR02MB39967134A1C1169A8A406BE8E7240 AT DB7PR02MB3996 DOT eurprd02 DOT prod DOT outlook DOT com>
 <8f40571c-1a37-8e4b-f1bd-ecf40175d0d7 AT gmail DOT com>
 <179bbaf0-02b4-1c63-0083-5fa2a8833ea9 AT cornell DOT edu>
 <06e9cf44-8cc8-267e-12b3-e8a866a01c80 AT gmail DOT com>
 <ee4d59c7-00d3-a2e6-fa13-04622ed8391f AT cornell DOT edu>
 <87h7s45a4c DOT fsf AT Rainer DOT invalid>
 <3305b90c-41f2-7377-092d-0f151a83da1c AT cornell DOT edu>
 <abb3d78e-7679-db16-e219-7e32b04942c0 AT SystematicSw DOT ab DOT ca>
 <9c515b79-42e2-5a9b-1996-9281f3c2b0e8 AT cornell DOT edu>
From: Brian Inglis <Brian DOT Inglis AT SystematicSw DOT ab DOT ca>
Organization: Systematic Software
Subject: Re: postinstall: fontconfig abnormal exit
Message-ID: <cc8d11d5-ca74-4a0b-a108-47623088cd2a@SystematicSw.ab.ca>
Date: Tue, 20 Jul 2021 11:00:39 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.12.0
MIME-Version: 1.0
In-Reply-To: <9c515b79-42e2-5a9b-1996-9281f3c2b0e8@cornell.edu>
Content-Language: en-CA
X-CMAE-Envelope: MS4xfC0jtycNcRdN6UZBsrl1nSFyP/m/GFRuGEqUG2iPCIAnmBaTZeyKgZoXELQlWZZlbWxztDlJPmlAY9QYKLTpHdRMtWJYLXnMqlSdgGCz/G3sjaAY9793
 2eu3N/rcVQQeVXlJKsYjgztALzotKYo414S68pC53csAkKKiclJCW8C0DBRc5FT+4X/8wwdYDeyrS/YNpWQpxCQ6AkTP6oM2WC0=
X-Spam-Status: No, score=-1161.1 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS,
 KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_BARRACUDACENTRAL,
 RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE,
 TXREP autolearn=no autolearn_force=no version=3.4.4
X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Reply-To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 16KH15o5029959

On 2020-09-12 06:56, Ken Brown via Cygwin wrote:
> On 9/12/2020 1:18 AM, Brian Inglis wrote:
>> On 2020-09-11 15:13, Ken Brown via Cygwin wrote:
>>> On 9/11/2020 4:30 PM, Achim Gratz wrote:
>>>> Ken Brown via Cygwin writes:
>>>>> Unfortunately, this doesn't yet fix the problem with
>>>>> fontconfig_dtd.sh.  The latter will now succeed if it is run after
>>>>> libxml2.sh, but not if it is run first.  I'm not aware of any way to
>>>>> force setup to run one postinstall script before another.
>>>>
>>>> Multiple ways:
>>>>
>>>> 1. Make the libxml postinstall script sort lexically before any
>>>> others that depend on it. Obviously this is brittle, but it
>>>> might work in this particular instance (autorebase does this).
>>>>
>>>> 2. Make the libxml catalog creation a perpetual postinstall
>>>> script with prefix "0p_". That only works if it doesn't depend
>>>> on other postinstall scripts having their work completed.
>>>>
>>>> 3. Implement and use the stratified postinstall concept originally
>>>> outlined at:
>>>> https://sourceware.org/legacy-ml/cygwin-apps/2014-12/msg00148.html
>>>>
>>>> 4. Use the package dependency order to order the postinstall script
>>>> activation.  With libzypp we should have the correct information, we'd
>>>> just need to somehow make the packagemeta iterator use it. That still
>>>> won't work if we have dependency loops.
>>>>
>>>> 5. (Try to) Run any failed postinstalls again in setup and bail only if
>>>> the number of fails does not decrease from the last iteration.
>>>>
>>>> At the time it was deemed too complicated and so we only use
>>>> the "end" strata for the perpetual postinstall scripts.  As
>>>> said then, there would need to be some serious discussion on
>>>> how to coordinate the strata assignments.

>>> There's no dependency relation between libxml2 and 
>>> libfontconfig-common, so #4 wouldn't fix the problem. And some of
>>> the other suggestions would require work on setup.exe that
>>> someone would have to do. The current problem is simple to fix
>>> and shouldn't have to wait for that.
>>>
>>> I like your idea of using perpetual postinstall scripts.  I think
>>> the way to do it is probably to make fontconfig_dtd.sh perpetual
>>> with prefix "zp_". That way if libfontconfig-common is installed
>>> without libxml2 but then libxml2 is installed later, the
>>> xmlcatalog command gets run.  We would have to check that no harm
>>> is done if that xmlcatalog command gets run more than once.

>> As libxml2 supplies /usr/bin/xmlcatalog, that postinstall script 
>> should both conditionally create the catalog and only if just
>> created, also conditionally add fonts.dtd,

> No, the libxml2 postinstall script shouldn't add fonts.dtd. That
> should be added only when/if fontconfig is installed. So it's an
> appropriate task for a fontconfig postinstall script.

>> [TL;DR: We do not want to add more permanent postinstall scripts 
>> unless essential!

> Making fontconfig_dtd.sh perpetual may or may not be essential, but it's 
> the best way I can think of to solve the problem of this bug report.  
> Can you propose a better way?

>> Permanent postinstall scripts can greatly extend the run time of
>> the Cygwin Setup program, when fontconfig, man-db, or tex just seem
>> to decide sometimes for some reason that another whole new world of
>> fonts, man pages, or whatever has appeared that they have to
>> reprocess,

> If you see a perpetual postinstall script doing something time-consuming 
> that you think is unnecessary, please make a bug report.

Unfortunately it's often too late to get any information once you notice 
setup is still running in one script.

>> pushing the Setup run time and service (cron) downtime to hours

> I've never seen this.  Please give details.

I have never been able to figure out what texlive postinstall command 
keeps going.

I have multiple distros man pages available under Cygwin for 
convenience, so I manually run man-db postinstall after updates (nohup 
... &), to try to avoid long setup postinstalls, but sometimes setup 
man-db postinstall does lengthy updates anyway.

For fontconfig fc-cache-1 appears to have been creating thousands (on 
Cygwin 64 millions) of small <1KB 
/var/cache/fontconfig/%8x-%4x-%4x-%4x-%12x-le{64,32d8}.cache-7 files.

The problems could have originally been caused by an old bug:

https://gitlab.freedesktop.org/fontconfig/fontconfig/-/issues/107

combined with many font additions around that time, mainly working on 
Cygwin 64, where I use X, and manually run fontconfig postinstall 
script, to try to avoid long setup postinstalls, whereas on Cygwin 32 I 
don't use X or manually run postinstall scripts, just get run after setup.

I have about 200 Windows MS font files, 1000 non-MS font files, and 
about 800 Cygwin font files, from multiple distros and elsewhere, 
including some with full BMP coverage, some with SMP coverage, some for 
fallback code points, others with group ranges.

I rm'ed -rf /var/cache/fontconfig/ with a few thousand files on Cygwin 
32 and rebuilt it okay with only 65 cache files.

I tried rm -rf /var/cache/fontconfig/ on Cygwin 64 but got many 
permission errors and killed it.

I gave up waiting for ls -U to show any results on Cygwin 64 or Explorer 
on that dir to show any file details, but cmd /c dir | less displays the 
base info for hundreds of thousands of files, and wc reports millions.

I am still waiting for an elevated cmd to rmdir /s /q fontconfig there!

Do you know why fc-cache-1 is run rather than fc-cache and what the 
difference is?

What would give you useful information once I have the fontconfig cache 
cleared?
Might running FC_DEBUG=65535 fc-cache-1 -fsv provide useful info?
Or do so with strace?
Would running file on the font files give enough info about properties 
to be of any help?

What is the best approach to get the minimal cache files recreated?

What is the best approach to avoid thousand of cache files in future?

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple