DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 4B3M04ej2511484
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 266533858C41
Message-ID: <PH7PR11MB64521BADC2AEC9B131AD49C4A4362@PH7PR11MB6452.namprd11.prod.outlook.com>
Date: Tue, 3 Dec 2024 13:58:45 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: Calling GetConsoleProcessList in tight loop allocates new buffer
 via condrv ioctl results in excessive page-faults with Windows
 Terminal
To: Takashi Yano <takashi DOT yano AT nifty DOT ne DOT jp>, cygwin AT cygwin DOT com
References: <PH7PR11MB64522B9A097A9A5D963E5C23A4352 AT PH7PR11MB6452 DOT namprd11 DOT prod DOT outlook DOT com>
 <20241203204242 DOT 4bd45fc54e31431ceaf94816 AT nifty DOT ne DOT jp>
Content-Language: en-US
In-Reply-To: <20241203204242.4bd45fc54e31431ceaf94816@nifty.ne.jp>
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: PH7PR11MB6452:EE_|SN7PR11MB8027:EE_
X-MS-Office365-Filtering-Correlation-Id: dc4c4196-596a-4dfb-ab65-08dd13e5aa98
X-Microsoft-Antispam: BCL:0;
 ARA:14566002|461199028|6090799003|19110799003|5072599009|8060799006|15080799006|10035399004|440099028|3412199025|4302099013|1602099012;
X-Microsoft-Antispam-Message-Info: =?utf-8?B?WmlJbTNkblhCTXY3VUZtMnlaVzRBM2hIZldIazdDdFVWdWJnVyt1Tm5mazM5?=
 =?utf-8?B?Z1d0QmJNQ1pESndUcVJiekc1MlhIVnZHbk9tVTIwU1hib2xjQWJ0S1loNjNF?=
 =?utf-8?B?aC9Ya245dlZScU9nOStucmlZRStzbHJFU2xVdGJwczdvaHlURks3MTl1UDBD?=
 =?utf-8?B?M3pId3NLa2M2dWtoVFBRUzViWFdmVlNwWWhuekFFNUlVd2VVSU1YOGdhak1z?=
 =?utf-8?B?N2JJeGFrV1FtR2JQSXA2ejUzVGpNTGdJV2VWNWhaYzJDQmdDQXNWamJzVE42?=
 =?utf-8?B?WkNjWHF5MkhwNno1YkFIZDFoVE5XTERxTHMwT1YzUkUrZ0I3dmR4TnZUaG8z?=
 =?utf-8?B?TXNUMDdyV2hKWUxKSzNMNmxXa1ZFdVlCQTVEVWNuZCt4VjVtREgvMDN6bjRk?=
 =?utf-8?B?ZGhXeUJLR1NTejUzTTRCZnAyRWwwS2ZQb0dyNDJVUXZlcnJWcThzR0hveVNW?=
 =?utf-8?B?Z2ZpM2ZaVmVOenk2cnBKOVI3alc1T0RNS0xhMFZraUU0aDI2Yk9SYmVtNFh1?=
 =?utf-8?B?cCt6S0tmeEVEOVY5Q3AzOUxheTJLcWhERnFUU0tobUFiVmUrNzJGcnA3QkIv?=
 =?utf-8?B?cHdqa0x3L21Nb0hJblhDR09Mcm45MHpaR2RoMXZIdE51aUZqeTY0MmNLUDNq?=
 =?utf-8?B?bURJbWZQWVpPanp4YnFaOW83UXZnR2NtWTFPWm9SS2FiNnNNU2ZkUUxUMmZ6?=
 =?utf-8?B?SW93VXBsK3ZHUzFRQXNHWUJvMDU0azZMZGFhTlFucEQ0OHR2bUgydENXVGRM?=
 =?utf-8?B?R0FCMWRIendrUE1mV3JZamo5U1ViUVBFTHNaN2VQK2d3NDZXYWJ1NW9jMlJy?=
 =?utf-8?B?Sk4xSkZlS1BFbHZwRHZwZjErNkd0UnVydXVVRHFpZWRRaUt6a3VJalhrRWt2?=
 =?utf-8?B?b2Yyc3cwWmhiTEdIbmhuemdUVVkweEJnTXJyazlHbXJ0cEFWOFV5RENhV3lv?=
 =?utf-8?B?UU9UU2JKalRPYldUNS9ZSEdzaVNheXBnUGZrU2JyYWR6V2NKeVY3S1J2ZUI2?=
 =?utf-8?B?WHE3WDkxQ2hzR1pUZnNuUFhTVmpwMFR1UHNEb2trL1JYa2xNUjByNzk3MEt3?=
 =?utf-8?B?czRjNmh4bllMZHVKU0kxVWpoQ1F1eXZvYmFVOTRWaWlZN3RFSTI3MHJiZmdM?=
 =?utf-8?B?MnFVZjhyb2pDclE2ei9vdVNWbXJodTBGcm9TNUQyT0J2WkJhR2JBWktiem1w?=
 =?utf-8?B?RjlQNEIyYzVyRGZDQm04YVB3TnlySTNja1Q2RTE1Y3NaLzJtUGF5N2hId2Qy?=
 =?utf-8?B?Q2VtZ0p0Y0hEZmZCQ2RkUTRpTjQ5am00MnoxU0xtWEdhUFh4NkQyREV5MkdE?=
 =?utf-8?B?d3JXYkpLak4xZjRSUW5pT2lrdDY0ZGRRU0dqeUltVTY0QTJUMDgxY3h4MGZW?=
 =?utf-8?B?a2psbWxhQmp5T1FuZzZUWU5YYXNpS2tySFA1bGJTV2cxM3hMc0tXRDZoZ1FW?=
 =?utf-8?B?Y1JhdzhjcDdQdDBUWTd2eStoSkxJdTZPZEdpWnFtYVdqL09Ec2FpYW1NQjNS?=
 =?utf-8?B?TjhYWGhwaUxGMFYyUDhEb0pYNktrWkV2end0OWlkSVo1UlZuVTdLRlBTRC9t?=
 =?utf-8?B?cnovZz09?=
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a010enFiYlpuWnFiN0l2WFlobzdkQitqM3plS1JnQnBqbVhMcWI2RmI3WFdv?=
 =?utf-8?B?VlRBWmR1TW1LdU13NURZd3l5ZG4xN3d2akdqRmxRT0lSM0JyVlhmTlhSZGZi?=
 =?utf-8?B?NGNZSmRNcEI5RklsN29pa21YdHU1S3Q4MnlCblAwRnRIZlFsTzJiaUxTNytk?=
 =?utf-8?B?OFN5aE11cmpjRjMzZmpERi9XVWQyN2N4dDVucHpQZVhjWFkxalNKWjJ0T3BE?=
 =?utf-8?B?cGtDVGRSaEJodkRuQU9MYUpnOTRsQ0llSnp1OUJRRVVDdk1vd05Vb2xxaGh6?=
 =?utf-8?B?TVlvekY1RHZnZFNDd0tUZnBTUm94bDBNSTl5dDFsUmgvOHdPSkozT1dSZkNv?=
 =?utf-8?B?aWlLZ2pqL3FzYTRsclBvK0k0bk5uMU5uK2xHMDdLVS9yQy9SYkk2NzBnWCtp?=
 =?utf-8?B?eEhldG1rc21pK2pHL01PcXJFWEMwZDRGRnY0Rit4bUtzMTVHTXhuTm9HdEFz?=
 =?utf-8?B?YVFOdWRCYUZrbWhXNTRoazZXR2FwQlYyMVNyYUxZWXFpVkZNc1FLb0E5dFJN?=
 =?utf-8?B?cXhYSmJhTHNKVXBvaVRkMHVKSzJsRG9Ba1Y3R1JDRWVXd2hYMlBEejIwbUZY?=
 =?utf-8?B?Ry9HUW5DbGMwNWFrcnJ0RnVCZTF2RzFZT0tWZTVMVjFSUzlXZ21ZVXZvR21j?=
 =?utf-8?B?bmd1UU9KdW5hbElJRVZMc09ZWEhTZU9CQVpsY2xXajZXaEk1K3ZSWlVvek1q?=
 =?utf-8?B?WWZURk0rNStaTFVUNDRlQUlyTVJCcUdpandlY3pYMUl5dG44SGpoRFlaQlQ0?=
 =?utf-8?B?Wk9CbmtHUC85aXhoMWlySERSbjRZLzFGNEdKWmlmc0Z4MUpkZkxMT214dWpS?=
 =?utf-8?B?Q0x2Z0RGUlVUdkV1NU9jbi9VQWVoR01iNFV3Y3h4U3M4TlEyMjdXRE1mU2tY?=
 =?utf-8?B?Z2JIUjJXZ1lOTGVmc3dHLzBkbjI0cGhwSHI4MWVQYkQ3VGRaZmVjQUFzUVBR?=
 =?utf-8?B?Z1N0TlNjbzJ4bC9YVjl4eE9YR1NUY3NxbXZCNkVvVFBKZDc1WGg0NHJKWVVr?=
 =?utf-8?B?U2hYaUUzMmhINjBHQVpyS1ZYN3laOWZ6L09VUDFyc0FyQTJyZXF2QjF1b2VS?=
 =?utf-8?B?YXJ2V0Q0MDhXSkFqK0tRYWpkaXBXbkxBdlp3WVArVzE4MitpTUsvem1SVmEy?=
 =?utf-8?B?NXArZnY0a0Zhck8zTjNBM2tRZ2IwRnlaeG9DTVhUQ1FKVnFuUHJCSjhCdEVU?=
 =?utf-8?B?U1VsZ1djVnF2a3ArY1VJT2ZmWCtKZXlCQ0lXUVJ0ZGwzZ0xlWFpjVVNDdzUx?=
 =?utf-8?B?THNlQVVCN0hVZVpIQXZodFlQcDdCRGc3dDIxNktJY29vMWJHN0tFdloyQkQv?=
 =?utf-8?B?TEQ4T1pWcld0MktpM0MwcS9MU3MvTUgzTWZvNlBPMk1mZlQ4QUhWbkg5ZjZL?=
 =?utf-8?B?QWhibDcxaG9SdWlzL1RjcDE1UTduTWVRTjNERVIyblVLZWorWDduRmdES211?=
 =?utf-8?B?VDg5ZDdOY1RSVkQxby80MVUrRHowdy9WQVZMNlZGZ3NQZENaNWkrcnQrL3V4?=
 =?utf-8?B?R2VIYVM1MGVpblJUQWRacWZBK3orRE5WaEZ4aXNoNHZWMU5ON0UrSWxLbG5L?=
 =?utf-8?B?QWlMY3plZFVoamlkNmpJa3hZNVc1c2psZVlGU1kwL20xVWhFeWNtSXdKeGRF?=
 =?utf-8?Q?va+7ULeM3A7p7kQSWP/if3B0VHYNFYH4NZbA8Tscksxs=3D?=
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dc4c4196-596a-4dfb-ab65-08dd13e5aa98
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB6452.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Dec 2024 21:58:48.9085 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB8027
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Steven Buehler via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Steven Buehler <buehlersj AT outlook DOT com>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>

On 12/3/2024 3:42 AM, Takashi Yano wrote:
> On Mon, 2 Dec 2024 19:57:25 +0000
> Steven Buehler wrote:
>> I am experiencing an abnormal number of page-faults per second (averaging 800 to over 2000) when using Cygwin within the Windows Terminal app. This produces visible stutters and cursor movement during terminal screen redraws.
>>
>> I have opened an issue on the Windows Terminal GitHub issues page. The initial investigation by one of the Windows Terminal developers has determined that "Cygwin is calling console APIs in its steady state. It looks like it's calling GetConsoleProcessList in a tight loop, which results in the allocation of a new buffer that is returned to their process via condrv's ioctl interface. I don't think there's anything we can do about that, other than stopping them from doing so."
>>
>> Following this response, I am attempting to bring this issue to the Cygwin developer team's awareness for a possible resolution. Please see https://github.com/microsoft/terminal/issues/18264 for a detailed discussion and accompanying video demonstration of the page-fault counter.
> 
> Thanks for the report.
> However, I cannot reproduce your problem.
> Which cygwin version do you use?
> 

I am using Cygwin version 3.5.4.

FWIW, the GitHub issue at 
https://github.com/microsoft/terminal/issues/18264 has just been updated 
with the following comment by lhecker:

I figured out why this happens: When we read from the console pipe, we 
currently need to provide a buffer where the message is copied into, 
similar to ReadFile. In order to slightly amortize the cost of 
allocating a buffer we reuse the previous allocation unless the previous 
one was >16KiB and the next one is less than half the size. This avoids 
issues where a huge 100MB buffer never gets freed.
This however does not work well for this usage of the API, because the 
GetConsoleProcessList call needs 128KiB and the PeekConsoleInput call 
only needs 20 (?) bytes. This causes the buffer to be repeatedly freed 
and reallocated.

IMO there's 3 things we should do:

    * cygwin should avoid making "large" data requests if there's no 
reason for that. That DWORD buf[65536]; buffer is 128KiB large and in 
practice maybe 16 byte of that will ever be used. It could improve the 
code by only increasing the request size if the return value is equal to 
the buffer size.

    * We should increase the cutoff size from 16KiB to 128KiB, because 
that's the size of the cat chunk size anyway.

    * We should switch to a cheap allocator for processing console 
messages in the future, because malloc is a poor fit for it. But this 
doesn't fix the issue for large allocations, since we still need to be 
conservative about our resident set size. This means that even with a 
custom allocator, cygwin's usage pattern may still be suboptimal.


-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple