X-Recipient: archive-cygwin AT delorie DOT com
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org AD9353857C61
Authentication-Results: sourceware.org; dmarc=none (p=none dis=none)
 header.from=SystematicSw.ab.ca
Authentication-Results: sourceware.org;
 spf=none smtp.mailfrom=brian DOT inglis AT systematicsw DOT ab DOT ca
X-Authority-Analysis: v=2.4 cv=Yq/K+6UX c=1 sm=1 tr=0 ts=603756de
 a=T+ovY1NZ+FAi/xYICV7Bgg==:117 a=T+ovY1NZ+FAi/xYICV7Bgg==:17
 a=IkcTkHD0fZMA:10 a=sBsfJAgXvrhmUhQbWtwA:9 a=QEXdDO2ut3YA:10
To: cygwin AT cygwin DOT com
References: <DB7PR02MB39960C1E938B80D9B98CF40DE79F9 AT DB7PR02MB3996 DOT eurprd02 DOT prod DOT outlook DOT com>
 <87eeh5b39n DOT fsf AT Otto DOT invalid>
 <49e01de1-f7de-5d60-2996-3c9afb834396 AT cs DOT umass DOT edu>
 <20210224224149 DOT GM2979 AT dimstar DOT local DOT net>
 <7bdae030-12c7-b2fa-9386-5e279382637d AT SystematicSw DOT ab DOT ca>
From: Brian Inglis <Brian DOT Inglis AT SystematicSw DOT ab DOT ca>
Organization: Systematic Software
Subject: Re: Tee and file redirections are very slow to write anything.
Message-ID: <c0fed8e9-abfd-d159-20c5-3d6d0db956c4@SystematicSw.ab.ca>
Date: Thu, 25 Feb 2021 00:50:53 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.1
MIME-Version: 1.0
In-Reply-To: <7bdae030-12c7-b2fa-9386-5e279382637d@SystematicSw.ab.ca>
Content-Language: en-CA
X-CMAE-Envelope: MS4xfBy8BTJkZutZC4lmu0HKhButy2KqmyUfAremrJ+L12AN6ogPcJgsBeNLFGSbtdKVtbM7iSutyFoc2c9q+HGtcKaQG7HsnVAT57adZf6dmGcVL0g+iTI5
 NeitAfCC1GqJbnj/5p3C9HTrGNroNUkiTBIQB4yBrrIuTTnwdRZebfpgxi0dkkpi4pQU12Q2Szku3Y3Dq7gi0lGzYXWYNbIoU70=
X-Spam-Status: No, score=1.9 required=5.0 tests=BAYES_00, BODY_8BITS,
 KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A,
 RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,
 RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE,
 TXREP autolearn=no autolearn_force=no version=3.4.2
X-Spam-Level: *
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Reply-To: cygwin AT cygwin DOT com
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 11P7p0L8004912

On 2021-02-24 16:50, Brian Inglis wrote:
> On 2021-02-24 15:41, Duncan Roe wrote:
>> On Wed, Feb 24, 2021 at 04:58:24PM -0500, Eliot Moss wrote:
>>> On 2/24/2021 3:48 PM, ASSI wrote:
>>>> Hamish McIntyre-Bhatty via Cygwin writes:
>>>>> I found recently when trying to save output from a script for later
>>>>> inspection that "tee" and file redirections seem to have massive
>>>>> delays when run in Cygwin - usually nothing is written to file or
>>>>> stdout until after the command has finished - not very helpful.
>>>>
>>>> You will want to switch from fully buffered to line-buffered or even
>>>> unbuffered output.
> 
>>> And this does not have to do with Cygwin.  The same happens on Linux.
>>> The default is that terminal I/O is unbuffered while other stream are
>>> buffered.  Pipes come under "other streams".  One can make programmatic
>>> changes to get around this, but most programs won't override the
>>> default behavior on their own ...
> 
>> The (Linux) default is that terminal I/O is *line* buffered
>>
>> The man page for tee doesn't show an option to change buffering, while that for
>> grep does.
> 
> I believe the default for both Cygwin and Linux is 64KB pipe buffer, so if you 
> want to see smaller chunks as they are generated, you need to add some utility 
> that may allow you to change that e.g.
> 
>      $ tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq
> 
> but read the disclaimers on the stdbuf and grep man pages, which is why it is 
> not done more, especially under Cygwin where Windows adds its own performance 
> penalties.
> Some utilities may use read(2/3p), write(2/3p), or mmap(3) if they can and don't 
> care about text or lines, for more efficient access to disk files, rather than 
> buffered stream I/O functions.

 From what I have been able to find, Cygwin <stdio.h> BUFSIZ is only 1K, 
compared to Linux 8K, and Cygwin internal 64K, and that is used in many places 
in coreutils like tee, which will slow everything down by a factor of at least 8 
plus increased overhead.

Suggest <stdio.h> BUFSIZ be bumped to at least Linux value of 8K, if not 64K.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple