X-Recipient: archive-cygwin AT delorie DOT com
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 9DB043857026
Authentication-Results: sourceware.org; dmarc=none (p=none dis=none)
 header.from=SystematicSw.ab.ca
Authentication-Results: sourceware.org;
 spf=none smtp.mailfrom=brian DOT inglis AT systematicsw DOT ab DOT ca
X-Authority-Analysis: v=2.4 cv=fdJod2cF c=1 sm=1 tr=0 ts=6036e63c
 a=T+ovY1NZ+FAi/xYICV7Bgg==:117 a=T+ovY1NZ+FAi/xYICV7Bgg==:17
 a=IkcTkHD0fZMA:10 a=M5fnu4kae-PSQQVxgqEA:9 a=QEXdDO2ut3YA:10
To: cygwin AT cygwin DOT com
References: <DB7PR02MB39960C1E938B80D9B98CF40DE79F9 AT DB7PR02MB3996 DOT eurprd02 DOT prod DOT outlook DOT com>
 <87eeh5b39n DOT fsf AT Otto DOT invalid>
 <49e01de1-f7de-5d60-2996-3c9afb834396 AT cs DOT umass DOT edu>
 <20210224224149 DOT GM2979 AT dimstar DOT local DOT net>
From: Brian Inglis <Brian DOT Inglis AT SystematicSw DOT ab DOT ca>
Organization: Systematic Software
Subject: Re: Tee and file redirections are very slow to write anything.
Message-ID: <7bdae030-12c7-b2fa-9386-5e279382637d@SystematicSw.ab.ca>
Date: Wed, 24 Feb 2021 16:50:19 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.1
MIME-Version: 1.0
In-Reply-To: <20210224224149.GM2979@dimstar.local.net>
Content-Language: en-CA
X-CMAE-Envelope: MS4xfNEJS+ABlUcAfJKu8+Ln7Q8GdvmR/5vLV3qOYPYoDbjl21dfpiyU6TvQmTvYtOfTt0yRadOL4hmMoJSHj8LUaUtXz3ROL6sqMh2LvRAUJLctDrCT7iPK
 d+VxdaCuKnwvsmaPB/GMB+ryVV3BAUKSaGn6HAlT1bTyDSrm5HBIYjTttbxcAKogtKSYDwIHx18/Y6ob+bUvAT2fJ6xI46gvqSY=
X-Spam-Status: No, score=0.7 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS,
 KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_BARRACUDACENTRAL,
 RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,
 SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Reply-To: cygwin AT cygwin DOT com
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: cygwin-bounces AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces AT cygwin DOT com>

On 2021-02-24 15:41, Duncan Roe wrote:
> On Wed, Feb 24, 2021 at 04:58:24PM -0500, Eliot Moss wrote:
>> On 2/24/2021 3:48 PM, ASSI wrote:
>>> Hamish McIntyre-Bhatty via Cygwin writes:
>>>> I found recently when trying to save output from a script for later
>>>> inspection that "tee" and file redirections seem to have massive
>>>> delays when run in Cygwin - usually nothing is written to file or
>>>> stdout until after the command has finished - not very helpful.
>>>
>>> You will want to switch from fully buffered to line-buffered or even
>>> unbuffered output.

>> And this does not have to do with Cygwin.  The same happens on Linux.
>> The default is that terminal I/O is unbuffered while other stream are
>> buffered.  Pipes come under "other streams".  One can make programmatic
>> changes to get around this, but most programs won't override the
>> default behavior on their own ...

> The (Linux) default is that terminal I/O is *line* buffered
> 
> The man page for tee doesn't show an option to change buffering, while that for
> grep does.

I believe the default for both Cygwin and Linux is 64KB pipe buffer, so if you 
want to see smaller chunks as they are generated, you need to add some utility 
that may allow you to change that e.g.

	$ tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq

but read the disclaimers on the stdbuf and grep man pages, which is why it is 
not done more, especially under Cygwin where Windows adds its own performance 
penalties.
Some utilities may use read(2/3p), write(2/3p), or mmap(3) if they can and don't 
care about text or lines, for more efficient access to disk files, rather than 
buffered stream I/O functions.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple