X-Recipient: archive-cygwin AT delorie DOT com
DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:content-type
	:content-transfer-encoding:date:from:to:cc:subject:in-reply-to
	:references:message-id; q=dns; s=default; b=bWFN6zLhYvnyYWaLpTJc
	Yx7fYabT01Lrj+FDTEcC+EPmvSG9l0GwWytONr8oBHbxNKACeUszyxzoB0ZrVrtU
	YBH5PYf5NqasmdSCWuDrEOijZ2H7X/hzoIDybaPIaGwjTetBe4gRMzVVpsgVAMUb
	lXjv1/CFHE1ULfaXfKhAu6A=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:mime-version:content-type
	:content-transfer-encoding:date:from:to:cc:subject:in-reply-to
	:references:message-id; s=default; bh=sl8bm34I/clQOIyYyxzCExVZ0R
	k=; b=m8PTIgiIUXcgNoKPUes6C18RApqD17gXjnTabULefba8VrkIZ5MhVQJ7dU
	HEThLzAB+pqWcxbzuGZojzQqwVU6A1NrmMCTT64rizAADesFiHT27jhb+whoPty/
	NnH3NrNA0ALQ4OaJlQ+eqbOaCZBXNbS35WOvk42V6dTTwd7Bw=
Mailing-List: contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id: <cygwin.cygwin.com>
List-Subscribe: <mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive: <http://sourceware.org/ml/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender: cygwin-owner AT cygwin DOT com
Mail-Followup-To: cygwin AT cygwin DOT com
Delivered-To: mailing list cygwin AT cygwin DOT com
Authentication-Results: sourceware.org; auth=none
X-Virus-Found: No
X-Spam-SWARE-Status: No, score=-2.3 required=5.0 tests=AWL,BAYES_00,KB_WAM_FROM_NAME_SINGLEWORD,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 spammy=1280, preliminary, Hx-languages-length:1700, H*F:D*nl
X-HELO: lb1-smtp-cloud9.xs4all.net
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Date: Tue, 28 Nov 2017 18:12:02 +0100
From: Houder <houder AT xs4all DOT nl>
To: cygwin AT cygwin DOT com
Cc: djndnbvg AT gmail DOT com
Subject: Re: sort utility goes berzerk (x86_64)
In-Reply-To: <CAJ1FpuMVeneyhKmv0d8J_EPXjfrJ09FpVt=ojMEeJPRLyKes=A@mail.gmail.com>
References: <177e9f9fe723eadd4a8e41e5e032d56d AT smtp-cloud9 DOT xs4all DOT net> <eeee637eb6e6b01641d33930e178cbae AT xs4all DOT nl> <20171128090329 DOT GN547 AT calimero DOT vinschen DOT de> <09fc4a030cc7abb5cabf07672e683d70 AT xs4all DOT nl> <CAJ1FpuMVeneyhKmv0d8J_EPXjfrJ09FpVt=ojMEeJPRLyKes=A AT mail DOT gmail DOT com>
Message-ID: <944c377d08d67cda987edfcf4389ee89@xs4all.nl>
X-Sender: houder AT xs4all DOT nl
User-Agent: XS4ALL Webmail
X-CMAE-Envelope: MS4wfDzVLkoFEU304BGARK0HKKBpnK42vgnqdqVwwIl+U0Peqr0z4GxvLNzUGFrHyE2jyknGMqlrZiIypGUnSGlMvT00ICLAwH1rRwK6Yo/zKeFpClXjbsw1 sQp926tmiyqWsz9uHbZ81ulnZT1CDaNH3Z5c0ZUUVzyfpXJJzORRPRAo3cxiOR07f+MFOB1C9mpuKWMcqY73f4ABK46ToliBX8l69T7Yem9w/bwA3Yip7ZnE s5w6mh9YM7gH7Qou8+cyyA==
X-IsSubscribed: yes

On 2017-11-28 14:37, Doug Henderson wrote:

>>>> On 2017-11-25 14:23, Houder wrote:
>>>> > Hi,
>>>> >
>>>> > Anyone seeing this as well? sort goes berzerk on my system when piped
>>>> > into
>>>> > head (or less) when it is fed with a 'specially prepared' input file.
>>>> >
>>>> >  - only happens on x86_64
>>>> >  - does not happen for 'LC_COLLATE=C sort tt | head'
>>>> >
>>>> > 'specially prepared' input file? (see bottom of post).
>>>> 
>>>> Anyone ** NOT ** seeing this?

> Q4. After you kill a hung sort, are there files left over in /tmp? I
> am wondering if the problem files are too big to do an in-core sort,
> and something goes wrong when it switches to a temp file sort, but not
> for smaller or larger input files where it correctly predicts which
> type of sort it needs to do. Possibly using the "--buffer-size=N"
> option will change the behavior, making smaller input files fail and
> failing ones work.

Preliminary result:

# tt = 8150 lines
sort -S 1024 tt | head # 1024 * 1024 = 1M
sort -S 1152 tt | head # + 128
sort -S 1280 tt | head # + 256 <==== fail (1226?)

# ttt = 5050 lines
sort -S 512 ttt | head # 512 * 1024 = .5M
sort -S 640 ttt | head # + 128
sort -S 768 ttt | head # + 256 <==== fail (760?)

These numbers are correct for --parallel=8 (the eight "cores" that are
in my machine)

Specifying a smaller number of cores, also make the above numbers grow
smaller.

All of the above is about tuning the performance of sort; however it is
not an explanation for the way in which sort fails.

sort should not fail in the way it does; it does not on "Cygwin/x86" on
my machine; it does not on Linux (Fedora 24) (same machine).


Henri

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple