X-Recipient:	archive-cygwin AT delorie DOT com
DomainKey-Signature:	a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:from:to:cc:subject:message-id:in-reply-to
	:references:mime-version:content-type:content-transfer-encoding;
	q=dns; s=default; b=ZXK61wbnPEVRS320u48+xaSG5ZpX7hj9wdrZ9Iox18e
	FEs/he2W/wOP6+qkh5WAdZVcP1bxluRwjMU6IVMjxOWnMLxE5B3OO6zWDnZXDaba
	2GILN5wqezdm71oOZu7dded42ZMa+/zjlL1fVPuAFN0F3cNzzIcY8JoP6LTS+30o
	=
DKIM-Signature:	v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id
	:list-unsubscribe:list-subscribe:list-archive:list-post
	:list-help:sender:date:from:to:cc:subject:message-id:in-reply-to
	:references:mime-version:content-type:content-transfer-encoding;
	s=default; bh=bU4kUdR3RAlxRfq36OkUDJcuBVI=; b=LIWG8l84rPkCWsrLL
	xEc36BgVMQNifphdzf+6e2LregIg3bZFJATqhlE4BHRl0qsN9JXCNf1iEf2msdEx
	MBAEuUWdrxHS/nWFzr4E0S4FMKLguDgLwPTo9+povbvHMOdLkxKMDdaRoC9RTcyT
	PhpcRPZNRUlgMWm3MEkZWCm/t0=
Mailing-List:	contact cygwin-help AT cygwin DOT com; run by ezmlm
List-Id:	<cygwin.cygwin.com>
List-Subscribe:	<mailto:cygwin-subscribe AT cygwin DOT com>
List-Archive:	<http://sourceware.org/ml/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-help AT cygwin DOT com>, <http://sourceware.org/ml/#faqs>
Sender:	cygwin-owner AT cygwin DOT com
Mail-Followup-To:	cygwin AT cygwin DOT com
Delivered-To:	mailing list cygwin AT cygwin DOT com
Authentication-Results:	sourceware.org; auth=none
X-Virus-Found:	No
X-Spam-SWARE-Status:	No, score=1.6 required=5.0 tests=BAYES_50,RDNS_NONE autolearn=no version=3.3.2
X-HELO:	mailscanner01.zoner.fi
Date:	Sun, 1 Dec 2013 15:24:15 +0200
From:	Lasse Collin <lasse DOT collin AT tukaani DOT org>
To:	Denis Excoffier <cygwin AT Denis-Excoffier DOT org>
Cc:	Tom Honermann <thonermann AT coverity DOT com>,
	Cygwin Mailing List <cygwin AT cygwin DOT com>
Subject:	Re: Antivirus strikes back (probably) (Was: Intermittent failures retrieving process exit codes)
Message-ID:	<20131201152415.64cab6e3@tukaani.org>
In-Reply-To:	<9E1F6369-88DF-4EDB-A752-1340ACBA7564@Denis-Excoffier.org>
References:	<50C2498C DOT 2000003 AT coverity DOT com> <50C276AC DOT 9090301 AT mailme DOT ath DOT cx> <50D401EF DOT 9040705 AT coverity DOT com> <52844B2E DOT 5050902 AT coverity DOT com> <EFA04305-A94A-46F4-BCCE-8FB3ADA45E72 AT Denis-Excoffier DOT org> <20131125215842 DOT 0ff53134 AT tukaani DOT org> <9E1F6369-88DF-4EDB-A752-1340ACBA7564 AT Denis-Excoffier DOT org>
Mime-Version:	1.0
X-Antivirus-Scanner:	Clean mail though you should still use an Antivirus
Received-SPF:	none
X-MIME-Autoconverted:	from quoted-printable to 8bit by delorie.com id rB1DPKYI013347

On 2013-11-26 Denis Excoffier wrote:
> On 2013-11-25 à 21:58 +02:00, Lasse Collin wrote:
> > If you haven't already tried, please compile both 5.1.2alpha and
> > 5.1.3alpha from source while keeping everything else unchanged, and
> > see if the bug really only occurs with 5.1.3alpha.
> Already done. I did some strace-ing, and since i’m not so fluent with
> the result, i’ll send it there in a while (when i’m back on cygwin)
> if someone is interested. But the bug (contrary to what i said
> before) also _sometimes_ occurs with 5.1.2alpha or 5.0.5 and this
> makes me think now that:
> 
> a) my antivirus-anti-intrusion-whatever-software (that i can’t remove
> of course) creates some kind of "background noise" where a certain
> percentage of such ‘tar xf —use-compress-program’ commands will
> always fail
> 
> b) nevertheless, xz-5.1.3alpha (with its new file I/O code etc.)
> triggers some untypical configuration inside the antivirus that
> increases drastically the percentage, making the failure almost
> certain for some files.
> 
> It is not extraordinary that i cannot observe the failure on XP since
> i do not have this particular antivirus on XP.

OK, so the new I/O code in xz probably isn't the problem even if it may
affect how easily the actual problem gets triggered.

[...]
> When it fails (usually or pathologically), the last file of the
> archive gets truncated (see above), and _this_ is strange from an
> antivirus behaviour. After all, perhaps some flush() or similar is
> missing inside 5.1.3alpha.

xz uses write() which uses a file descriptor argument, so there is
nothing to flush separately. xz just has to write() everything.

When used with tar, xz writes to standard output (FILENO_STDOUT) which
with tar is a pipe. When xz finishes, it closes its end (the writer end)
of the pipe.

With xz 5.1.3alpha, O_NONBLOCK flag is set for FILENO_STDIN and
FILENO_STDOUT if the flag wasn't already set. If xz set the flag, it
will unset it before closing the file descriptor. The setting and
unsetting can be seen in the trace you sent and it seems to work
correctly. I don't have a guess if these fcntl() calls might cause the
difference between 5.1.3alpha and other versions, but it doesn't sound
too important since the bug occurs in some form with all versions.

From the trace file it seems that the last write() from xz gets lost.
xz first makes 173 writes of 8192 bytes and then one 6144-byte write,
totalling 1,423,360 bytes. tar gets 1,417,216 from xz, that is, 6144
bytes too little.

Since things go wrong with old xz versions that don't use non-blocking
I/O, I would expect you to see similar issues with other compressors
too. Maybe it would be worth testing with gzip and bzip2 in the same
way you did with xz 5.0.5.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

- Raw text -