X-Recipient:	archive-cygwin AT delorie DOT com
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.3.2 sourceware.org 254D83857C5F
Authentication-Results:	sourceware.org;
	dmarc=none (p=none dis=none) header.from=pdinc.us
Authentication-Results:	sourceware.org; spf=pass smtp.mailfrom=jpyeron AT pdinc DOT us
DKIM-Filter:	OpenDKIM Filter v2.11.0 mail1.pdinc.us 0BU1Z6qm014167
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=pdinc.us; s=default;
	t=1609292106; bh=m2Avb9m5PK1kuPf6lmHbHbdT99UAFahLtJDFQhUwgto=;
	h=From:To:References:In-Reply-To:Subject:Date:From;
	b=Z0z+CW1QLnWJIBnKH0TVgrQ4lMidIxsZsFXdOWxB8GjPsK1u09ECjJbB1oeIK1Aat
	dQsUymsXYppMx0Ez7ZZuu1++6PiKMTOQjPDUVMiJDHow4ZZ0HomlkCojF7scieL3My
	0CSOL6jP6GfZI0ZWQncwSkAMundEfB+bC1b7JEDHpasE99cEVbZwk1hHi1BdSiKZWE
	TBJKQWBOWASNrq75E7ABn3/vYYBcHAW+toW749CtIPb9FcIZGWFFP69Qwc0b3Gmod0
	oYwLO+CioaLJsljH5lZTNicHNvnplPp5+n7v1JA5qd5KxHPOVaaNYlFhbvZpA56r9k
	XXiYaHqHuOXaQ==
From:	"Jason Pyeron" <jpyeron AT pdinc DOT us>
To:	<cygwin AT cygwin DOT com>
References:	<DB7PR01MB5193A18F1D947ED4C276CD25D5980 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
	<1d1801d64677$bea56050$3bf020f0$@pdinc.us>
	<DB7PR01MB5193CC1C7630FB13B81B9DBAD5980 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
	<60bf1507-4edb-a03f-ec14-07e1ab7f0d94 AT cs DOT umass DOT edu>
	<DB7PR01MB519396195ADB3E3E1217B22BD5940 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
	<1b13fde4-0834-cd8b-0673-c2b14bbaa372 AT SystematicSw DOT ab DOT ca>
	<DB7PR01MB5193BA73A9E8E08912AEBADED5D80 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
	<121001d6dd8c$1dc8b0e0$595a12a0$@pdinc.us>
	<8cc9cad7-a5e2-3329-5bfc-14b5650489d9 AT SystematicSw DOT ab DOT ca>
In-Reply-To:	<8cc9cad7-a5e2-3329-5bfc-14b5650489d9@SystematicSw.ab.ca>
Subject:	RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
Date:	Tue, 29 Dec 2020 20:35:16 -0500
Organization:	PD Inc
Message-ID:	<08f401d6de4c$073fa2a0$15bee7e0$@pdinc.us>
MIME-Version:	1.0
X-Mailer:	Microsoft Outlook 16.0
Thread-Index:	AQGscau3vmYRj+s+e2PTQd4MF23JZQNbIR/1ATPrTR8BvLqaegH1Mf0MAb1HszEA12M7fAF/W2ZqAkNU+mqp7utLEA==
X-Spam-Status:	No, score=-2.4 required=5.0 tests=BAYES_00, DKIM_SIGNED,
	DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, KAM_INFOUSMEBIZ, SPF_HELO_PASS,
	SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.2
X-Spam-Checker-Version:	SpamAssassin 3.4.2 (2018-09-13) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Sender:	"Cygwin" <cygwin-bounces AT cygwin DOT com>

> -----Original Message-----
> From: Brian Inglis
> Sent: Tuesday, December 29, 2020 12:55 PM
> 
> On 2020-12-28 19:41, Jason Pyeron wrote:
> > On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote:
> >> On 23 June 2020 8:33 PM, Brian Inglis wrote:
> >>> I don't have the facilities to test, and there appear to be *NO* Windows
> >>> documentation details on error condition handling, but my suspicion is
> >>> that Unix reads and writes fail only *AFTER* reading or writing at the
> >>> end of the device, but Windows reads and writes extents may be checked
> >>> and failed *BEFORE* reading or writing any data near the end of the
> >>> device.
> >>> If the actual Windows error code returned is generic, Cygwin would need
> >>> to pre-check the device size as Windows does, and reduce read and write
> >>> sizes to the allowed maximum at the end of the device.
> 
> >> That's very helpful, thank you. Do you know if any more work has been done
> >> to attempt to fix this bug, and whether it's likely to be fixed anytime
> >> soon? It's crazy that such a commonly used command leaves so much data
> >> unwiped unbeknown to so many users, it's a very serious security hole and
> >> the sooner it can be fixed the better.
> 
> > Have you tried iflag=fullblock ? This causes special handling.
> 
> >> I didn't previously see this email, but the point is that this is a bug -
> >> dd should not require first making calculations based on the size of each
> >> drive or using the smallest possible block size (and hence taking a
> >> ridiculous amount of time) in order to do what
> 
> > Do you have any metrics that it is faster, by any meaningful amount? If so I
> > would be very interested in mitigating it, but I suspect not the actual
> > case.
> 
> >> it's meant to do. It should always wipe the last sector of the drive
> >> regardless, just as it does on other UNIX-like systems. This is why this
> >> behaviour is a bug that needs to be fixed.
> 
> > This does not appear to be a bug, but user error. Per the DD source "Some
> > devices require alignment on a sector or page boundary"
> > DD has never "dealt with error handling" except when conversion were in play.
> > When no conversions are in play it
> 
> >              {
> >                /* Write any partial block. */
> >                exit_status = EXIT_FAILURE;
> >                break;
> >              }
> 
> > On windows the block devices require respecting block device boundaries, any
> > change would be an upstream patch - not a Cygwin patch.
> 
> Your dd output appears to be ambiguous, relative to your claim that the last 48
> sectors are not written, and may appear to indicate that all sectors of the
> drive may have been written, assuming that you mean 512 byte sectors.
> 
> > 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s
> 
> 1000182120448 == 238462*4*1024^2
> 
> > dd: error writing '/dev/sda': No space left on device
> > 238468+0 records in
> 
> 1000207286272 == 238468*4*1024^2
> 
> > 238467+0 records out
> 
> 1000203091968 == 238467*4*1024^2
> 
> > 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s
> 
> 1000204861440 == 238467*4*1024^2 + 27*64*1024
> 
> None of these numbers +/-48*512 bytes, which have odd factors, make a lot of
> sense as a disk size.
> 
> Could you please state explicitly, how many bytes/sectors/blocks/pages/clusters
> of what size you expect to get written, and how many
> bytes/sectors/blocks/pages/clusters of what size are actually written?
> 
> If anyone has access to a Linux system which has write access to a Windows drive
> over the network (e.g. Samba, NFS) where this can be reproduced, we can try to
> take this upstream, get their take, suggest an incremental reseek and write half
> buffer size patch, if they agree this is an issue and could be tackled in this
> manner.

I do, and I am even willing to spend a few dollars in testing this too.

BUT, I want a clear test plan first, with clearly articulated issues BECAUSE I do not believe there are any issues actually existing.

The best I can glean from the thread is 

1. Cygwin is agedly breaking dd, but is also broken in the windows native dd [1]
2. Using "correct" (as I have previously defined it e.g. [4]) values is both
2.1. too slow on the write [2]
2.2. inappropriately complicated of a process for the user
3. It has been well investigated 
3.1. with a root cause was identified as a windows OS issue [3,5,6]
3.2. with a mitigation [4,5]

I am happy to spend time and money on 2.1.
I will not spend my time on dealing with 2.2 - except providing an example and documentation update for upstream.

Respectfully,

Jason Pyeron

1: > From: Hashim Aziz
> Sent: Saturday, June 20, 2020 1:31 PM
> To: The Cygwin Mailing List
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <DB7PR01MB5193CC1C7630FB13B81B9DBAD5980 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>

2: > From: Hashim Aziz
> Sent: Tuesday, June 23, 2020 11:29 AM
> To: cygwin 
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <DB7PR01MB519396195ADB3E3E1217B22BD5940 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>

3: > From: Christian Franke
> Sent: Sunday, June 28, 2020 10:35 AM
> To: cygwin AT cygwin DOT com
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <5c9c5b44-c8a6-a075-705e-1761533cc966 AT t-online DOT de>

4: > From: Jason Pyeron
> Sent: Sunday, June 28, 2020 1:50 PM
> To: cygwin AT cygwin DOT com
> Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <00f601d64d74$959826d0$c0c87470$@pdinc.us>

5: > From: Brian Inglis
> Sent: Sunday, June 28, 2020 4:28 PM
> To: cygwin AT cygwin DOT com
> Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <9614fca7-4006-06f2-9956-b30dd14cc49e AT SystematicSw DOT ab DOT ca>

6: > From: Jason Pyeron
> Sent: Monday, December 28, 2020 9:42 PM
> To: cygwin AT cygwin DOT com
> Subject: RE: [cygwin] DD bug fails to wipe last 48 sectors of a disk
> Message-ID: <121001d6dd8c$1dc8b0e0$595a12a0$@pdinc.us>


--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -