X-Recipient: archive-cygwin AT delorie DOT com
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org A552B3857012
Authentication-Results: sourceware.org; dmarc=none (p=none dis=none)
 header.from=SystematicSw.ab.ca
Authentication-Results: sourceware.org;
 spf=none smtp.mailfrom=brian DOT inglis AT systematicsw DOT ab DOT ca
X-Authority-Analysis: v=2.4 cv=INe8tijG c=1 sm=1 tr=0 ts=5feb6d62
 a=kiZT5GMN3KAWqtYcXc+/4Q==:117 a=kiZT5GMN3KAWqtYcXc+/4Q==:17
 a=IkcTkHD0fZMA:10 a=s17s8D9eeDpB4MjGXPUA:9 a=QEXdDO2ut3YA:10
From: Brian Inglis <Brian DOT Inglis AT SystematicSw DOT ab DOT ca>
Subject: Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
To: cygwin AT cygwin DOT com
References: <DB7PR01MB5193A18F1D947ED4C276CD25D5980 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
 <1d1801d64677$bea56050$3bf020f0$@pdinc.us>
 <DB7PR01MB5193CC1C7630FB13B81B9DBAD5980 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
 <60bf1507-4edb-a03f-ec14-07e1ab7f0d94 AT cs DOT umass DOT edu>
 <DB7PR01MB519396195ADB3E3E1217B22BD5940 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
 <1b13fde4-0834-cd8b-0673-c2b14bbaa372 AT SystematicSw DOT ab DOT ca>
 <DB7PR01MB5193BA73A9E8E08912AEBADED5D80 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
 <121001d6dd8c$1dc8b0e0$595a12a0$@pdinc.us>
Organization: Systematic Software
Message-ID: <8cc9cad7-a5e2-3329-5bfc-14b5650489d9@SystematicSw.ab.ca>
Date: Tue, 29 Dec 2020 10:54:41 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.6.0
MIME-Version: 1.0
In-Reply-To: <121001d6dd8c$1dc8b0e0$595a12a0$@pdinc.us>
Content-Language: en-CA
X-CMAE-Envelope: MS4xfABjcNE53sXrYLy6+xx5wBIU57doLttg7C6+Q2WOqcHaH0FSoa8zxF8RzZZAfY6iOMiFSNDTaYq7TDUe1ELn6ql96U8abhcFg0MulvJtcdHlrpXmVKwA
 XFgjxCmnT0ZK/qW3320PBaUlvnSHpnVxqnf+WD4yGDh7iddILx69bu/gIKIaEKsDT9+XBX8YN/disNTIspGk2IC39I+CJ715AoI=
X-Spam-Status: No, score=-6.5 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS,
 KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,
 RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE,
 TXREP autolearn=ham autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on
 server2.sourceware.org
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Reply-To: cygwin AT cygwin DOT com
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: cygwin-bounces AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces AT cygwin DOT com>

On 2020-12-28 19:41, Jason Pyeron wrote:
> On Monday, December 28, 2020 7:46 PM, Hashim Aziz wrote:
>> On 23 June 2020 8:33 PM, Brian Inglis wrote:
>>> I don't have the facilities to test, and there appear to be *NO* Windows 
>>> documentation details on error condition handling, but my suspicion is
>>> that Unix reads and writes fail only *AFTER* reading or writing at the
>>> end of the device, but Windows reads and writes extents may be checked
>>> and failed *BEFORE* reading or writing any data near the end of the
>>> device.
>>> If the actual Windows error code returned is generic, Cygwin would need
>>> to pre-check the device size as Windows does, and reduce read and write
>>> sizes to the allowed maximum at the end of the device.

>> That's very helpful, thank you. Do you know if any more work has been done 
>> to attempt to fix this bug, and whether it's likely to be fixed anytime
>> soon? It's crazy that such a commonly used command leaves so much data
>> unwiped unbeknown to so many users, it's a very serious security hole and
>> the sooner it can be fixed the better.

> Have you tried iflag=fullblock ? This causes special handling.

>> I didn't previously see this email, but the point is that this is a bug - 
>> dd should not require first making calculations based on the size of each
>> drive or using the smallest possible block size (and hence taking a
>> ridiculous amount of time) in order to do what

> Do you have any metrics that it is faster, by any meaningful amount? If so I 
> would be very interested in mitigating it, but I suspect not the actual
> case.

>> it's meant to do. It should always wipe the last sector of the drive
>> regardless, just as it does on other UNIX-like systems. This is why this
>> behaviour is a bug that needs to be fixed.

> This does not appear to be a bug, but user error. Per the DD source "Some
> devices require alignment on a sector or page boundary"
> DD has never "dealt with error handling" except when conversion were in play.
> When no conversions are in play it

>              {
>                /* Write any partial block. */
>                exit_status = EXIT_FAILURE;
>                break;
>              }

> On windows the block devices require respecting block device boundaries, any 
> change would be an upstream patch - not a Cygwin patch.

Your dd output appears to be ambiguous, relative to your claim that the last 48 
sectors are not written, and may appear to indicate that all sectors of the 
drive may have been written, assuming that you mean 512 byte sectors.

> 1000182120448 bytes (1.0 TB, 931 GiB) copied, 8284 s, 121 MB/s

1000182120448 == 238462*4*1024^2

> dd: error writing '/dev/sda': No space left on device
> 238468+0 records in

1000207286272 == 238468*4*1024^2

> 238467+0 records out

1000203091968 == 238467*4*1024^2

> 1000204861440 bytes (1.0 TB, 932 GiB) copied, 8284.89 s, 121 MB/s

1000204861440 == 238467*4*1024^2 + 27*64*1024

None of these numbers +/-48*512 bytes, which have odd factors, make a lot of 
sense as a disk size.

Could you please state explicitly, how many bytes/sectors/blocks/pages/clusters 
of what size you expect to get written, and how many 
bytes/sectors/blocks/pages/clusters of what size are actually written?

If anyone has access to a Linux system which has write access to a Windows drive 
over the network (e.g. Samba, NFS) where this can be reproduced, we can try to 
take this upstream, get their take, suggest an incremental reseek and write half 
buffer size patch, if they agree this is an issue and could be tackled in this 
manner.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple