X-Recipient:	archive-cygwin AT delorie DOT com
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.3.2 sourceware.org 694283890405
Authentication-Results:	sourceware.org; dmarc=none (p=none dis=none)
	header.from=SystematicSw.ab.ca
Authentication-Results:	sourceware.org;
	spf=none smtp.mailfrom=brian DOT inglis AT systematicsw DOT ab DOT ca
X-Authority-Analysis:	v=2.3 cv=LKf9vKe9 c=1 sm=1 tr=0
	a=kiZT5GMN3KAWqtYcXc+/4Q==:117 a=kiZT5GMN3KAWqtYcXc+/4Q==:17
	a=IkcTkHD0fZMA:10 a=RZ24vCjvlsqmbDxLIRQA:9 a=sFpdAXOpkQbpkCxh:21
	a=9iGHbNHWrO7Q1_wU:21 a=QEXdDO2ut3YA:10
Subject:	Re: [cygwin] DD bug fails to wipe last 48 sectors of a disk
To:	cygwin AT cygwin DOT com
References:	<DB7PR01MB5193A18F1D947ED4C276CD25D5980 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
	<1d1801d64677$bea56050$3bf020f0$@pdinc.us>
	<DB7PR01MB5193CC1C7630FB13B81B9DBAD5980 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
	<60bf1507-4edb-a03f-ec14-07e1ab7f0d94 AT cs DOT umass DOT edu>
	<DB7PR01MB519396195ADB3E3E1217B22BD5940 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
	<CAKNeuBqZpacuUwTD6U+kKPGOO052VYhpRnttm=HYr7m+OT7tiA AT mail DOT gmail DOT com>
	<DB7PR01MB519371C6CAE2D3CFFEF41DCAD5940 AT DB7PR01MB5193 DOT eurprd01 DOT prod DOT exchangelabs DOT com>
	<1814912576 DOT 20200624132126 AT yandex DOT ru>
	<5c9c5b44-c8a6-a075-705e-1761533cc966 AT t-online DOT de>
	<00f601d64d74$959826d0$c0c87470$@pdinc.us>
From:	Brian Inglis <Brian DOT Inglis AT SystematicSw DOT ab DOT ca>
Autocrypt:	addr=Brian DOT Inglis AT SystematicSw DOT ab DOT ca; prefer-encrypt=mutual;
	keydata=
	mDMEXopx8xYJKwYBBAHaRw8BAQdAnCK0qv/xwUCCZQoA9BHRYpstERrspfT0NkUWQVuoePa0
	LkJyaWFuIEluZ2xpcyA8QnJpYW4uSW5nbGlzQFN5c3RlbWF0aWNTdy5hYi5jYT6IlgQTFggA
	PhYhBMM5/lbU970GBS2bZB62lxu92I8YBQJeinHzAhsDBQkJZgGABQsJCAcCBhUKCQgLAgQW
	AgMBAh4BAheAAAoJEB62lxu92I8Y0ioBAI8xrggNxziAVmr+Xm6nnyjoujMqWcq3oEhlYGAO
	WacZAQDFtdDx2koSVSoOmfaOyRTbIWSf9/Cjai29060fsmdsDLg4BF6KcfMSCisGAQQBl1UB
	BQEBB0Awv8kHI2PaEgViDqzbnoe8B9KMHoBZLS92HdC7ZPh8HQMBCAeIfgQYFggAJhYhBMM5
	/lbU970GBS2bZB62lxu92I8YBQJeinHzAhsMBQkJZgGAAAoJEB62lxu92I8YZwUBAJw/74rF
	IyaSsGI7ewCdCy88Lce/kdwX7zGwid+f8NZ3AQC/ezTFFi5obXnyMxZJN464nPXiggtT9gN5
	RSyTY8X+AQ==
Organization:	Systematic Software
Message-ID:	<9614fca7-4006-06f2-9956-b30dd14cc49e@SystematicSw.ab.ca>
Date:	Sun, 28 Jun 2020 14:28:04 -0600
User-Agent:	Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
	Thunderbird/68.9.0
MIME-Version:	1.0
In-Reply-To:	<00f601d64d74$959826d0$c0c87470$@pdinc.us>
X-CMAE-Envelope:	MS4wfFOYttZXB+7fJ6vVznJ8vPlo+1g9QtD4j7mZJSDs0HcO3jLRS3wzx9H5r3yxEGbRGOWyuhqckbfRkHe5NgFzaprSCZlpxenLZ21LZ9s6j/JrzD4+kJXY
	ZTs08utnFodgbZl1oDP/i5Tx/hog6M+jd6gfmZr/MOHgXTQS+oXdLvDW4IJSC1+M8NZp0RaOcJOEmg==
X-Spam-Status:	No, score=-10.0 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS,
	KAM_LAZY_DOMAIN_SECURITY, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,
	RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE,
	TXREP autolearn=no autolearn_force=no version=3.4.2
X-Spam-Checker-Version:	SpamAssassin 3.4.2 (2018-09-13) on
	server2.sourceware.org
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.29
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<http://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<http://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
Reply-To:	cygwin AT cygwin DOT com
Errors-To:	cygwin-bounces AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 05SKSbTe001194

On 2020-06-28 11:50, Jason Pyeron wrote:
> On Sunday, June 28, 2020 10:35 AM, Christian Franke wrote:
>> Andrey Repin wrote:
>>>> dd if=/dev/zero of=/dev/sda iflag=fullblock bs=4M status=progress
>> The root of the problem is that the Windows WriteFile() function
>> apparently does not support truncated writes at EOM. If seek_position +
>> write_size > disk_size, then WriteFile() does nothing and returns an error.
>>> oflag=direct
>>> Although I'm unsure how Cygwin/Windows handles it. But without this flag, the
>>> write is cached, and the problem may be outside dd, or even Cygwin.
>> If 'oflag=direct' is used, dd passes O_DIRECT flag to open() call of
>> output file. Cygwin's open() function then passes
>> FILE_NO_INTERMEDIATE_BUFFERING to NtCreateFile() and the write()
>> function calls WriteFile() directly with original address and size.
>> Without O_DIRECT, Cygwin ensures that address and size passed to
>> WriteFile() are both aligned to sector size. All writes are then done
>> through a 64KiB internal buffer.
>> As a consequence, oflag=direct in the above dd command may increase
>> speed but would also let the final 4MiB WriteFile() fail. Without
>> oflag=direct, only the last 64KiB WriteFile() fails.
>> To clear the last sectors of the disk, use an appropriate small block
>> size. I did this several times with Cygwin 'dd seek=... bs=512 ...' to
>> get rid of Intel RST RAID metadata.

It appears that write(2) allows:

"An error return value while performing write() using direct I/O does not mean
the entire write has failed. Partial data may be written and the data at the
file offset on which the write() was attempted should be considered inconsistent."

and write(3p) can return -1 errno EAGAIN to suggest a retry with a smaller
buffer size, in regards to pipes and FIFOs, so this could be allowed on devices:

"Deferred: ret=−1, errno=[EAGAIN]
This  error  indicates that a later request may succeed. It does not indicate
that it shall succeed, even if nbyte≤{PIPE_BUF}, because if no process reads
from the pipe or FIFO, the write never succeeds. An application could usefully
count the number of times [EAGAIN] is caused by a particular value of
nbyte>{PIPE_BUF} and perhaps do later writes with a smaller value, on the
assumption that the effective size of the pipe may have decreased.
Partial and deferred writes are only possible with O_NONBLOCK set."

"There is no exception regarding partial writes when O_NONBLOCK is set. With the
exception of writing to an empty pipe, this volume of POSIX.1‐2008 does not
specify exactly when a partial write is performed since that would require
specifying internal details of the implementation. Every application should be
prepared to handle partial writes when O_NONBLOCK is set and the requested
amount is greater than {PIPE_BUF}, just as every application should be prepared
to handle partial writes on other kinds of file descriptors."

and Cygwin write() could return -1 errno EAGAIN when a write at the end of a
device fails, and the application (dd) should be prepared to retry with a
smaller blocksize (half while even, then round up to even, as I/Os are
often/always? required to be even or binary multiple sizes) e.g. if the last
block below returned an error 39072726/2 -> 19536363 so +1 -> 19536364, or a
larger binary multiple such as 512, 64K, 1M using integer arithmetic: ((n - m +
1)/m + 1)*m, e.g in the trivial case above: (19536363 - 2 + 1)/2 + 1)*2 ->
19536364, either until a write succeeded, or the size was small enough e.g. 512,
that the result should be returned as a failure.

> The reason I have never encountered this is because I use a block size which 
> is the largest practical GCD of the drive size and 512 bytes (typically
> between 32 MB and 64 MB).
> 
> E.g. I have a drive that is 160,041,885,696 bytes, which divides 312,581,808
> times evenly into 512. I would use a block size of ‭39,072,726‬ bytes, which
> gives 4,096 blocks to write.
I am somewhat surprised that Unix and Cygwin support non-blocksize aligned
writes, but it seems you have used this successfully, correct?

A useful way to approach this uses coreutils bc and factor:

	$ bc <<< 160041885696/512	# blocks
	312581808
	$ factor 312581808		# prime factors
	312581808: 2 2 2 2 3 3 3 7 11 9397

checking and splitting off the non-binary factors, we have to pick some binary
multiple of the non-binary factor product in a reasonable range:

	$ bc <<< 9397*11*7*3^3*2^4\;9397*11*7*3^3\;2^4\;9397*11*7*3^3*2
	312581808
	19536363
	16
	39072726

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -