DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org F40E3384640E
Date: Wed, 24 Apr 2024 22:11:52 +0000
To: cygwin AT cygwin DOT com
Subject: native symlinks and non-existent targets
Message-ID: <ZimDqKx1R13FdV2q@mercury.signalpunk.com>
MIME-Version: 1.0
Content-Disposition: inline
From: Christopher Layne via Cygwin <cygwin AT cygwin DOT com>
Reply-To: clayne AT anodized DOT com
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: "Cygwin" <cygwin-bounces+archive-cygwin=delorie DOT com AT cygwin DOT com>

Since I recently sent an email about symlinks and cygdrive mounts, I
figured I'd report another issue that's plagued me over the years and
that I know others have reported in the past: You can't create native
symlinks to non-existent targets and this causes a bunch of issues when
rsyncing directories containing symlinks unless one does a multi-pass
approach which takes special precautions to sync all the non-symlink
contents first and then syncs the symlinks right after (note: this also
has its own problems with links to links).

Example:

clayne AT sv590:/tmp/link-test $ ls -la
total 32
drwxr-xr-x 1 clayne None 0 Apr 24 14:13 .
drwxrwxrwt 1 clayne None 0 Apr 24 14:34 ..
lrwxrwxrwx 7 clayne None 3 Apr 24 14:11 _foo -> foo
-rw-r--r-- 5 clayne None 0 Apr 24 14:11 foo
lrwxrwxrwx 4 clayne None 3 Apr 24 14:13 foo-ln -> foo

clayne AT sv590:/tmp/link-test $ rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/
sending incremental file list
created directory /tmp/link-test-sync-test
./
rsync: [generator] symlink "/tmp/link-test-sync-test/_foo" -> "foo" failed: No such file or directory (2)
rsync: [generator] symlink "/tmp/link-test-sync-test/foo-ln" -> "foo" failed: No such file or directory (2)
foo
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=1/4)

sent 178 bytes  received 89 bytes  534.00 bytes/sec
total size is 6  speedup is 0.02
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]

clayne AT sv590:/tmp/link-test $ rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/
sending incremental file list
_foo -> foo
foo-ln -> foo

sent 134 bytes  received 18 bytes  304.00 bytes/sec
total size is 6  speedup is 0.04

This only works in a straight-forward manner when using non-native symlinks
(which can also _change_ the symlinks such that they're broken/unusable
outside of cygwin):

clayne AT sv590:/tmp/link-test $ rm -rf /tmp/link-test-sync-test
clayne AT sv590:/tmp/link-test $ CYGWIN= rsync -vaHSP /tmp/link-test/ /tmp/link-test-sync-test/
sending incremental file list
created directory /tmp/link-test-sync-test
./
_foo -> foo
foo
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=1/4)
foo-ln -> foo

sent 184 bytes  received 95 bytes  558.00 bytes/sec
total size is 6  speedup is 0.02

Or another very simple test-case not involving rsync:

clayne AT sv590:/tmp/link-test $ ln -s this-does-not-exist some-link
ln: failed to create symbolic link 'some-link': No such file or directory

Relevant sections of an strace for the above:

  102   83054 [main] ln 1379 symlink_info::check: 0x0 = NtCreateFile (\??\C:\cygwin64\tmp\link-test)
  114   83168 [main] ln 1379 symlink_info::check: not a symlink
   91   83259 [main] ln 1379 symlink_info::check: 0 = symlink.check(C:\cygwin64\tmp\link-test, 0x7FFFFB140) (mount_flags 0x30008, path_flags 0x0)
   92   83351 [main] ln 1379 path_conv::check: this->path(C:\cygwin64\tmp\link-test\this-does-not-exist), has_acls(1)
   91   83442 [main] ln 1379 seterrno_from_win_error: /usr/src/debug/cygwin-3.5.3-1/winsup/cygwin/path.cc:2063 windows error 2
   90   83532 [main] ln 1379 geterrno_from_win_error: windows error 2 == errno 2
   83   83615 [main] ln 1379 symlink_worker: -1 = symlink_worker(this-does-not-exist, /tmp/link-test/some-link, 0)

However, you _can_ do this:

clayne AT sv590:/tmp/link-test $ tmp="$(mktemp)" && ln -s "$tmp" some-link && rm -f "$tmp" && ls -la some-link
lrwxrwxrwx 1 clayne None 19 Apr 24 14:58 some-link -> /tmp/tmp.o7xpJxaqig

And here's a case showing where "hard-linked" symlinks work with
non-existent targets:

clayne AT sv590:/tmp/link-test $ find -type l -print0 | rsync -avSHP --files-from=- --from0 --link-dest=/tmp/link-test /tmp/link-test /tmp/link-test-link-dest
building file list ... 
3 files to consider
created directory /tmp/link-test-link-dest

sent 113 bytes  received 59 bytes  344.00 bytes/sec
total size is 6  speedup is 0.03

clayne AT sv590:/tmp/link-test $ ls -l /tmp/link-test-link-dest
total 0
lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 _foo -> foo
lrwxrwxrwx 5 clayne None 3 Apr 24 14:13 foo-ln -> foo

clayne AT sv590:/tmp/link-test $ ls -lai _foo /tmp/link-test-link-dest/_foo
48413695994484407 lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 /tmp/link-test-link-dest/_foo -> foo
48413695994484407 lrwxrwxrwx 8 clayne None 3 Apr 24 14:11 _foo -> foo

This only works because the symlinks are being straight up cloned
and not being newly created on the destination side. This "works" if
one is doing a --link-dest of an entire directory to essentially create
a hard-linked copy but anything outside of that use case will still
require multiple runs.

Based on past threads I've read I believe the issue is actually with
windows not allowing a symlink to be created with a non-existent target,
but I do know windows does not care if you break a link after the fact.
For the non-existent target case, is there any way we could just hack-fix
this or workaround it at the cygwin layer by create a symlink to a temp
file representing the eventual target (I realize this is way easier said
than done) and then immediately removing the temp file? Obviously there
are some atomicity issues here because one might not be able to just
create an in-situ file without creating the directory hierarchy it needs
to live in first. On the other hand it doesn't seem ideal to require an
rsync to run multiple times nor would it make sense to require every
application or utility involved with symlinks to somehow know of this
cygwin/windows specific issue (breaks abstraction). There has got to be
a better way of handling this issue such that it's transparent to the
caller.

-cl

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple