Repository navigation
Ticket #4720: skip entries vanished during deletion like AppleDouble - #4722
Conversation
|
now i wonder ... given that these are "shadow" files that always go along with "real" files, and we're talking about a situation of actually running on macos (otherwise the case of the files magically disappearing would be irrelevant), why aren't the files hidden by the os? this looks like a shortcoming that would cause all kinds of problems. which brings me to the next point: shouldn't for example move/rename also have related code? that would suggest that this should be handled at the vfs level. but then there is copy. how would copying a file with forks even work on an fs that supports them natively, when they are invisible to regular unix apis? or does the posix layer actually inject the shadow files? i guess this should be properly researched before going any further with this. |
b2363ba to
a179515
Compare
a179515 to
5469c92
Compare
I did some basic research on that, and I can't tell why the designers of macOS made the decision to expose the resource forks like this via the POSIX API. I have discovered that if the files are deleted, then their resource forks are removed by "magic". If the files are copied, the resource forks are apparently taken along (or created anew, it's hard to find out). If the files are moved, then their resource forks are also magically moved with them. So probably ideally one could implement a similar mechanism for moves, as in, first move everything that doesn't look like a fork, and then there is probably nothing else to move. However, in cases of copies and moves the situation is much less annoying than in the case of deletion. If the fork is moved first, apparently it gets re-created. And then the newly created one is taken along when the file is moved. If the file is moved, then its fork is moved, and so there is nothing else to move. For copies, it also strangely seems to work somehow. So, I wouldn't do anything here because it doesn't present a practical problem (apparently), and makes things more complicated. Also, I find it quite annoying that apparently there is no good way to detect forks other than by name, and even then you don't know if it's a valid file or a fork. Anyways, it has been like that for decades. If this problem gets reported, then we could do something about that. A big issue that I have with our whole system is that we are doing things recursively, so it's not possible to build a list first and then just sort it before executing the command. Changing this is way beyond my time and capabilities at the moment. |
5469c92 to
3e940ae
Compare
|
@mc-worker I'm now working a lot with FAT32 volumes on macOS, and this has become a real problem for me. I would like to merge the current solution because it works and doesn't seem to make any issues. |
On Linux, if file |
Hmmm, this is very strange... Thank you for testing! It seems that I am doing something wrong with the return codes in our crazy recursive system :( Do you happen to know why or how to schedule files to be deleted last properly? |
|
Maybe I should return |
Press |
|
I think I've fixed this issue. |
|
/rebase |
48d633d to
e976b7a
Compare
e976b7a to
114f665
Compare
|
the near-identical commit messages tell me that this should be squashed. |
|
So, I've burned my last $5 of Claude cloud credit on this PR, and here are the results: Claude checked XNU kernel sources, and contrary to my previous (unsubstantiated) belief, the problem with our loop is not that The "proper" fix would be to check whether the entries have vanished during the deletion process, and skip such entries, as the goal of having them deleted has been achieved through external influence - be it the kernel playing tricks on This removes the filename heuristic and second recursive pass, at the expense of not explicitly erroring out when deleting files that disappear during deletion through something else other than our explicit action. However, I think this is actually an expected behavior. Incidentally, this needs a fix in two places, but resolves both F8 and F6 issues. The copy part of the story is apparently not affected, and I assume there is nothing to fix there, because whatever magic XNU does, it's somehow designed such as not to break the copying. If not, this is a separate topic either way. For verification, check the following branch:
Claude has written LD_PRELOAD'able P.S. Apparently, the official name for those pesky |
Well, I'm always at a loss as to when you want to have something squashed, and when you want it in separate commits - partly it appears to be changes/volume/problem/explanation driven, and partly arbitrary. The reasoning for two commits is that the first one is fixing F8, and the second one a similar related F6 problem. But I can squash them in one commit, of course - I don't care. |
|
the guiding principle is always "make changes atomic". the question is what to prioritize when different aspects conflict. my general stance is that if the virtually same small change is done in multiple locations, then it should be one commit. this reduces clutter, and makes things more easily searchable in some cases (e.g., with |
On macOS, file systems without native extended attributes support (FAT, exFAT, SMB, ...) keep file metadata in AppleDouble "._" companion files. Unlinking a file makes the kernel unlink its companion, so the companion entry already returned by readdir() cannot be stat'ed anymore. recursive_erase() treated this as an error and bailed out with FILE_RETRY, so only one file and its companion were deleted per attempt. Instead, skip directory entries that disappeared after the directory was read. Also, when moving a directory to another file system, the sources are erased at the end in the copy order, so a "Cannot remove file ... No such file or directory" error popped up for every "._" file. Instead, treat ENOENT in try_remove_file() from unlink() as success. Assisted-By: Claude Opus 5.5 Signed-off-by: Yury V. Zaytsev <yury@shurup.com>
114f665 to
4bd2f87
Compare
Yes, exactly, and your prioritization algorithm appears to be non-deterministic. It would be nice to make a mechanical gate out of it, actually. Then it can be simply added to the CI. Obviously, it will not always be correct, but if it produces only some number of false negatives and doesn't produce false positives, this can be very valuable in my opinion.
I have squashed two commits into one. |
|
it's actually fairly deterministic, but it's rather fine-grained, which makes it hard to follow from the outside. the only written document i have is https://wiki.qt.io/Commit_Policy point 8. i don't think this can be automated to any meaningful degree, unless you put an AI into the loop. the PR LGTM now. |
Well, I don't think you necessarily need an LLM for that. You'd be surprised (as I was surprised) how much of this "judgement" stuff can be automated deterministically by using some sort of sum signals and gates, where signals can be regex matches, or some other functions. I used stuff like that to try to partially cure Claude's 1) unbearable comment diarrhea and 2) totally bad design judgement. I can't automatically apply it for this project though, because it's all Python stuff, but that would be actually a good use of tokens by sloppers... |
Proposed changes
Resolves: #4720