moon: [bug] Intermittently, cached moon outputs get partially restored
Describe the bug
I’ve been encountering this off and on since adopting moon and had been holding off on reporting it until I could see a pattern on when it happens. Unfortunately I still haven’t but I want to describe the situation and maybe we can figure it out together.
I have a task messaging-api:gen-prisma-client
that happens fairly on in my project. The outputs of the task are all sent to a single folder within the source tree of a project. Included in the task outputs are some test files (.js
, .d.ts
and .json
files) and binary files (.node
extension) . There are only a couple of inputs for the task and they do not change often which means the output for this task can often be restored from cache.
Intermittently, when this task is executed subsequent tasks that depend upon it will fail because a file that is a .js
output of the task has invalid syntax. It appears that when restoring the file from cache only part of a memory buffer is flushed to disk because it looks like the .js
file starts halfway through.
I have not noticed if the partially restored file is always a particular part of the file.
Steps to reproduce
I wish I had these ☹️
Expected behavior
The previous directory would be cleared out or the files would be cleanly replaced.
Screenshots

Environment
npx envinfo --system --SDKs --languages --managers --binaries
System:
OS: macOS 13.2.1
CPU: (8) arm64 Apple M1
Memory: 184.09 MB / 16.00 GB
Shell: 5.2.15 - /opt/homebrew/bin/bash
Binaries:
Node: 16.19.0 - ~/.volta/tools/image/node/16.19.0/bin/node
Yarn: 3.4.1 - ~/.volta/tools/image/yarn/1.22.11/bin/yarn
npm: 8.19.2 - ~/.volta/tools/image/npm/8.19.2/bin/npm
Watchman: 2023.03.13.00 - /opt/homebrew/bin/watchman
Managers:
Cargo: 1.68.0 - ~/.cargo/bin/cargo
Gradle: 8.0.2 - /opt/homebrew/bin/gradle
Homebrew: 4.0.6 - /opt/homebrew/bin/brew
Maven: 3.9.1 - /opt/homebrew/bin/mvn
pip3: 22.3.1 - /opt/homebrew/bin/pip3
RubyGems: 3.0.3.1 - /usr/bin/gem
Languages:
Bash: 5.2.15 - /opt/homebrew/bin/bash
Go: 1.20.2 - /opt/homebrew/bin/go
Java: 11.0.17 - /usr/bin/javac
Perl: 5.30.3 - /usr/bin/perl
Python3: 3.11.2 - /opt/homebrew/bin/python3
Ruby: 2.6.10 - /usr/bin/ruby
Rust: 1.68.0 - /Users/bryanjswift/.cargo/bin/rustc
Additional context
The task I mentioned at the start is defined like this, the :env-compare
task does not get cached (it has cache: false
in options).
gen-prisma-client:
command: 'prisma generate --schema=prisma/schema.prisma'
deps:
- 'root:env-compare'
inputs:
- '/.env'
- 'prisma/schema.prisma'
outputs:
- 'src/generated/prisma-client'
The output of the task looks like this:
packages/messaging-api/src/generated/prisma-client/
├── index-browser.js
├── index.d.ts
├── index.js
├── libquery_engine-darwin-arm64.dylib.node
├── libquery_engine-rhel-openssl-1.0.x.so.node
├── package.json
├── runtime
│ ├── index-browser.d.ts
│ ├── index-browser.js
│ ├── index.d.ts
│ ├── library.d.ts
│ └── library.js
└── schema.prisma
2 directories, 12 files
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 22
Released a patch, let me know if it crops up again.
I’m pretty confident this is resolved. It’s been more than a week since I’ve seen the error reported in this issue. I’m closing it. Thanks @mrobinsn for investigating and @milesj for fixing!
@milesj I think I see the problem here. In the tree differ logic, there is a bit where if the sizes are the same it falls back to a byte level equality check. That involves reading from the source tar, which advances its file pointer, so when you land back into the untar logic after some portion of the source/dest files were not equal, it will only extract data starting from that position. This explains the behavior in the Discord thread where the hydrated files were missing some number of bytes at the beginning.
It might also explain why this happens so rarely and inconsistently. It would only happen when the source and destination both exist, are exactly the same size, but differ in content (but not fully).
Good to know, will dig into.