WSL: lstat error when use yarn to install nodejs package

  • Your Windows build number: windows 10 1803

  • What you’re doing and what’s happening: When i use yarn to install nodejs package, it will cause error with lstat.

  • What’s wrong / what should be happening instead:

Error Logs:

Yarn version: 
  1.6.0

Node version: 
  8.9.1

Platform: 
  linux x64

Trace: 
  Error: ENOENT: no such file or directory, lstat '/mnt/***/node_modules/asynckit/index.js'

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 21
  • Comments: 22

Most upvoted comments

fsutil behavior set disablelastaccess 1 in PowerShell worked for me

Still get errors reported by @Fumler Sometimes it works sometimes it does not. Different files reported missing every time it does not work. New install - npm and yarn installed after case=off in wls.conf

OS Name Microsoft Windows 10 Pro Version 10.0.17134 Build 17134 Ubuntu 18.04

untitled

Video: https://drive.google.com/file/d/1QWwVblsVIxnopNWu7CeuoYavyFELKv8q/view?usp=sharing

@therealkenc, I’m running on Microsoft Windows [Version 10.0.17134.286] and still waiting for 1809 to be released. I’m sorry right now I’m not running on 17760. I think I need someone else to help me testing this.

Preparing Sample Directory for Testing

As well, those repro steps are going need to include steps to populate “the directory” that has many child directories not “a directory” that has many child directories (or build the tree in the test code). Also some indications on what “likely” means. “Will fail” is what you are shooting for here, because if someone cuts-and-pastes your repro steps and it doesn’t fail, the new issue will go dark pretty quick. Loop the test if you have to.

Thank you very much for your tips. At the moment, I don’t know how big exactly the directory tree should be to trigger the failure. I’m trying this on a directory, /mnt/c/Users/nieltg/Projects, which is actually an SSD and contains all of my active projects, which consists of, approximately, 12 Node projects, 3 Django projects, some Go projects, etc.

10 out of 10 invocations of the sample program on /mnt/c/Users/nieltg/Projects have failed. Here is the snippet of command I used for testing in zsh:

ok=0
fail=0
for ((i=0; i<10; i++)); do
  if ( async-dirtree-test > /dev/null 2>&1 ); then
    ((ok++))
  else
    ((fail++))
  fi
done
echo ok: $ok
echo fail: $fail

Here is the result of /mnt/c/Users/nieltg/Projects:

( async-dirtree-test > /dev/null 2>&1; )  1.83s user 5.17s system 181% cpu 3.865 total
( async-dirtree-test > /dev/null 2>&1; )  15.34s user 45.44s system 188% cpu 32.309 total
( async-dirtree-test > /dev/null 2>&1; )  12.94s user 25.73s system 316% cpu 12.201 total
( async-dirtree-test > /dev/null 2>&1; )  13.67s user 28.77s system 315% cpu 13.440 total
( async-dirtree-test > /dev/null 2>&1; )  24.11s user 52.55s system 309% cpu 24.778 total
( async-dirtree-test > /dev/null 2>&1; )  27.45s user 56.59s system 316% cpu 26.594 total
( async-dirtree-test > /dev/null 2>&1; )  11.83s user 23.14s system 267% cpu 13.063 total
( async-dirtree-test > /dev/null 2>&1; )  2.44s user 5.70s system 261% cpu 3.108 total
( async-dirtree-test > /dev/null 2>&1; )  14.91s user 30.75s system 302% cpu 15.078 total
( async-dirtree-test > /dev/null 2>&1; )  3.50s user 5.55s system 222% cpu 4.074 total
ok: 0
fail: 10

I’ve also tried this on AvaKathrynShaw/React-16-Demo which can be prepared with these commands:

$ git clone https://github.com/AvaKathrynShaw/React-16-Demo
$ cd React-16-Demo && yarn

5 out of 5 invocations of the sample program on it have failed on my machine. Here is the result:

( async-dirtree-test > /dev/null 2>&1; )  6.50s user 10.72s system 279% cpu 6.164 total
( async-dirtree-test > /dev/null 2>&1; )  3.05s user 5.36s system 262% cpu 3.204 total
( async-dirtree-test > /dev/null 2>&1; )  4.48s user 7.77s system 285% cpu 4.289 total
( async-dirtree-test > /dev/null 2>&1; )  2.11s user 3.31s system 228% cpu 2.368 total
( async-dirtree-test > /dev/null 2>&1; )  4.73s user 8.47s system 282% cpu 4.668 total
( async-dirtree-test > /dev/null 2>&1; )  8.75s user 14.81s system 265% cpu 8.860 total
( async-dirtree-test > /dev/null 2>&1; )  9.56s user 15.27s system 272% cpu 9.109 total
( async-dirtree-test > /dev/null 2>&1; )  7.52s user 11.69s system 214% cpu 8.939 total
( async-dirtree-test > /dev/null 2>&1; )  10.38s user 15.47s system 230% cpu 11.208 total
( async-dirtree-test > /dev/null 2>&1; )  9.14s user 15.28s system 231% cpu 10.536 total
ok: 5
fail: 5

More Things and strace Log

After conducting those and more experiments, I found more things:

  • The sample program itself is unlikely to be failed if it does other things other than calling fs.stats(), like logging, so it does make me harder to debug.
  • The sample program itself is unlikely to be failed if I run it on LxFs. It needs ~3 minutes to fail while running on WSL’s home directory, /home/nieltg, but it only needs ~10 seconds to fail while running on /mnt/c/Users/nieltg/Projects which is on DrvFs.
  • Tracing the sample program using strace make it needs a longer time then without strace before it failed.
  • The fastest way to make it fail is running it without any tracing and logging mechanism, for example, async-dirtree-test > /dev/null which discards all of the output it produces so the lstat calls are made more frequent.
  • The failure stills appear with or without case=force in DrvFs mount options on my machine.

From this part of article and from the strace logs that I collected from the sample program, I learnt that lstat syscall isn’t actually asynchronous and is being called on Node’s worker threads. Here is a portion of the strace log of a single worker thread:

...
lstat("compfest/2/composite", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
lstat("compfest/2/observer", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
lstat("compfest/2/strategy", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
lstat("compfest/4/go-academy-game01", {st_mode=S_IFDIR|0777, st_size=512, ...}) = 0
...

It’s a good effort, so thanks for that. It unfortunately makes for a less than ideal test case, because while the code is short there are a lot of big moving pieces (nodejs, V8, libuv). But it’s more likely than not you’re onto something.

You’re welcome. 😃 And with all of those facts, I wonder whether the C/C++ version will show the same behavior as the Node version. I’m thinking of a simple program which just calls lstat() and readdir() recursively until all files in the directory and its descendants are visited and maybe with some threading.

The complete strace logs are available on this archive and the command I used to harvest them is available below:

% strace -ff -o strace.txt async-dirtree-test > /dev/null
/home/nieltg/.config/yarn/global/node_modules/async-dirtree-test/node_modules/rxjs/internal/util/hostReportError.js:4
    setTimeout(function () { throw err; });
                             ^

Error: ENOENT: no such file or directory, lstat 'lazy-transform/lazy-transform/node_modules/date-fns/is_this_hour'
strace -ff -o strace.txt async-dirtree-test > /dev/null  23.98s user 64.06s system 59% cpu 2:27.00 total

Summary

Well, it feels that I was just writing an article. Anyway, I think this can be tested by following this guide:

  1. Turn off Windows Defender to speed up things.
  2. Install the test program.
    $ yarn global add async-dirtree-test
    
  3. Go to any directory in DrvFs, for example, My Documents:
    $ cd /mnt/c/Users/[ Place your Windows username here ]/Documents
    
  4. Prepare a sample directory.
    $ git clone https://github.com/AvaKathrynShaw/React-16-Demo
    $ cd React-16-Demo && yarn
    
  5. Run the test program with the output discarded. If it doesn’t get failed, repeat it 5-10 times.
    $ async-dirtree-test > /dev/null
    

Thank you very much. I’m looking forward to more hearing about this.

UPD: seems to be fixed in 1809

@yshterev Are you sure your D:\workspace is a brand new folder created after enabling case=off? You can try on cmd with fsutil file queryCaseSensitiveInfo D:\workspace fsutil file queryCaseSensitiveInfo D:\workspace\go fsutil file queryCaseSensitiveInfo D:\workspace\go\src and so on… If some of them say is Enabled then you will have this issue. Otherwise it could be a race condition like #2712

Thank you @onomatopellan !

Putting yarn cache location on a Lxfs folder fixed the issue.

I’m having the same issue since the April update, it happens for one project though. The rest of my projects use yarn and no issues there, other than random nodemon restarts.

I tried the same project on other machine and it also fails, it fails specifically when running jest tests. Happens on both node 9.8 and 10. Yarn version is 1.6.0