react-native: [Packager] doesn't resolve modules that are symlinks

Hi all, being tired of how screwed up this situation is for us I wrote a simple tool that listens to changes in one folder and copies them to another folder. It’s called WML (Watchman-Links, since it’s based on watchman). Check it out here.

It’s very simple to use:

# add the link to wml using `wml add <src> <dest>`
wml add ~/my-package ~/main-project/node_modules/my-package
# start watching for changes
wml start

Just ran into this trying to develop a npm library locally using npm link

Any ideas when this would be resolved?

This is really frustrating 😕

I’ve just upgraded to React Native 0.35 which includes #9792 and thus should fix this issue.

I’ve created a node_modules/src symlink to src but when I include a component (import Button from 'src/components/Button/') it says that I’m requiring an unknown module. With plain React (created with create-react-app) under Firefox it works.

I’ve tried to rerun both react-native run-android and react-native start but it doesn’t change anything

Figured out a little more about the cause – the issue goes away when switching to the Node-based watcher instead of the Watchman-based one. The watchman homepage says, “Watchman does not follow symlinks. It knows they exist, but they show up the same as any other file in its reporting.” so this could be the root case.

Support for symlinks seemed to get pretty strong pushback in https://github.com/facebook/watchman/issues/105 so perhaps the packager could use a combination of Watchman for most files and the Node watcher for symlinked packages. npm linked packages reside in the first node_modules directory (you’d have to go out of your way to end up with node_modules/pkg1/node_modules/<linkedpkg>), so that might simplify things a little. I tried adding a --noWatchman option to the packager but ran into fd limit issues unfortunately.

The npm link workflow is really important for anyone developing a React Native module so I do think this is a priority for the external RN community.

This is not a good practice. We need to get support for symlinks easily! Come on!

I hope it is annoying enough so they finally fix a 2 year old issue.

Someone should really find a proper solution for this thing. developing without npm link is kinda foolish. copying the dependency into node_modules, is also strange, what do you do on deployment?

I have posted the solution in http://stackoverflow.com/a/38976236/3360028

This is hindering me from making really (or relatively) awesome RN modules 😦

+1

@lacker I think we’re close.

I just tried now on using react-native@0.42.0-rc2 and lerna@2.0.0-beta.37 in https://github.com/infinitered/reactotron (specifically https://github.com/infinitered/reactotron/tree/master/packages/demo-react-native)

I tried a few different things.

Thing 1

If we just try to do a symlink, and the target library has it’s own symlink it, then the packager can’t resolve it.

Thing 2

If i symlink a react-native-based dependency, the packager detects it just fine 🎉 , but if that dependency has it’s own react-native in its node_modules, then all hell breaks loose. You get duplicate @providesModules if they’re the same RN, and conflicted @providesModules if they’re different.

Lerna Almost Saves The Day

Lerna can now deal with this by hoisting that dependency up to the root of the repo. 🎉

But if you do that for both the host app, then the cli doesn’t work (because the host app doesn’t have node_modules/react-native/*.

So, if we manually symlink that to ‘…/…/node_modules/react-native’, the package goes off on a spiritual journey rounding up everything in the mono repo, eventually dying because of conflicts. I understand that. The packager was designed to be at the root level of an app, not a monorepo.

The idea experience for me

Up until 40, I’ve been just developing my 3rd party stuff right in the node_modules. I’d then manually copy & paste the stuff out. It felt bad (like editing files in production) and I cried myself to sleep, but you know what? It worked. Live reloading worked too. But now, the packager has to be restarted.

It’d be great if we could use npm link to drive this. It’d feel like normal node module development.

I haven’t explored rn-cli.config.js blacklisting & package roots yet.

I also volunteer as tribute if you need people to test stuff.

@mkonicek I think there are 2 issues. Consider a repo layout like the following and running the RN packager from ./example

my_package
├── example
│   ├── node_modules/my_package (-> ../../)
│   ├── package.json
├── lib
├── node_modules
├── package.json

1. Watchman doesn’t follow symlinks, so changes made in ./lib won’t be noticed by the RN packager.
2. Without watchman, the RN packager will follow the symlink and through it find ./node_modules which will typically (for e.g. testing purposes) contain all of the same dependencies as exist in ./example/node_modules, which leads to the RN packager complaining about duplicate source files.

Hi guys, any update on this ?

My only way to develop a react native library is to git clone my repo inside my node_modules, isn’t it ?

Perhaps related, relative requiring a module above the iOS/React Native directory fails the same way:

// in index.ios.js
var Auth = require('../common/auth');

Error: Requiring unknown module “…/common/auth”. If you are sure the module is there, try restarting the packager.

I tried symlinking that higher-level directory within the React Native project to fix, but it failed as well, leading me to this issue.

We definitely need to find way how to run React Native Packager with --preserve-symlinks option.

Looks like the pain will be over soon 😃 https://github.com/facebook/react-native/pull/9009

OK, so here’s the repro steps:

1. react-native init AwesomeProject

2. mkdir test

3. cd test

4. npm init (all defaults)

5. Make an index.js, set it to:

   
   module.exports = function () {
     console.log('hey')
   }
   

6. npm link

7. cd …/AwesomeProject

8. npm link test

9. Add this to your index.android.js :

   import myFunction from 'test'
   
   myFunction()
   

10. brew uninstall watchman

11. react-native run-android

12. Open chrome debugger to check the console.log hey

13. Change console.log(‘hey’) to console.log(‘ho’)

14. Reload the javascript in the android emulator

15. Still says ‘hey’

Environment:

Mac OS X 10.11.3 react-native 0.20.0 react-native-cli 0.1.10 node 5.5.0 npm 3.7.2 Genymotion Galaxy S6 API 23

Thank you 😄

Here’s a crude workaround:

Create file watchAndCopy.js in your react-native project root:

var fs = require('fs-extra');
var watch = require('node-watch');
var rimraf = require('rimraf');
var exec = require('child_process').exec;

var packageName = 'my-package';
var packagePath = '../'+packageName;

console.log('Cleaning node_modules/'+packageName)
rimraf('node_modules/'+packageName, function () {
  console.log('Copying '+packageName)
  fs.copy(packagePath, 'node_modules/'+packageName, function (err) {
    if (err) return console.error(err)
  })

  // Couldn't get rimraf to delete .git via function call.
  // This seems to work, though.
  exec('rimraf node_modules/'+packageName+'/.git', function () {
    console.log('Watching '+packageName)
    watch(packagePath, function(filename) {
      var localPath = filename.split(packageName).pop()
      var destination = 'node_modules/'+packageName+localPath
      console.log('Copying '+filename+' to '+destination)
      fs.copy(filename, destination, function (err) {
        if (err) return console.error(err)
      })
    })
  })
})

Replace my-package with your package and make sure the path is right. Make sure rimraf is installed globally (npm install rimraf -g). Run node watchAndCopy.js and you’re good to go.

Plus side is it works with watchman so it’s super fast 😃

The RN packager was built for Facebook’s needs; which unfortunately (for us) didn’t include symlinks.

But… packager will be getting some love shortly. Hoping to see some news this week.

In the meantime:

• Have a look at haul by the Callstack crew. Provided your symlink’d dependency doesn’t have its own react-native dependency, this works really well. (if it does you can delete the files temporarily).
• It is possible to stage some work out of node_modules again with live reloading. Yes, it’s pretty derpy, but pair it with a file watcher and it’s better than nothing.

I couldn’t use a second project root, because the RN packager would find duplicate files in my package’s node_modules dir. I ended up syncing the package’s changed files into the consuming app’s node_modules dir: https://github.com/artsy/emission/commit/1049aa79c8117b0490f4ce2565ad22d628e90154#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R12

@nikki93 investigated using bindfs, which lets you mount a directory elsewhere in your filesystem, and found that it partly works – the packager/watchman can see the files within the mounted directory, but it doesn’t pick up changes so it’s not 100% useful for development & iteration. It seems that bindfs doesn’t propagate whatever mechanism watchman uses to detect changes.

While there’s progress being made here, we should improve the error message and throw an exception if there’s a symlink pointing to this issue/product pains. Otherwise people will keep getting confused until they figure out that react native doesn’t support symlinks

I’d also love to see this fixed! In the meantime it would great if there was something written up on the best workflow to develop third-party components, specifically when it comes to testing them in an example app (which is where this problem comes into play).

Can you move this to https://github.com/facebook/metro-bundler now that the packager has been moved out of React Native?

This wasn’t in the original post, but please note that this isn’t some obscure feature request. npm link, for example, uses symbolic links behind the scenes, so it breaks with that too. I can confirm that copying, rather then symlinking, does work, however that becomes impractical when working on different npm packages locally.

Also, I tried it on macOS and linux and same issue occurs.

Not sure but probably node option --preserve-symlinks can help you somehow. https://nodejs.org/dist/latest-v6.x/docs/api/cli.html#cli_preserve_symlinks

I had similar problem with symlinked files/modules (but on regular node project) and this option resolves problems for me.

Hey guys, I got tired with dealing with the symlink handling so I forked react-native and patched the packager to fix the issues I was having. This might not fix everyones issues but it has certainly fixed mine. The issue that I was having (and have had since the dawn of time in using react-native) was caused by the packagers inability of discovering peer dependencies on symlinked modules (npm link or yarn link). I’ve updated the resolver for the packager to be aware of the root paths of the project and automatically add these paths to the search query used to resolve modules. The prior implementation would try to be smart about resolving the paths based on the modules real path which breaks on symlinked modules.

Take it or leave it, but you are all welcome to use it. I’ve patched both the 0.42-stable branch and the 0.43-stable branch but haven’t really tested 0.43 since it relies on an alpha version of react.

Github repo

To use 0.42-stable update package.json react-native dependency to: "react-native": "https://github.com/braunsquared/react-native/tarball/b576d07039dfaa0045fcff8b463f7490a86a7465"

For 0.43-stable use: "react-native": "https://github.com/braunsquared/react-native/tarball/843be3173f617de0e13fc219f1aa2a809b94d1a3"

Any ideas when this would be resolved?

Yeah, I think symlinks are still busted. @ericvicenti is looking into the general problem of “how does one develop a React Native library with best practices” so 1/ people should throw their random ideas at him and 2/ I am hopeful this will get fixed en route.

I’m having issues with this as well. I can pass in the project roots and the packer will pick it up but it will only watch for file changes in the root. Here’s what I’ve seen so far with

• react-native 0.22.2
• Uninstall watchman
• let’s say i have /mysymlink
• Use --projectRoots or make a rn-cli.config.js to inject /mysymlink into projectRoots

Changes to the files in that folder root are picked up, but it does not recurse at all, so /mysymlink/index.js works but /mysymlink/foo/bar.js does not.

Debugging this a little bit it seems like:

The packager server uses a filewatcher https://github.com/facebook/react-native/blob/master/packager/react-packager/src/Server/index.js#L13

The filewatcher comes from node-haste https://github.com/facebook/node-haste/blob/master/src/FileWatcher/index.js#L80 Which uses sane

Sane uses walker to walk the tree https://github.com/amasad/sane/blob/master/src/node_watcher.js#L329 (note the dir callback)

and finally the walker https://github.com/daaku/nodejs-walker/blob/master/lib/walker.js#L76 doesn’t recurse down symbolic links (whereas it does if isDirectory() is true)

So somewhere in this whole chain it needs to recurse for this solution to actually work

I don’t know the advantages of Watchman but like others have pointed out, a fix in the meantime is just uninstalling Watchman:

brew uninstall watchman

According to https://github.com/facebook/react-native/blob/master/local-cli/util/Config.js, you can put custom configuration for the packager. It should be named rn-cli.config.js.

Here is an example configuration: https://gist.github.com/andon/bdf061e607f68def26fa#file-rn-cli-config-js

Declaring my local dependency as follow : dependencies: { my_local_module: "file:../../path/to/local/module" } then npm install my_local_module seems to work for me.

@tuckerconnelly if you can come up with an isolated reproduction steps I’ll happily fix it (or accept patches). The underlying watching library is Sane.

Also cc @kittens who was looking into making node the default watcher

It worked after uninstalling watchman! Hooray for fallbacks 😃

However when I update my symlinked package, the packager doesn’t pick up on it and re-build the app. Any way to fix this?

How will we get better development, modules, tools, if developing modules that use other modules is a complete pain?

Sorry for the delay, was out for a while.

@amasad - seems like this was working for some people in 0.7.1 and not in 0.8.0-rc, as reported in #2076

What happened is that we started using watchman for the filesystem crawling. My understanding is that watchman doesn’t follow symlinks but should treat them like any other file. Which in theory should still work. I can try to debug this soon. Alternatively we can add an option to force not using watchman for filesystem crawling.

I’ve found that this can be worked around by uninstalling watchman and therefore forcing react native to use the default node file watcher, however this does have speed drawbacks, and you can come up against issues with the number of files being accessed depending on the project.

-------- Original message -------- From: Braden Simpson notifications@github.com Date: 14/08/2015 20:25 (GMT+00:00) To: facebook/react-native react-native@noreply.github.com Cc: Gethin Webster gethin.webster@agilityworks.co.uk Subject: Re: [react-native] [Packager] doesn’t resolve modules that are symlinks (#637)

Is there any good workaround here?

Reply to this email directly or view it on GitHubhttps://github.com/facebook/react-native/issues/637#issuecomment-131214239.

👍 Same problem here. Symlinks are very useful, especially when I don’t want to make a node package just to share a component between two related React apps.

I have an internal fix for this. Aiming to get it out next sync 😃

is it a bug or is it by design? symlinks would be really handy actually 😦