lerna: [Bug] Import failing (2.0.0-beta.24)

ADD: A bit of context. Monorepo concept is relatively new and I believe many companies are deciding to use it to consolidate bits of the existing code. To me the import feature is one of the biggest selling points (we’ve currently decided to start using lerna for clusters of the related modules).

So I believe smooth import process (fixing the current issue + wokring with the remotes #269) is crucial for the existing projects to migrate to lerna.

Hi @ericf, just want to let you know that I haven’t forgotten about this. I spent some time looking into it, and it actually doesn’t look like there’s an “easy” solution. The trouble arises from merge commits. When changes are pulled in with merge commits the output of git format-patch is always relative to the topic branch that the commit was originally made on. That’s not guaranteed to apply to the target branch it wound up in, though. Only the final state of the topic branch needs to apply cleanly to the branch it’s merged into. Intervening commits may conflict. There may even ultimately be conflict resolution in the merge commit itself.

Git itself actually doesn’t produce a patch-set that applies in this case. For example, in master of intl-messageformat this fails on the same commit that lerna import does:

$ git format-patch 09b7b91 # generate patches for every commit after the initial commit
$ git reset --hard 09b7b91 # rewind to the initial commit
$ git am 0*.patch          # re-play commit history
...
error: patch failed: tests/message.js:27
error: tests/message.js: patch does not apply
Patch failed at 0023 Move `lib/message.js` to `./index.js`.

I’m not sure if there’s going to be a solution to this that lets us preserve one-for-one commit history from the external repository. We might wind up needing to squash branches that contain patches that don’t apply cleanly. Unless a better solution presents itself before I have a chance to take this on, that’s what I’ll do.

I have no idea if this is relevant to anybody else, but a slightly modified form of the function in this stack overflow answer worked great for me. For OS X users, make sure to install and use gnu-sed (brew install gnu-sed was all I had to do, but others may need to update the script to use gsed).

Define this function in a shell script somewhere (I put it in my ~/.bash_profile temporarily b/c I’m lazy and/or stupid):

git_add_repo()
{
    repo="$1"
    dir="$(echo "$2" | sed 's/\/$//')"
    path="$(pwd)"
    mkdir -p "$dir"

    tmp="$(mktemp -d)"
    remote="$(echo "$tmp" | sed 's/\///g'| sed 's/\./_/g')"

    git clone "$repo" "$tmp"
    cd "$tmp"

    git filter-branch --index-filter '
        git ls-files -s |
        sed "s,\t,&'"$dir"'/," |
        GIT_INDEX_FILE="$GIT_INDEX_FILE.new" git update-index --index-info &&
        mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"
    ' HEAD

    cd "$path"
    git remote add -f "$remote" "file://$tmp/.git"
    git fetch "$remote"
    git merge --allow-unrelated-histories -m "Merge repo $repo into master" --no-edit "$remote/master"
    git remote remove "$remote"
    rm -rf "$tmp"
}

export -f git_add_repo

Then from the root of your lerna mono-repo, run:

git_add_repo git@github.com:your_user/your-repo.git packages/your-repo

It’ll take a bit to run but when it finishes, your git history should look like you’ve been committing files under packages/your-repo all along. Everything on GitHub (history, merges, commits, blame, diffs) look to be working.

I haven’t tested whether it’s possible to pull in upstream changes from the original repo (but I seriously doubt it), so make sure you stop committing to the original one and fully switch over.

I don’t have any special understanding of why this works, I likely can’t help if it doesn’t work for you. When in doubt, do it in a freshly cloned local version of your lerna repo so you can delete it and try again. Good luck!

Having the same issue, since I don’t have time to investigate further I just copied the files over without git history. I’m on beta.30 btw.

Having the same issue here. git rebase --interactive didn’t help unfortunately.

Rather than errorring out can we be interactive and allow the user to take an action on a commit?

Even if that means retrying to import over and over, but each time something doesn’t work let the user do something that fixes it.

Just throwing ideas out there

@gigabo thanks for digging deep into this one — definitely a messy Git situation.

We might wind up needing to squash branches that contain patches that don’t apply cleanly.

I think that seems fine me to as a strategy to employ when there’s an issue in a merge commit.