gocd: Repositories with a `git` folder + complex submodules still generate errors

Steps to reproduce

Create a new Pipeline (optionally in a new pipeline group).
Add the material CF-Release using the SSH (git@github.com:cloudfoundry/cf-release.git) or HTTPS (git@github.com:cloudfoundry/cf-release.git) retrieval links.
Add a default job/task which does something inconsequential, such as ls -la
Attempt to start the pipeline.
_Note_: This repository contains submodules within submodules. This is another edge case to consider.

Expected Result

The pipeline runs, and the ls shows the presence of a cf-release folder.

Actual Result

The pipeline is never scheduled. The following errors/warnings are in the go-server.log file:

2014-04-29 22:13:04,629  WARN [87@MessageListener for MaterialUpdateListener] MaterialDatabaseUpdater:117 - [Material Update] Modification check failed for material: URL: https://github.com/cloudfoundry/cf-release.git, Branch: master
java.lang.RuntimeException:
    at com.thoughtworks.go.domain.materials.git.GitCommand.gitLog(GitCommand.java:82)
    at com.thoughtworks.go.domain.materials.git.GitCommand.latestModification(GitCommand.java:66)
    at com.thoughtworks.go.config.materials.git.GitMaterial.latestModification(GitMaterial.java:104)
    at com.thoughtworks.go.server.service.materials.GitPoller.latestModification(GitPoller.java:31)
    at com.thoughtworks.go.server.service.materials.GitPoller.latestModification(GitPoller.java:27)
    at com.thoughtworks.go.server.service.MaterialService.latestModification(MaterialService.java:107)
    at com.thoughtworks.go.server.materials.LegacyMaterialChecker.findLatestModification(LegacyMaterialChecker.java:49)
    at com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:52)
    at com.thoughtworks.go.server.materials.ScmMaterialUpdater.addNewMaterialWithModifications(ScmMaterialUpdater.java:72)
    at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.addNewMaterialWithModifications(MaterialDatabaseUpdater.java:171)
    at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.initializeMaterialWithLatestRevision(MaterialDatabaseUpdater.java:130)
    at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.access$000(MaterialDatabaseUpdater.java:46)
    at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$1.doInTransaction(MaterialDatabaseUpdater.java:92)
    at com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:24)
    at com.thoughtworks.go.server.transaction.TransactionTemplate$3.doInTransaction(TransactionTemplate.java:53)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
    at com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:49)
    at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:89)
    at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:48)
    at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:29)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:65)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:50)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.thoughtworks.go.util.command.CommandLineException: Working directory "/var/vcap/store/gocd/pipelines/flyweight/6badd25c-53bf-440f-a768-b40c91e24795/src/router" does not exist!
    at com.thoughtworks.go.util.command.CommandLine.checkWorkingDir(CommandLine.java:343)
    at com.thoughtworks.go.util.command.CommandLine.setWorkingDir(CommandLine.java:334)
    at com.thoughtworks.go.util.command.CommandLine.withWorkingDir(CommandLine.java:417)
    at com.thoughtworks.go.domain.materials.git.GitCommand.cleanUnversionedFiles(GitCommand.java:183)
    at com.thoughtworks.go.domain.materials.git.GitCommand.cleanAllUnversionedFiles(GitCommand.java:140)
    at com.thoughtworks.go.domain.materials.git.GitCommand.fetchAndReset(GitCommand.java:129)
    at com.thoughtworks.go.domain.materials.git.GitCommand.fetchAndResetToHead(GitCommand.java:166)
    at com.thoughtworks.go.domain.materials.git.GitCommand.gitLog(GitCommand.java:79)
    ... 22 more

And…

2014-04-29 22:15:45,916  WARN [materialUpdateThread] MaterialUpdateService:146 - [Material Update] Skipping update of material GitMaterial{url=https://github.com/cloudfoundry/cf-release.git, branch='master', submoduleFolder='null'} which has been in-progress since Tue Apr 29 22:15:15 UTC 2014

In an earlier run of the same pipeline, we witnessed something slightly different, although we are having some difficulty recreating this particular message:

2014-04-29 01:53:14,541  WARN [78@MessageListener for ScheduleCheckListener] BuildCauseProducerService:189 - Error while scheduling pipeline: cf-release Possible Reasons: (1) Upstream pipelines have not been built yet. (2) Materials do not match between configuration and build-cause.

About this issue

Original URL
State: closed
Created 10 years ago
Comments: 15 (5 by maintainers)

Commits related to this issue

#152 - Test (ignored) for reproducing this. — committed to arvindsv/gocd by arvindsv 10 years ago
Merge pull request #278 from arvindsv/submodule_fail_name_path_different #152 - Test (ignored) for reproducing submodule issue — committed to gocd/gocd by sachinsudheendra 10 years ago
#152 - git version below 1.8 does not support submodule --name — committed to arvindsv/gocd by arvindsv 10 years ago
Merge pull request #282 from arvindsv/submodule_fail_name_path_different #152 - git version below 1.8 does not support submodule --name — committed to gocd/gocd by mdaliejaz 10 years ago
#152 - Fix GitCommandTest for submodule add. — committed to arvindsv/gocd by arvindsv 10 years ago
Merge pull request #285 from arvindsv/submodule_fail_name_path_different #152 - Fix GitCommandTest for submodule add. — committed to gocd/gocd by mdaliejaz 10 years ago
#152 - Fix GitCommandTest for submodule add. — committed to ermauliks/gocd by arvindsv 10 years ago

Most upvoted comments

In any case, it would be useful to have an option to ignore the submodules:

It will allow you to handle them with your own scripts/commands
It will allow you to clone them just when needed (eg: you only need the promote/deploy/publish scripts that are in your superproject)

cattz on Jan 19, 2017

/cc @mmb

Apologies again for the back and forth - we seem to have narrowed down the cause. In the cf-release .gitmodules file, there is an entry:

[submodule "src/router"]
  path = src/gorouter
  url = https://github.com/cloudfoundry/gorouter

Here, the name differs from the path and it appears from the newly released source code that this command/regex combination is insufficient to reveal that:

git config --get-regexp ^submodule\\..+\\.url

What may be required is using the equivalent of this:

grep path .gitmodules | sed 's/.*= //'

, which actually outputs the ‘path’ of the submodule.

Finally, based on the git-submodules documentation, we surmise that having a name differing from a path is legitimate.

We’ll update with more information when we have some.

gajwani on Apr 29, 2014