backstage: Backend failed to start up, Error: The migration directory is corrupt, the following files are missing

Expected Behavior

The backend should start up successfully

Actual Behavior

An error occurs.

[1] Backend failed to start up, Error: The migration directory is corrupt, the following files are missing: 20200511113813_init.js, 20200520140700_location_update_log_table.js, 20200527114117_location_update_log_latest_view.js, 20200702153613_entities.js, 20200721115244_location_update_log_latest_deduplicate.js, 20200805163904_location_update_log_duplication_fix.js, 20200807120600_entitySearch.js, 20200809202832_add_bootstrap_location.js, 20200923104503_case_insensitivity.js, 20201005122705_add_entity_full_name.js, 20201006130744_entity_data_column.js, 20201006203131_entity_remove_redundant_columns.js, 20201007201501_index_entity_search.js, 20201019130742_add_relations_table.js, 20201123205611_relations_table_uniq.js, 20201210185851_fk_index.js, 20201230103504_update_log_varchar.js, 20210209121210_locations_fk_index.js, 20210302150147_refresh_state.js, 20210622104022_refresh_state_location_key.js, 20210813143113_add_refresh_state_hash.js, 20210925102509_add_refresh_state_input_hash.js, 20220116144621_remove_legacy.js, 20220222164811_reprocess_for_relation_refs.js

Steps to Reproduce

Configure Postgres connection as such:

  database:
    # config options: https://node-postgres.com/api/client
    client: pg
    connection:
      connectionString: 'postgres://backstage:secret@machinename/backstage'
      # host: machinename
      # port: 5432
      # user: backstage
      # database: 'backstage'
      # password: secret
      # https://node-postgres.com/features/ssl
      ssl: # see https://www.postgresql.org/docs/current/libpq-ssl.html Table 33.1. SSL Mode Descriptions (e.g. require)
        rejectUnauthorized: false

Run yarn dev
See error

Context

I’ve been trying to get my Backstage instance connected to a Postgres database for a few days now but I always end up back in this same spot.

Things I’ve tried:

Clean install/redo the changes I need
Deleted all node_modules folders and reinstalled
Dropped and recreated all databases related to Backstage
Manually removed rows from migrations table

Also, I am using the connectionString configuration property because I need to specify the maintenance database. We cannot get read/write access to the default postgres database per our account security policies, and the database configuration property is not adhered to it seems.

I found these files in the catalog-backend which appears to be exactly what the error message is pointing to.

Any help would be greatly appreciated!

Your Environment

Running Ubuntu 20

Browser Information: n/a
Output of yarn backstage-cli info:

OS:   Linux 5.4.0-107-generic - linux/x64
node: v16.15.0
yarn: 1.22.18
cli:  0.17.0 (installed)

Dependencies:
  @backstage/app-defaults                        1.0.1
  @backstage/backend-common                      0.13.2
  @backstage/backend-tasks                       0.3.0
  @backstage/catalog-client                      1.0.1
  @backstage/catalog-model                       1.0.1
  @backstage/cli-common                          0.1.8
  @backstage/cli                                 0.17.0
  @backstage/config-loader                       1.1.0
  @backstage/config                              1.0.0
  @backstage/core-app-api                        1.0.1
  @backstage/core-components                     0.9.3
  @backstage/core-plugin-api                     1.0.1
  @backstage/errors                              1.0.0
  @backstage/integration-react                   1.0.1
  @backstage/integration                         1.1.0
  @backstage/plugin-api-docs                     0.8.4
  @backstage/plugin-app-backend                  0.3.31
  @backstage/plugin-auth-backend                 0.13.0
  @backstage/plugin-auth-node                    0.2.0
  @backstage/plugin-azure-devops-backend         0.3.10
  @backstage/plugin-azure-devops-common          0.2.2
  @backstage/plugin-azure-devops                 0.1.20
  @backstage/plugin-catalog-backend-module-azure 0.1.2
  @backstage/plugin-catalog-backend              1.1.1
  @backstage/plugin-catalog-common               1.0.1
  @backstage/plugin-catalog-graph                0.2.16
  @backstage/plugin-catalog-import               0.8.7
  @backstage/plugin-catalog-react                1.0.1
  @backstage/plugin-catalog                      1.1.0
  @backstage/plugin-github-actions               0.5.4
  @backstage/plugin-org                          0.5.4
  @backstage/plugin-permission-common            0.6.0
  @backstage/plugin-permission-node              0.6.0
  @backstage/plugin-permission-react             0.4.0
  @backstage/plugin-proxy-backend                0.2.25
  @backstage/plugin-scaffolder-backend           1.1.0
  @backstage/plugin-scaffolder-common            1.0.1
  @backstage/plugin-scaffolder                   1.1.0
  @backstage/plugin-search-backend-node          0.6.0
  @backstage/plugin-search-backend               0.5.1
  @backstage/plugin-search-common                0.3.3
  @backstage/plugin-search-react                 0.1.0
  @backstage/plugin-search                       0.8.0
  @backstage/plugin-tech-radar                   0.5.11
  @backstage/plugin-techdocs-backend             1.1.0
  @backstage/plugin-techdocs-node                1.1.0
  @backstage/plugin-techdocs-react               0.1.0
  @backstage/plugin-techdocs                     1.1.0
  @backstage/plugin-user-settings                0.4.3
  @backstage/release-manifests                   0.0.2
  @backstage/search-common                       0.3.3
  @backstage/test-utils                          1.0.1
  @backstage/theme                               0.2.15
  @backstage/types                               1.0.0
  @backstage/version-bridge                      1.0.1

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (8 by maintainers)

Most upvoted comments

Managed to fix the error, connected to the local db then went to backstage_plugin_catalog and then deleted the corrupt migrations from both knex_migrations and knex_migration_lock and that resolved the issue!

These were created recently when i tried upgrading to the lates version of backstage (1.9.3), I still don’t know what caused the creation of those, probably a package upgrade

moby-codes on Jan 13, 2023

@arhill05 I’m stuck on this issue. Can you please provide some more information. I’m currently have the following config:

app:
  # Should be the same as backend.baseUrl when using the `app-backend` plugin.
  baseUrl: https://example.herokuapp.com
  title: example

backend:
  # Note that the baseUrl should be the URL that the browser and other clients
  # should use when communicating with the backend, i.e. it needs to be
  # reachable not just from within the backend host, but from all of your
  # callers. When its value is "http://localhost:7007", it's strictly private
  # and can't be reached by others.
  baseUrl: https://example.herokuapp.com
  listen:
    port:
      $env: PORT

  # config options: https://node-postgres.com/api/client
  database:
    client: pg
    ensureExists: false
    pluginDivisionMode: 'schema'
    connection: ${DATABASE_URL}
    ssl: true
    knexConfig:
      searchPath: ['knex', 'public']
    plugin:
      auth:
        database:
          $env: DATABASE_URL
      catalog:
        database:
          $env: DATABASE_URL
techdocs:
  builder: 'local' # Alternatives - 'external'
  generator:
    runIn: 'docker' # Alternatives - 'local'
  publisher:
    type: 'local' # Alternatives - 'googleGcs' or 'awsS3'. Read documentation for using alternatives.


catalog:
  # Overrides the default list locations from app-config.yaml as these contain example data.
  # See https://backstage.io/docs/features/software-catalog/software-catalog-overview#adding-components-to-the-catalog for more details
  # on how to get entities into the catalog.
  locations: []

I’m still running in the error:

2022-10-24T19:39:55.297434+00:00 app[web.1]: {"level":"info","message":"Loaded config from app-config.yaml, app-config.production.yaml","service":"backstage"}
2022-10-24T19:39:55.307329+00:00 app[web.1]: {"level":"info","message":"1 secret found in the config which will be redacted","service":"backstage"}
2022-10-24T19:39:55.322286+00:00 app[web.1]: {"level":"info","message":"Created UrlReader predicateMux{readers=azure{host=dev.azure.com,authed=false},bitbucketCloud{host=bitbucket.org,authed=false},github{host=github.com,authed=false},gitlab{host=gitlab.com,authed=false},awsS3{host=amazonaws.com,authed=false},fetch{}","service":"backstage"}
2022-10-24T19:39:55.347242+00:00 app[web.1]: {"level":"info","message":"Performing database migration","plugin":"catalog","service":"backstage","type":"plugin"}
2022-10-24T19:39:55.463702+00:00 app[web.1]: Backend failed to start up Error: The migration directory is corrupt, the following files are missing: 20200511113813_init.js, 20200520140700_location_update_log_table.js, 20200527114117_location_update_log_latest_view.js, 20200702153613_entities.js, 20200721115244_location_update_log_latest_deduplicate.js, 20200805163904_location_update_log_duplication_fix.js, 20200807120600_entitySearch.js, 20200809202832_add_bootstrap_location.js, 20200923104503_case_insensitivity.js, 20201005122705_add_entity_full_name.js, 20201006130744_entity_data_column.js, 20201006203131_entity_remove_redundant_columns.js, 20201007201501_index_entity_search.js, 20201019130742_add_relations_table.js, 20201123205611_relations_table_uniq.js, 20201210185851_fk_index.js, 20201230103504_update_log_varchar.js, 20210209121210_locations_fk_index.js, 20210302150147_refresh_state.js, 20210622104022_refresh_state_location_key.js, 20210813143113_add_refresh_state_hash.js, 20210925102509_add_refresh_state_input_hash.js, 20220116144621_remove_legacy.js, 20220222164811_reprocess_for_relation_refs.js, 20220616202842_refresh_keys.js
2022-10-24T19:39:55.463708+00:00 app[web.1]: at validateMigrationList (/app/node_modules/knex/lib/migrations/migrate/Migrator.js:566:11)
2022-10-24T19:39:55.463708+00:00 app[web.1]: at Migrator.latest (/app/node_modules/knex/lib/migrations/migrate/Migrator.js:69:7)

JoostSaanen on Oct 24, 2022

Hi, could we have some help and guidance from the backstage team ? Looks like a lot of us are unable to make it work with heroku postgres databases.

Regards

kwent on Dec 8, 2022

Gotcha. Is there any easy way to find what plugins need databases/what databases they will create automatically so I can create them by hand?

I managed to get it working by setting the combination of the backend.database.ensureExists: false and the backend.database.pluginDivisonMode: 'schema' configuration settings. It’d still be nice to have the maintenance database that is used when connecting to the admin database in that code adhere to the database configuration property or the database specified in the connection string.

Thanks for your help 😃

arhill05 on May 9, 2022

Hm. I wonder if the problem here is actually that you shouldn’t specify a logical database name at all. Every single plugin has their own logical database, isolated from each other.

I can’t omit specifying a database due to security policies at my workplace - I can’t use the default postgres database for the maintenance database.

Besides, I can see the plugin databases being created and populated, so I know that part is working. I’ve also tried both the backend.database.pluginDivisionMode and backend.database.ensureExists configuration options to no avail.

arhill05 on May 9, 2022