wagtail: null value in column "draft_title" violates not-null constraint

Issue Summary

Running unit tests in our local and CI (Docker) environments, results in the following exception:

psycopg2.errors.NotNullViolation: null value in column "draft_title" violates not-null constraint
DETAIL:  Failing row contains (15, 0001, 1, 1, Root, root, t, f, /, , f, , null, null, f, 1, null, f, null, null, null, null, null, null, null).

Notes

  • The DJango setup comprises apps and test-suites that were written prior to the introduction of Wagtail as its own app.
  • Wagtail is running off its own DB connector which currently just points to the “default” connection but I have observed the same problem when connecting to an entirely separate schema with the same connection name (in Postgres)
  • I’m suspecting that migrations are probably being run out of order. But seeing that DJango migrations are auto-generated then they should therefore “know best”, we cannot easily alter their order to some arbitrary definition

Steps to Reproduce

Running unit tests using BasicNoseRunner we see:

#> docker-compose exec my-app bin/runner test --noinput --verbosity=3 --config=nose.cfg
DefaultPluginManager load plugin xcover = nosexcover.nosexcover:XCoverage
DefaultPluginManager load plugin nose_ignoredoc = nose_ignoredoc:IgnoreDocstrings
nosetests --config=/usr/share/my-app/nose.cfg --verbosity=3 --config=nose.cfg
DefaultPluginManager load plugin xcover = nosexcover.nosexcover:XCoverage
DefaultPluginManager load plugin nose_ignoredoc = nose_ignoredoc:IgnoreDocstrings
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.plugins.xcover: INFO: Coverage report will include only packages: ['app', 'app']
nose.plugins.cover: INFO: Coverage report will include only packages: ['app', 'app']
Creating test database for alias 'default' ('test_foo')...
Got an error creating the test database: database "test_foo" already exists

Destroying old test database for alias 'default' ('test_foo')...
Operations to perform:
  Synchronize unmigrated apps: admindocs, api, ckeditor, django_extensions, django_filters, django_nose, humanize, invites, manage, messages, modelcluster, personalgroups, pipeline, rest_framework, settings, staticfiles, wagtailapi_v2, wagtailsites, wagtailsnippets, wizard
  Apply all migrations: accounts, admin, auth, cms, contenttypes, eventlog, sessions, sites, switch, taggit, wagtailadmin, wagtailcore, wagtaildocs, wagtailembeds, wagtailforms, wagtailimages, wagtailredirects, wagtailsearch, wagtailsearchpromotions, wagtailusers
Running pre-migrate handlers for application contenttypes
Running pre-migrate handlers for application auth
Running pre-migrate handlers for application sessions
Running pre-migrate handlers for application sites
...
...
Running pre-migrate handlers for application wagtailforms
Running pre-migrate handlers for application wagtailredirects
Running pre-migrate handlers for application wagtailembeds
Running pre-migrate handlers for application wagtailusers
Running pre-migrate handlers for application wagtailsnippets
Running pre-migrate handlers for application wagtaildocs
Running pre-migrate handlers for application wagtailimages
Running pre-migrate handlers for application wagtailsearch
Running pre-migrate handlers for application wagtailsearchpromotions
Running pre-migrate handlers for application wagtailadmin
Running pre-migrate handlers for application wagtailcore
Running pre-migrate handlers for application cms
Running pre-migrate handlers for application modelcluster
Running pre-migrate handlers for application django_extensions
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  Applying contenttypes.0001_initial... OK (0.034s)
  Applying auth.0001_initial... OK (0.164s)
  ...
  ...
  Applying contenttypes.0002_remove_content_type_name... OK (0.100s)
  Applying auth.0002_alter_permission_name_max_length... OK (0.013s)
  Applying auth.0003_alter_user_email_max_length... OK (0.039s)
  Applying auth.0004_alter_user_username_opts... OK (0.031s)
  Applying auth.0005_alter_user_last_login_null... OK (0.033s)
  Applying auth.0006_require_contenttypes_0002... OK (0.003s)
  Applying auth.0007_alter_validators_add_error_messages... OK (0.035s)
  Applying auth.0008_alter_user_username_max_length... OK (0.036s)
  Applying auth.0009_alter_user_last_name_max_length... OK (0.034s)
  Applying auth.0010_alter_group_name_max_length... OK (0.023s)
  Applying auth.0011_update_proxy_permissions... OK (0.080s)
  Applying wagtailcore.0001_squashed_0016_change_page_url_path_to_text_field...Traceback (most recent call last):
  File "/opt/virtual-environment/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.NotNullViolation: null value in column "draft_title" violates not-null constraint
DETAIL:  Failing row contains (15, 0001, 1, 1, Root, root, t, f, /, , f, , null, null, f, 1, null, f, null, null, null, null, null, null, null).

Relevant Info:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': fetch_config(config.get, "database", "name"),
        'USER': fetch_config(config.get, "database", "user"),
        'HOST': fetch_config(config.get, "database", "host"),
        'PASSWORD': fetch_config(config.get, "database", "password")
    },
    # "cms" is just a dumb DB connection back to the default DB. We do this to make it easier in future
    # to modify Puppet/Ansible config and point credentials/vars at a separate DB
    'cms': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': fetch_config(config.get, "database", "name"),
        'USER': fetch_config(config.get, "database", "user"),
        'HOST': fetch_config(config.get, "database", "host"),
        'PASSWORD': fetch_config(config.get, "database", "password")
    },
}
import re, sys


class DBRouter():
    """
    Switch to the appropriate DB connection based on a model's `app_label`.

    Note: Migrations will default to using the "default" connection in DATABASES unless
    `--database <name>` is passed.
    """

    def _is_cms(self, model):
        return re.match(r"^wagtail", model._meta.app_label)

    def db_for_read(self, model, **hints):
        if self._is_cms(model):
            return 'cms'

        return 'default'

    def db_for_write(self, model, **hints):
        if self._is_cms(model):
            return 'cms'

        return 'default'

    def allow_relation(self, obj1, obj2, **hints):
        return True

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        return True

Any other relevant information. For example, why do you consider this a bug and what did you expect to happen instead?

  • I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: “Yes”

Technical details

  • Python version: 3.6
  • Django version: 2.2
  • Wagtail version: 2.8
  • Browser version: Brave

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 16 (6 by maintainers)

Commits related to this issue

Most upvoted comments

Hey! If you’re having the same specific scenario as mine where I need to migrate the test database, i just reconfigured the DATABASES dict to set the default to my test_database.

Did that, called ./manage.py migrate, and then it migrated successfully.

@eeintech I believe so, yes.

@gasman, sorry for replying to a closed issue but I believe this is not the case. According to the docs, a router determines which connection is used for read or write operations and whether a relation is allowed between two objects. The allow_migrate() method simply allows or denies a migration based on its arguments.

The --database=somedatabase argument for manage.py migrate can be easily supported in migration RunPython methods (see the RunPython docs) by using schema_editor.connection.alias (schema_editor is the 2nd argument for the method provided to RunPython) like (example from the docs):

def forwards_func(apps, schema_editor):
    # We get the model from the versioned app registry;
    # if we directly import it, it'll be the wrong version
    Country = apps.get_model("myapp", "Country")
    db_alias = schema_editor.connection.alias  # <- the alias passed in via --database
    Country.objects.using(db_alias).bulk_create([
        Country(name="USA", code="us"),
        Country(name="France", code="fr"),
    ])

So I would suggest that at the minimum the RunPython methods should simply operate on schema_editor.connection.alias instead of ‘hard coded’ on the default connection.

Of course this doesn’t automatically propagate to underlying methods but for the initial_data() method from core.0001_squashed_0016_change_page_url_path_to_text_field.py it would simply mean adding the .using() call on the querysets:

db_alias = schema_editor.connection.alias
root = Page.objects.using(db_alias).create(
    ...

The “hints” suggested in #2234 are not a solution for simply allowing developers to run migrations on a specific connection/alias.

I’m currently working on a workaround for my current project where I want to create a ‘template’ testing database next to my production database but it would help ALOT if Wagtail’s migrations would simply use the --database argument in the migrations.

Regards!

Thanks for the additional details. It sounds like this is a duplicate of #2234 - essentially, if you want to use multiple databases, then it’s your responsibility to set up a database router to define how models are divided or replicated between those databases. In this case, the errors are happening because the migrations are running for a second time but performing insertions against the first database.

It may well be that Wagtail isn’t currently doing everything it should be doing to support multi-database setups, but if all we know is “there are some multi-database configurations that fail” then we have no way to tell whether that’s user error or a Wagtail bug. What we’ve been missing up to now is someone with enough familiarity with multi-DB setups to definitively say “with router config X, A should be happening but Wagtail is doing B instead”.