wagtail: null value in column "draft_title" violates not-null constraint
Issue Summary
Running unit tests in our local and CI (Docker) environments, results in the following exception:
psycopg2.errors.NotNullViolation: null value in column "draft_title" violates not-null constraint
DETAIL: Failing row contains (15, 0001, 1, 1, Root, root, t, f, /, , f, , null, null, f, 1, null, f, null, null, null, null, null, null, null).
Notes
- The DJango setup comprises apps and test-suites that were written prior to the introduction of Wagtail as its own app.
- Wagtail is running off its own DB connector which currently just points to the “default” connection but I have observed the same problem when connecting to an entirely separate schema with the same connection name (in Postgres)
- I’m suspecting that migrations are probably being run out of order. But seeing that DJango migrations are auto-generated then they should therefore “know best”, we cannot easily alter their order to some arbitrary definition
Steps to Reproduce
Running unit tests using BasicNoseRunner we see:
#> docker-compose exec my-app bin/runner test --noinput --verbosity=3 --config=nose.cfg
DefaultPluginManager load plugin xcover = nosexcover.nosexcover:XCoverage
DefaultPluginManager load plugin nose_ignoredoc = nose_ignoredoc:IgnoreDocstrings
nosetests --config=/usr/share/my-app/nose.cfg --verbosity=3 --config=nose.cfg
DefaultPluginManager load plugin xcover = nosexcover.nosexcover:XCoverage
DefaultPluginManager load plugin nose_ignoredoc = nose_ignoredoc:IgnoreDocstrings
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.plugins.xcover: INFO: Coverage report will include only packages: ['app', 'app']
nose.plugins.cover: INFO: Coverage report will include only packages: ['app', 'app']
Creating test database for alias 'default' ('test_foo')...
Got an error creating the test database: database "test_foo" already exists
Destroying old test database for alias 'default' ('test_foo')...
Operations to perform:
Synchronize unmigrated apps: admindocs, api, ckeditor, django_extensions, django_filters, django_nose, humanize, invites, manage, messages, modelcluster, personalgroups, pipeline, rest_framework, settings, staticfiles, wagtailapi_v2, wagtailsites, wagtailsnippets, wizard
Apply all migrations: accounts, admin, auth, cms, contenttypes, eventlog, sessions, sites, switch, taggit, wagtailadmin, wagtailcore, wagtaildocs, wagtailembeds, wagtailforms, wagtailimages, wagtailredirects, wagtailsearch, wagtailsearchpromotions, wagtailusers
Running pre-migrate handlers for application contenttypes
Running pre-migrate handlers for application auth
Running pre-migrate handlers for application sessions
Running pre-migrate handlers for application sites
...
...
Running pre-migrate handlers for application wagtailforms
Running pre-migrate handlers for application wagtailredirects
Running pre-migrate handlers for application wagtailembeds
Running pre-migrate handlers for application wagtailusers
Running pre-migrate handlers for application wagtailsnippets
Running pre-migrate handlers for application wagtaildocs
Running pre-migrate handlers for application wagtailimages
Running pre-migrate handlers for application wagtailsearch
Running pre-migrate handlers for application wagtailsearchpromotions
Running pre-migrate handlers for application wagtailadmin
Running pre-migrate handlers for application wagtailcore
Running pre-migrate handlers for application cms
Running pre-migrate handlers for application modelcluster
Running pre-migrate handlers for application django_extensions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Running migrations:
Applying contenttypes.0001_initial... OK (0.034s)
Applying auth.0001_initial... OK (0.164s)
...
...
Applying contenttypes.0002_remove_content_type_name... OK (0.100s)
Applying auth.0002_alter_permission_name_max_length... OK (0.013s)
Applying auth.0003_alter_user_email_max_length... OK (0.039s)
Applying auth.0004_alter_user_username_opts... OK (0.031s)
Applying auth.0005_alter_user_last_login_null... OK (0.033s)
Applying auth.0006_require_contenttypes_0002... OK (0.003s)
Applying auth.0007_alter_validators_add_error_messages... OK (0.035s)
Applying auth.0008_alter_user_username_max_length... OK (0.036s)
Applying auth.0009_alter_user_last_name_max_length... OK (0.034s)
Applying auth.0010_alter_group_name_max_length... OK (0.023s)
Applying auth.0011_update_proxy_permissions... OK (0.080s)
Applying wagtailcore.0001_squashed_0016_change_page_url_path_to_text_field...Traceback (most recent call last):
File "/opt/virtual-environment/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.NotNullViolation: null value in column "draft_title" violates not-null constraint
DETAIL: Failing row contains (15, 0001, 1, 1, Root, root, t, f, /, , f, , null, null, f, 1, null, f, null, null, null, null, null, null, null).
Relevant Info:
- DJango 2.2 / Wagtail 2.8 / docker-compose 1.23.2
- Similar description to this SO post (which suggests the underlying problem was fixed?) https://stackoverflow.com/questions/51985358/wagtail-running-tests-raises-psycopg2-integrityerror-null-value-in-column-dr#51985754
- Using x2
DATABASEconnections “default” and “cms” with simpleDATABASE_ROUTERSimplementation:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': fetch_config(config.get, "database", "name"),
'USER': fetch_config(config.get, "database", "user"),
'HOST': fetch_config(config.get, "database", "host"),
'PASSWORD': fetch_config(config.get, "database", "password")
},
# "cms" is just a dumb DB connection back to the default DB. We do this to make it easier in future
# to modify Puppet/Ansible config and point credentials/vars at a separate DB
'cms': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': fetch_config(config.get, "database", "name"),
'USER': fetch_config(config.get, "database", "user"),
'HOST': fetch_config(config.get, "database", "host"),
'PASSWORD': fetch_config(config.get, "database", "password")
},
}
import re, sys
class DBRouter():
"""
Switch to the appropriate DB connection based on a model's `app_label`.
Note: Migrations will default to using the "default" connection in DATABASES unless
`--database <name>` is passed.
"""
def _is_cms(self, model):
return re.match(r"^wagtail", model._meta.app_label)
def db_for_read(self, model, **hints):
if self._is_cms(model):
return 'cms'
return 'default'
def db_for_write(self, model, **hints):
if self._is_cms(model):
return 'cms'
return 'default'
def allow_relation(self, obj1, obj2, **hints):
return True
def allow_migrate(self, db, app_label, model_name=None, **hints):
return True
Any other relevant information. For example, why do you consider this a bug and what did you expect to happen instead?
- I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: “Yes”
Technical details
- Python version: 3.6
- Django version: 2.2
- Wagtail version: 2.8
- Browser version: Brave
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (6 by maintainers)
Hey! If you’re having the same specific scenario as mine where I need to migrate the test database, i just reconfigured the
DATABASESdict to set thedefaultto mytest_database.Did that, called
./manage.py migrate, and then it migrated successfully.@gasman, sorry for replying to a closed issue but I believe this is not the case. According to the docs, a router determines which connection is used for read or write operations and whether a relation is allowed between two objects. The
allow_migrate()method simply allows or denies a migration based on its arguments.The
--database=somedatabaseargument formanage.py migratecan be easily supported in migrationRunPythonmethods (see theRunPythondocs) by usingschema_editor.connection.alias(schema_editoris the 2nd argument for the method provided toRunPython) like (example from the docs):So I would suggest that at the minimum the
RunPythonmethods should simply operate onschema_editor.connection.aliasinstead of ‘hard coded’ on the default connection.Of course this doesn’t automatically propagate to underlying methods but for the
initial_data()method fromcore.0001_squashed_0016_change_page_url_path_to_text_field.pyit would simply mean adding the.using()call on the querysets:The “hints” suggested in #2234 are not a solution for simply allowing developers to run migrations on a specific connection/alias.
I’m currently working on a workaround for my current project where I want to create a ‘template’ testing database next to my production database but it would help ALOT if Wagtail’s migrations would simply use the
--databaseargument in the migrations.Regards!
Thanks for the additional details. It sounds like this is a duplicate of #2234 - essentially, if you want to use multiple databases, then it’s your responsibility to set up a database router to define how models are divided or replicated between those databases. In this case, the errors are happening because the migrations are running for a second time but performing insertions against the first database.
It may well be that Wagtail isn’t currently doing everything it should be doing to support multi-database setups, but if all we know is “there are some multi-database configurations that fail” then we have no way to tell whether that’s user error or a Wagtail bug. What we’ve been missing up to now is someone with enough familiarity with multi-DB setups to definitively say “with router config X, A should be happening but Wagtail is doing B instead”.