ArchiveBox: Bugfix: sqlite3.IntegrityError: NOT NULL constraint failed: core_archiveresult.cmd_version and .output
Describe the bug
Cannot start archivebox after moving from nikisweeting/archivebox to archivebox/archivebox. Django/Sqlite error ensues.
Steps to reproduce
Might be related to #596 ? I killed my docker container running nikisweeting/archivebox:latest and tried recreating a container with archivebox/archivebox:latest, linking to the same old volume. I did prepend the server command with archivebox init in order to run 3 new migrations. The container fails to start, issuing the following error.
Looks like this is the migration that not going through, which might point at a corrupted/faulty json index file?
Screenshots or log output
[*] Verifying main SQL index and running migrations...
√ /data/index.sqlite3
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: NOT NULL constraint failed: core_archiveresult.cmd_version
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python3.9/site-packages/django/db/migrations/migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
self.code(from_state.apps, schema_editor)
File "/app/archivebox/core/migrations/0007_archiveresult.py", line 33, in forwards_func
ArchiveResult.objects.create(extractor=extractor, snapshot=snapshot, cmd=result["cmd"], cmd_version=result["cmd_version"],
File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 447, in create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 753, in save
self.save_base(using=using, force_insert=force_insert,
File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 790, in save_base
updated = self._save_table(
File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 895, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 933, in _do_insert
return manager._insert(
File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1254, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: NOT NULL constraint failed: core_archiveresult.cmd_version
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/archivebox", line 33, in <module>
sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')())
File "/app/archivebox/cli/__init__.py", line 123, in main
run_subcommand(
File "/app/archivebox/cli/__init__.py", line 63, in run_subcommand
module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore
File "/app/archivebox/cli/archivebox_init.py", line 33, in main
init(
File "/app/archivebox/util.py", line 113, in typechecked_function
return func(*args, **kwargs)
File "/app/archivebox/main.py", line 323, in init
for migration_line in apply_migrations(out_dir):
File "/app/archivebox/util.py", line 113, in typechecked_function
return func(*args, **kwargs)
File "/app/archivebox/index/sql.py", line 102, in apply_migrations
call_command("migrate", interactive=False, stdout=out)
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 168, in call_command
return command.execute(*args, **defaults)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 85, in wrapped
res = handle_func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 243, in handle
post_migrate_state = executor.migrate(
File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 229, in apply_migration
migration_recorded = True
File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/schema.py", line 35, in __exit__
self.connection.check_constraints()
File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 324, in check_constraints
violations = cursor.execute('PRAGMA foreign_key_check').fetchall()
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 78, in _execute
self.db.validate_no_broken_transaction()
File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 447, in validate_no_broken_transaction
raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
Software versions
- OS: Docker on Synology NAS
- ArchiveBox version: archivebox/archivebox:latest
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (8 by maintainers)
Pulled dev, built, and ran init. No errors thrown, and successful! I tested with
docker-compose up -dand checking in browser. There are some weird things here and there, like main index showing 0 sources for a link, but navigating to a single link’s index shows that it has all sources intact. But I’ll file a separate bug report for that, as this issue is resolved! Thanks again @pirate and @cdvv7788 for the outstanding work and support.Here is one with user/drive snipped out:
Here is another:
Let me know if you need any more, and much appreciated!
This is fixed in the next minor patch version v0.5.4 (not yet out). If you want it early you can build it by doing
git checkout dev; git pull; docker build . -t archivebox.@coisnepe while impressive, in the future I advise against running manual migrations like that, because it may be hard to install v0.5.4 without conflicting with your manual migration 😉
You may be able to figure out how to apply v0.5.4 without breaking anything, but if not just lemme know and I can probably help you apply v0.5.4 on top of your manual fix.
@pirate https://github.com/ArchiveBox/ArchiveBox/pull/599 I gave it a default. It should work unless there are
cmd_versionfields with explicit nulls.@cdvv7788 can you make that field nullable, I suspected we might have some old archive users with no cmd versions in their detail json files.