sqlalchemy: Bulk insert on conflict in SQLite fails for missing columns in some rows, works on mysql

Thanks for the really fast reply. Interestingly this is a difference between SQLite and MySQL dialects. Bulk upsert in MySQL (see discussion https://github.com/sqlalchemy/sqlalchemy/discussions/9328) works fine. I worked up a SSCCE for this also:

import sqlalchemy as db
import sqlalchemy.dialects.sqlite as sqlite
from sqlalchemy import delete, select, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = 'user'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(30), nullable=False)
    count: Mapped[int] = mapped_column(nullable=True)


engine = db.create_engine('sqlite:///:memory:')
conn = engine.connect()

# setup step 0 - ensure the table exists
Base().metadata.create_all(bind=engine)

# setup step 1 - clean out rows with id 1..5
del_stmt = delete(User).where(User.id.in_([1, 2, 3, 4, 5]))
conn.execute(del_stmt)
conn.commit()
sel_stmt = select(User)
users = list(conn.execute(sel_stmt))
print(f'Table size after cleanout: {len(users)}')

# setup step 2 - insert 4 rows
ins_stmt = sqlite.insert(User).values(
    [
        {'id': 1, 'name': 'x', 'count': 10},
        {'id': 2, 'name': 'y', 'count': 20},
        {'id': 3, 'name': 'w'}, # no count
        {'id': 4, 'name': 'z', 'count': 40},
    ]
)
# FAILS HERE IF COUNT IS MISSING
conn.execute(ins_stmt)
conn.commit()
users = list(conn.execute(sel_stmt))
print(f'Table size after insert: {len(users)}')

# demonstrate upsert
ups_stmt = sqlite.insert(User).values(
    [
        {'id': 1, 'name': 'xx', 'count': 10},
        {'id': 2, 'name': 'yy', 'count': 20},
        {'id': 3, 'name': 'ww'}, # no count
        {'id': 5, 'name': 'new', 'count': 50},
    ]
)
ups_stmt = ups_stmt.on_conflict_do_update(
    index_elements=['id'],
    set_=dict(
        name=ups_stmt.excluded.name,
        count=ups_stmt.excluded.count))
# if you want to see the compiled result
x = ups_stmt.compile(dialect=sqlite.dialect())
print(x.string, x.construct_params())
conn.execute(ups_stmt)
conn.commit()

users = list(conn.execute(sel_stmt))
print(f'Table size after upsert: {len(users)}')

_Originally posted by @chrisinmtown in https://github.com/sqlalchemy/sqlalchemy/discussions/9702#discussioncomment-5733073_

Error:

sqlalchemy.exc.CompileError: INSERT value for column ip_time.time is explicitly rendered as a boundparameter in the VALUES clause;
a Python-side value or SQL expression is required

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 16 (13 by maintainers)

Most upvoted comments

confirmed mysql.insert().values( one parameter dict missing a key ) fails with the same error.