sqlite-utils: CSV files with too many values in a row cause errors

Original title: csv.DictReader can have None as key

In some cases, csv.DictReader can have None as key for unnamed columns, and a list of values as value. sqlite_utils.utils.rows_from_file cannot handle that:

url="https://artsdatabanken.no/Fab2018/api/export/csv"
db = sqlite_utils.Database(":memory")

with urlopen(url) as fab:
    reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")   
    db["fab2018"].insert_all(reader, pk="Id")

Result:

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 2924, in insert_all
    chunk = list(chunk)
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in fix_square_braces
    if any("[" in key or "]" in key for key in record.keys()):
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in <genexpr>
    if any("[" in key or "]" in key for key in record.keys()):
TypeError: argument of type 'NoneType' is not iterable

Code: https://github.com/simonw/sqlite-utils/blob/59be60c471fd7a2c4be7f75e8911163e618ff5ca/sqlite_utils/db.py#L3454

sqlite-utils insert from command line is not affected by this issue.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (20 by maintainers)

Commits related to this issue

Most upvoted comments