sqlite-utils: CSV files with too many values in a row cause errors
Original title: csv.DictReader can have None as key
In some cases, csv.DictReader can have None as key for unnamed columns, and a list of values as value.
sqlite_utils.utils.rows_from_file cannot handle that:
url="https://artsdatabanken.no/Fab2018/api/export/csv"
db = sqlite_utils.Database(":memory")
with urlopen(url) as fab:
reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")
db["fab2018"].insert_all(reader, pk="Id")
Result:
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 2924, in insert_all
chunk = list(chunk)
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in fix_square_braces
if any("[" in key or "]" in key for key in record.keys()):
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in <genexpr>
if any("[" in key or "]" in key for key in record.keys()):
TypeError: argument of type 'NoneType' is not iterable
sqlite-utils insert from command line is not affected by this issue.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (20 by maintainers)
Commits related to this issue
- rows_from_file(... ignore_extras: bool, restkey: str), refs #440 — committed to simonw/sqlite-utils by simonw 2 years ago
- Tests for rows_from_file, refs #440 — committed to simonw/sqlite-utils by simonw 2 years ago
- mypy fixes, refs #440 — committed to simonw/sqlite-utils by simonw 2 years ago
- One more typing fix, refs #440 — committed to simonw/sqlite-utils by simonw 2 years ago
- flake8 fix, refs #440 — committed to simonw/sqlite-utils by simonw 2 years ago
- Update test for renamed restkey, refs #440, #443 — committed to simonw/sqlite-utils by simonw 2 years ago
- Release 3.27 Refs #434, #435, #436, #440, #441, #442, #443 — committed to simonw/sqlite-utils by simonw 2 years ago
- Release 3.27 Refs #434, #435, #436, #440, #441, #442, #443 — committed to simonw/sqlite-utils by simonw 2 years ago
- Docs for rows_from_file Closes #440, closes #443 — committed to polyrand/sqlite-utils by simonw 2 years ago
Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file