timesketch: Large plaso files upload failure
Describe the bug Upload fails using timesketch_import_client when uploading a large plaso file.
To Reproduce Steps to reproduce the behavior:
- upload to timesketch using timesketch_import_client a plaso file larger than (3gb) preumblay the upload will fail when the file size exceeds 999999999 bytes
Expected behavior The plaso file should be correctly uploaded and then processing and indexing of the file should start
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS:Ubuntu 20.04.2 LTS
- Browser : Firefox
- Version: 86.0
Additional context
- Timesketch docker version:latest
- wsgi_error.log:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 458, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 89, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 573, in dispatch_request
resp = meth(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/timesketch/api/v1/resources/upload.py", line 415, in post
return self._upload_file(
File "/usr/local/lib/python3.8/dist-packages/timesketch/api/v1/resources/upload.py", line 355, in _upload_file
return self._upload_and_index(
File "/usr/local/lib/python3.8/dist-packages/timesketch/api/v1/resources/upload.py", line 206, in _upload_and_index
db_session.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/scoping.py", line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1036, in commit
self.transaction.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 503, in commit
self._prepare_impl()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 482, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2479, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2617, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise
raise value
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2577, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
_emit_insert_statements(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1136, in _emit_insert_statements
result = cached_connections[connection].execute(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1095, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.NumericValueOutOfRange) integer out of range
[SQL: INSERT INTO datasource (created_at, updated_at, timeline_id, user_id, provider, context, file_on_disk, file_size, original_filename, data_label) VALUES (now(), now(), %(timeline_id)s, %(user_id)s, %(provider)s, %(context)s, %(file_on_disk)s, %(file_size)s, %(original_filename)s, %(data_label)s) RETURNING datasource.id]
[parameters: {'timeline_id': 8, 'user_id': 1, 'provider': 'Imported via the importer library.', 'context': 'N/A', 'file_on_disk': '/usr/share/timesketch/upload/7894e7b9a77d43c3bca8511561395f84', 'file_size': 3203993600, 'original_filename': '*****', 'data_label': 'plaso'}]
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 20 (20 by maintainers)
ok, so according to: https://www.postgresql.org/docs/9.1/datatype-numeric.html
we are using an Integer value for the file size in the data source model, which means from:
-2147483648 to +2147483647probably it would be better to designate this field as either a bigint, or as a decimal.