timesketch: Large plaso files upload failure

Describe the bug Upload fails using timesketch_import_client when uploading a large plaso file.

To Reproduce Steps to reproduce the behavior:

  1. upload to timesketch using timesketch_import_client a plaso file larger than (3gb) preumblay the upload will fail when the file size exceeds 999999999 bytes

Expected behavior The plaso file should be correctly uploaded and then processing and indexing of the file should start

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS:Ubuntu 20.04.2 LTS
  • Browser : Firefox
  • Version: 86.0

Additional context

  • Timesketch docker version:latest
  • wsgi_error.log:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 458, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 573, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/timesketch/api/v1/resources/upload.py", line 415, in post
    return self._upload_file(
  File "/usr/local/lib/python3.8/dist-packages/timesketch/api/v1/resources/upload.py", line 355, in _upload_file
    return self._upload_and_index(
  File "/usr/local/lib/python3.8/dist-packages/timesketch/api/v1/resources/upload.py", line 206, in _upload_and_index
    db_session.commit()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/scoping.py", line 162, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1036, in commit
    self.transaction.commit()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 503, in commit
    self._prepare_impl()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 482, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2479, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2617, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2577, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
    persistence.save_obj(
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
    _emit_insert_statements(
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1136, in _emit_insert_statements
    result = cached_connections[connection].execute(
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1095, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1245, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 581, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.NumericValueOutOfRange) integer out of range

[SQL: INSERT INTO datasource (created_at, updated_at, timeline_id, user_id, provider, context, file_on_disk, file_size, original_filename, data_label) VALUES (now(), now(), %(timeline_id)s, %(user_id)s, %(provider)s, %(context)s, %(file_on_disk)s, %(file_size)s, %(original_filename)s, %(data_label)s) RETURNING datasource.id]
[parameters: {'timeline_id': 8, 'user_id': 1, 'provider': 'Imported via the importer library.', 'context': 'N/A', 'file_on_disk': '/usr/share/timesketch/upload/7894e7b9a77d43c3bca8511561395f84', 'file_size': 3203993600, 'original_filename': '*****', 'data_label': 'plaso'}]

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (20 by maintainers)

Most upvoted comments

ok, so according to: https://www.postgresql.org/docs/9.1/datatype-numeric.html

we are using an Integer value for the file size in the data source model, which means from: -2147483648 to +2147483647

probably it would be better to designate this field as either a bigint, or as a decimal.