splunk-sdk-python: splunklib.binding.AuthenticationError: Autologin succeeded, but there was an auth error on next request. Something is very wrong.

Version of the project : splunk-sdk==1.6.5
Platform version : Ubuntu 16.04.3 LTS x86_64
Framework version : Python 3.5.2
Splunk version : Splunk Enterprise 7.1.1

Code that produces the exception :

import splunklib.client as client
import splunklib.results as results
service = client.connect(host='xx', username='xx', password='xx', autologin=True)

search_oneshot = 'search index=xx sourcetype=xx .......'
kwargs_export = {"earliest_time": "-4Hr",
                 "latest_time": "now",
                 "search_mode": "normal"}

oneshot_results = service.jobs.oneshot(search_oneshot, **kwargs_export)
reader = results.ResultsReader(oneshot_results)
for item in reader:
        print(item)

This simple script executes sometimes without any issue. But when not, it produces :

Traceback (most recent call last):
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 289, in wrapper
    return request_fun(self, *args, **kwargs)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 71, in new_f
    val = f(*args, **kwargs)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 742, in post
    response = self.http.post(path, all_headers, **query)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 1208, in post
    return self.request(url, message)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 1228, in request
    raise HTTPError(response)
splunklib.binding.HTTPError: HTTP 401 Unauthorized -- call not properly authenticated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 231, in _handle_auth_error
    yield
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 300, in wrapper
    return request_fun(self, *args, **kwargs)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 71, in new_f
    val = f(*args, **kwargs)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 742, in post
    response = self.http.post(path, all_headers, **query)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 1208, in post
    return self.request(url, message)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 1228, in request
    raise HTTPError(response)
splunklib.binding.HTTPError: HTTP 401 Unauthorized -- call not properly authenticated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "splunk_ex.py", line 10, in <module>
    oneshot_results = service.jobs.oneshot(search_oneshot, **kwargs_export)
  File "path_to_env/lib/python3.5/site-packages/splunklib/client.py", line 3041, in oneshot
    **params).body
  File "path_to_env/lib/python3.5/site-packages/splunklib/client.py", line 808, in post
    return self.service.post(path, owner=owner, app=app, sharing=sharing, **query)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 300, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "path_to_env/lib/python3.5/site-packages/splunklib/binding.py", line 234, in _handle_auth_error
    raise AuthenticationError(msg, he)
splunklib.binding.AuthenticationError: Autologin succeeded, but there was an auth error on next request. Something is very wrong.

I am not able to predict when it fails and when not. Its pretty random. Am I hitting the resource limit or is it something else?

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 19 (5 by maintainers)

Commits related to this issue

fixes: https://github.com/splunk/splunk-sdk-python/issues/219 — committed to demisto/content by deleted user 3 years ago
fixes: https://github.com/splunk/splunk-sdk-python/issues/219 — committed to demisto/content by deleted user 3 years ago
fixes: https://github.com/splunk/splunk-sdk-python/issues/219 — committed to demisto/content by deleted user 3 years ago

Most upvoted comments

Just an update if anyone else runs into this and has the same problem that I did… while it’s not documented, the connect() method, and the underlying Service() and Context() class constructors, do take a basic=True keyword argument. This fixed it for me. It seems like, just like the Ruby SDK, this completely ignores cookies set by Splunk which breaks the load balancing.

jantman on Apr 9, 2020

I’ve just now run into this issue and the basic=True trick does not work for me. There are several calls to the Splunk service in my Python script, and it is random which call it fails on.

Frechetta on May 7, 2020

I just ran into this issue myself… a little sad that there hasn’t been any comment here in a year and a half. I’m still debugging it, but is there a chance that your Splunk instance is load balanced?

I hit this identical issue with the Ruby SDK, and it turned out that my company’s Splunk Cloud instance (i.e. https://mycompany.splunkcloud.com:8089) is a load balanced cluster, but the Splunk Ruby SDK was ignoring the session stickyness cookies it returns… so auth would work on any requests that happened to be load balanced to the same cluster node as the initial login, or fail on all others. The Ruby SDK has a basic option on the connection to force it to send basic auth with every request to avoid this, but the Python SDK doesn’t seem to have that option.

jantman on Apr 9, 2020