connectors: [taxii2] Infinite loop in handling "next" value
Description
This pull request adds code to check “next” value in TAXII 2.1 against a list of available collections.
However, “next” is not a collection, it’s a value that the server provides for pagination based on each server’s individual implementation.
From the TAXII 2.1 docs:
This property identifies the server provided value of the next record or set of records in the paginated data set. This property MAY be populated if the more property is set to true.
This value is opaque to the client and represents something that the server knows how to deal with and process.
For example, for a relational database this could be the index autoID, for elastic search it could be the Scroll ID, for other systems it could be a cursor ID, or it could be any string (or int represented as a string) depending on the requirements of the server and what it is doing in the background.
This causes the connector to enter an infinite loop:
# Assuming newer versions will support next
while True:
objects = process_response(objects, response, version)
# Check if "more" exists in response and its value is True
if "more" in response and response["more"] == True:
filters["next"] = response["next"]
###
### checks for "next" value in list of collections
###
if (
response["next"] in self.available_collections
and len(self.available_collections) > 0
):
response = collection.get_objects(**filters)
###
### "next" value is not a collection so it will continue in the loop
###
###
### "response" is not updated, meaning "more" is still True, so this will never execute to break the loop
###
else:
# "more" doesn't exist or is not True, exit the loop
break
Environment
Reproduced logic with samples of raw taxii2.py
code.
Reproducible Steps
Add TAXII2.1 feed using connector.
About this issue
- Original URL
- State: closed
- Created 6 months ago
- Comments: 21 (21 by maintainers)
Commits related to this issue
- Rolled back 34eb547 as per #1683 — committed to netbroom/connectors by netbroom 6 months ago
- [taxii]Rolled back 34eb547 as per #1683 (#1686) — committed to OpenCTI-Platform/connectors by netbroom 6 months ago
For visibility. https://github.com/davidonzo/Threat-Intel/issues/45
I just spun up the misp feed connector with the digitalside feed and it works out of the box on OpenCTI v5.11.14. I’ll test their taxii server now.
I get the feeling that when you changed the code, the uuid in next might not have been a collection id like you thought, so by not matching, you aren’t pulling the second request and creating the error you were seeing. But this is all hypothetical.
@annoyingapt I created #1686 to apply on top of the latest code in master in the interest of time and so your additional changes would not be lost.