cti-python-stix2: Filtering for un-revoked objects does not work without explicit revoked=False

I use Python 3.6, with the latest github stix2 version If you try to filtre for revoked (or not revoked) objects on FileSystemStore and MemoryStore you obtain errors or incorrect behaviours

Example from stix2 import FileSystemStore, MemoryStore, Filter, AttackPattern fs = FileSystemStore('/Users/andrea/test') mem = MemoryStore() filt_false = Filter('revoked', '=', False) filt_true = Filter('revoked', '=', True) patt1 = AttackPattern(name='test1') patt2 = AttackPattern(name='test2') fs.add(patt1) mem.add(patt2)

fs.query(filt_false) Traceback (most recent call last): File “<stdin>”, line 1, in <module> File “/usr/local/lib/python3.6/site-packages/stix2/sources/init.py”, line 90, in query return self.source.query(*args, **kwargs) File “/usr/local/lib/python3.6/site-packages/stix2/sources/filesystem.py”, line 298, in query stix_obj = json.load(open(os.path.join(root, file_))) File “/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/init.py”, line 296, in load return loads(fp.read(), File “/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/codecs.py”, line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x85 in position 1083: invalid start byte

fs.query(filt_true) Traceback (most recent call last): File “<stdin>”, line 1, in <module> File “/usr/local/lib/python3.6/site-packages/stix2/sources/init.py”, line 90, in query return self.source.query(*args, **kwargs) File “/usr/local/lib/python3.6/site-packages/stix2/sources/filesystem.py”, line 298, in query stix_obj = json.load(open(os.path.join(root, file_))) File “/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/init.py”, line 296, in load return loads(fp.read(), File “/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/codecs.py”, line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x85 in position 1083: invalid start byte

mem.query(filt_false) []

mem.query(filt_true) []

patt1 = patt1.revoke() patt2 = patt2.revoke() fs.add(patt1) mem.add(patt2)

fs.query(filt_true) Traceback (most recent call last): File “<stdin>”, line 1, in <module> File “/usr/local/lib/python3.6/site-packages/stix2/sources/init.py”, line 90, in query return self.source.query(*args, **kwargs) File “/usr/local/lib/python3.6/site-packages/stix2/sources/filesystem.py”, line 298, in query stix_obj = json.load(open(os.path.join(root, file_))) File “/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/init.py”, line 296, in load return loads(fp.read(), File “/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/codecs.py”, line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x85 in position 1083: invalid start byte

mem.query(filt_false) []

mem.query(filt_true) [AttackPattern(type=‘attack-pattern’, id=‘attack-pattern–3e65f786-25a0-471d-8ab8-1929313424a3’, created=‘2018-01-19T09:29:36.104Z’, modified=‘2018-01-19T09:30:49.983Z’, name=‘test2’, revoked=True)]

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 21 (20 by maintainers)

Most upvoted comments

FTR - Dev Meeting 11 April 2018:

  • Proposed simple solution of having the two properties (revoked, sightings.summary) have default values of “False”. This solves the usage bug that this thread was originally started on. But need to think through the effects of doing this for serialize() and python-stix2 output in general.
  • The above solution is to address the issue in the short term before 1.0 release, the deeper issue of the design of the Property object is still open.

(@clenk, @gtback - correct if I misinterpreted)

FTR: after dev meeting (2018 March 28) there was general consensus about:

  • modifying the creation of properties to specific mutually exclusive options (i.e. optional, required, defaulted etc…) for a more clearer, expressive creation pf properties(long-term, especially before 1.0 release)
  • Not doing bookkeeping for tracking of properties that will/will not be serialized/exported but rather allowing a user to specify at some level whether to export “semantically default” properties
  • Further exploration of how many types of STIX properties fit into these corner cases of being optional (from a STIX specification perspective) but semantically defaulted from an API perspective. As Chris noted, he found “revoked” and “Sighting.summary” to be the only instances of optional STIX properties that also had defaults.

(Note Im paraphrasing here and enumerating from memory. So @gtback , @clenk please correct any incorrect statements)