opa: sprintf panics when given malformed format strings and a non-zero arguments list
Short description
sprintf
now can panic when given malformed format strings. However, this is a difficult panic
to invoke outside of direct use of the parser/evaluation functions in Golang.
Examples:
- OPA version: edge (commit
50b79b92fbc99c, several PRs after thev0.41.0
tag) - Example output that OPA returned:
runtime error: index out of range [0] with length 0
Steps To Reproduce
- Add a new test case to the end of
test/cases/testdata/sprintf/test-sprintf.yaml
:
- data:
modules:
- |
package test
p = x {
exampleval:= 0
x = sprintf("%+v",[exampleval])
}
note: sprintf/array/regression 1
query: data.test.p = x
want_result:
- x: "0"
- Observe the
panic
:
--- FAIL: TestRego (1.13s)
--- FAIL: TestRego/sprintf/array/regression_1 (0.00s)
panic: runtime error: index out of range [0] with length 0 [recovered]
panic: runtime error: index out of range [0] with length 0
goroutine 1124 [running]:
testing.tRunner.func1.2({0xb39580, 0xc0007fcc78})
/usr/local/go/src/testing/testing.go:1389 +0x24e
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1392 +0x39f
panic({0xb39580, 0xc0007fcc78})
/usr/local/go/src/runtime/panic.go:838 +0x207
github.com/open-policy-agent/opa/topdown.builtinSprintf({0xcbd418?, 0xc000dfdfe0?}, {0xcbcf18?, 0xc000c2d140?})
/github-devel-folder/opa/topdown/strings.go:507 +0x41c
github.com/open-policy-agent/opa/topdown.functionalWrapper2.func1({{0xcbc150, 0xc0000240c0}, {0xcbe1a0, 0xc0008114e0}, {0xcb77e0, 0xc00007e2d0}, 0xc00075f710, {0x0, 0x0}, 0x0, ...}, ...)
/github-devel-folder/opa/topdown/builtins.go:146 +0x73
github.com/open-policy-agent/opa/topdown.evalBuiltin.eval({0xc00109b4a0, 0x10aaa00, {{0xcbc150, 0xc0000240c0}, {0xcbe1a0, 0xc0008114e0}, {0xcb77e0, 0xc00007e2d0}, 0xc00075f710, {0x0, ...}, ...}, ...}, ...)
/github-devel-folder/opa/topdown/eval.go:1686 +0x33f
github.com/open-policy-agent/opa/topdown.(*eval).evalCall(0xc00109b4a0, {0xc0002e0ab0, 0x4, 0x6}, 0xc0002fcc60)
/github-devel-folder/opa/topdown/eval.go:813 +0xa85
github.com/open-policy-agent/opa/topdown.(*eval).evalStep(0xc00109b4a0, 0xc000d70d50)
/github-devel-folder/opa/topdown/eval.go:358 +0x75c
github.com/open-policy-agent/opa/topdown.(*eval).evalExpr(0xc00109b4a0, 0xc0002fb7a0)
/github-devel-folder/opa/topdown/eval.go:332 +0xec
github.com/open-policy-agent/opa/topdown.(*eval).next(...)
/github-devel-folder/opa/topdown/eval.go:164
github.com/open-policy-agent/opa/topdown.(*eval).evalExpr.func1(0xc00109b4a0)
/github-devel-folder/opa/topdown/eval.go:333 +0x29
github.com/open-policy-agent/opa/topdown.(*eval).evalStep.func1()
/github-devel-folder/opa/topdown/eval.go:353 +0x39
github.com/open-policy-agent/opa/topdown.(*eval).biunifyValues(0xc00109b4a0, 0xc000b85e78, 0xc000b85e90, 0xc0002fcab0, 0xc0002fcab0, 0xc0002fcba0)
...
Expected behavior
sprintf
should not panic
when confronted with malformed format strings; ideally, it should just print them as a normal string, and ignore the arguments list parts that don’t have a 1-to-1 match with the format string.
Bug investigation
Internally, this panic
seems to be caused by an iteration over 2x arrays, where one array’s size is dependent on the length of the array of arguments handed to sprintf
, and the other array’s length is determined by how many formatting args were detected via a regex. In the case of malformed format strings, there can be a mismatch in size between these two arrays, and the sprintf
builtin explodes with a panic
related to array-OOB accesses because one array is shorter than the other. (Link to source location generating the panic
)
Additional context
This bug was uncovered while I was running a backtest suite as part of verifying my changes while trying to fix #4672. The two issues are wholly unrelated, and this issue was only discovered because my feature branch tracked OPA edge, while the reference tests were tracking the v0.41.0
tag. The sudden occurrence of mystery panics helped me find this bug.
This bug appears to have been introduced by PR #4620 (merged after the v0.41.0
release), which substantially changed how sprintf
wrangles its formatting arguments.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16 (15 by maintainers)
Commits related to this issue
- Revert "sprintf: fix formatting of numbers in float/int verbs (#4620)" This reverts commit 2fbc99c324e92026468b5b2afb5361f1c5ddbb57. See #4771 for details. We'll get this back in, but we're backing... — committed to srenatus/opa by srenatus 2 years ago
- Revert "sprintf: fix formatting of numbers in float/int verbs (#4620)" (#4794) This reverts commit 2fbc99c324e92026468b5b2afb5361f1c5ddbb57. See #4771 for details. We'll get this back in, but w... — committed to open-policy-agent/opa by srenatus 2 years ago
Those bitwise ops would be using integers, only, though? I’m wondering if “pick one” with int wouldn’t be the best option here. But that might be my bias never having to use floats much.
I think for now, if there are no objections, the best path forward is to strike
%x
/%X
from the supported verbs list.If someone complains about wanting this kind of formatting support, we can revisit this decision later, and try to do something reasonable. (Easier to drop now and add it back later, than to add something half-baked.)
Thank you for your inputs, @anderseknert and @srenatus!
Just some thoughts from the hammock here, but let’s not forget that options that might be very useful for bitwise operations, aren’t necessarily very useful for policy authoring in general. The number of policies in the wild using the bitwise functions ought to be somewhere close to a handful, and the number of policies also pretty-printing these closer to zero.
The fixes previously made to handling float/int presentation with sprintf however came from real problems reported by end-users. If we need to prioritize one over the other, it should be an easy choice.
Looks like
%T
isn’t as bad as I thought – we’re not leaking golang struct types – but it’s still got some things to fix:It doesn’t need to happen right away, but printing the rego type for %T would be nice.
Currently, it looks like our tests cover:
Verbs:
%d
%f
%s
%v
Verb forms:
%<verb>
(ex:%s
)%<prefix><verb>
(ex:%.1f
)%[<idx>]<verb>
(ex:%[2]d
)%<prefix>[<idx>]<verb>
(ex:%.3[1]f
)