schemathesis: [BUG] False positive for "Accepted negative data"
Checklist
- I checked the FAQ section of the documentation
- I looked for similar issues in the issue tracker
- I am using the latest version of Schemathesis
Describe the bug
After upgrading to the new schemathesis 4.0.1 I was getting new errors related to negative and positive data. I adjusted the code and the openapi spec to handle those cases but there is one that seems a false positive bug.
The project https://github.com/tecnickcom/rndpwd/ schemathesis fails with the following error:
GET /password
1. Test Case ID: 70oXOM
- Accepted negative data
Allowed statuses: 400, 401, 403, 404, 406, 422, 428, 5xx
[200] OK:
`["lnlnlnlulnnlllln","ulnnluunlnlnlnun"]`
Reproduce with:
curl -X GET 'http://rndpwd:8071/password?charset=null&length=16&quantity=2'
The full log can be seen at:
The charset=null is a valid input and the output is as expected.
However, schemathesis treats this case as negative data.
The same issue has been observed with other testing strings like false.
To Reproduce
🚨 Mandatory 🚨: Steps to reproduce the behavior:
Clone the https://github.com/tecnickcom/rndpwd/ project and run make x.
Schemathesis is running with the following parameters:
schemathesis run \
--checks=all \
--request-timeout=2000 \
--max-examples=100 \
--url='${API_TEST_URL}' \
${OPENAPI_FILE}
Please include a minimal API schema causing this issue:
Expected behavior
Pass when the string is null or false if the parameter accepts arbitrary strings.
Environment
- OS: Linux
- Schemathesis version: 4.0.1
- Spec version: Open API 3.0.3
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 1
- Comments: 15 (8 by maintainers)
I just upgraded to 4.0.9 and the issue with ‘API accepted schema-violating request’ flaky test cases seems to be fixed!
@dimitarkovachev
Could you, please, share a VCR cassette for such a test run?
--report=vcr- it contains details what exactly is negated in the test caseI assume that it is the opposite case, when the mutator changes the
typevalue fromintegertostring, and the generated string then has the same representation as a number.