schemathesis: [BUG] False positive for "Accepted negative data"

Checklist

  • I checked the FAQ section of the documentation
  • I looked for similar issues in the issue tracker
  • I am using the latest version of Schemathesis

Describe the bug

After upgrading to the new schemathesis 4.0.1 I was getting new errors related to negative and positive data. I adjusted the code and the openapi spec to handle those cases but there is one that seems a false positive bug.

The project https://github.com/tecnickcom/rndpwd/ schemathesis fails with the following error:

GET /password 
 1. Test Case ID: 70oXOM

- Accepted negative data

    Allowed statuses: 400, 401, 403, 404, 406, 422, 428, 5xx

[200] OK:

    `["lnlnlnlulnnlllln","ulnnluunlnlnlnun"]`

Reproduce with: 

    curl -X GET 'http://rndpwd:8071/password?charset=null&length=16&quantity=2'

The full log can be seen at:

The charset=null is a valid input and the output is as expected. However, schemathesis treats this case as negative data. The same issue has been observed with other testing strings like false.

To Reproduce

🚨 Mandatory 🚨: Steps to reproduce the behavior:

Clone the https://github.com/tecnickcom/rndpwd/ project and run make x.

Schemathesis is running with the following parameters:

schemathesis run \
	--checks=all \
	--request-timeout=2000 \
	--max-examples=100 \
	--url='${API_TEST_URL}' \
	${OPENAPI_FILE}

Please include a minimal API schema causing this issue:

Expected behavior

Pass when the string is null or false if the parameter accepts arbitrary strings.

Environment

- OS: Linux
- Schemathesis version: 4.0.1
- Spec version: Open API 3.0.3

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 1
  • Comments: 15 (8 by maintainers)

Most upvoted comments

I just upgraded to 4.0.9 and the issue with ‘API accepted schema-violating request’ flaky test cases seems to be fixed!

@dimitarkovachev

Could you, please, share a VCR cassette for such a test run? --report=vcr - it contains details what exactly is negated in the test case

I assume that it is the opposite case, when the mutator changes the type value from integer to string, and the generated string then has the same representation as a number.