prism: [Bug] 422 Response Not Being Used For Validation Response

Describe the bug

Hello! I could really use some help and guidance!

I have an Open API spec that is valid when I used Spectral

I can successfully run it with Prism

I have unit tests that validate an endpoint, so I am using Prism like so prism proxy --errors example.yaml http://localhost:8080

When I send an invalid request it keeps giving me this error

{
  type: 'https://stoplight.io/prism/errors#UNPROCESSABLE_ENTITY',
  title: 'Invalid request',
  status: 422,
  detail: 'Your request is not valid and no HTTP validation response was found in the spec, so Prism is generating this error for you.',
  validation: [
    {
      location: [Array],
      severity: 'Error',
      code: 'required',
      message: "should have required property 'firstname'"
    },
    {
      location: [Array],
      severity: 'Error',
      code: 'required',
      message: "should have required property 'lastname'"
    },
    {
      location: [Array],
      severity: 'Error',
      code: 'required',
      message: "should have required property 'email'"
    },
    {
      location: [Array],
      severity: 'Error',
      code: 'required',
      message: "should have required property 'password'"
    }
  ]
}

Which is wild because this DOES exist.

It keeps stating Your request is not valid and no HTTP validation response was found in the spec, so Prism is generating this error for you which is not true at all because I have a 422 response declared for the endpoint that’s being hit!

I have tried setting a 400 response. I’ve tried setting different data for the response. And nothing is working.

What am I doing wrong?

To Reproduce

Use this file

---
openapi: '3.0.2'

servers:
    - url: http://localhost:8080
      description: Local deployment from your computer

paths:
    /api/users/register:
        post:
            summary: 'Register New User'
            description: 'Register a new user'
            operationId: 'registerNewUser'
            tags: ['Users']
            requestBody:
                required: true
                content:
                    application/json:
                        schema:
                            type: 'object'
                            properties:
                                firstname:
                                    type: 'string'
                                    minLength: 2
                                    maxLength: 100
                                    example: 'John'
                                lastname:
                                    type: 'string'
                                    minLength: 2
                                    maxLength: 100
                                    example: 'Roberts'
                                email:
                                    type: 'string'
                                    minLength: 4
                                    maxLength: 100
                                    example: 'john@example.com'
                                password:
                                    type: 'string'
                                    minLength: 8
                                    maxLength: 50
                                    example: 'abc123456'
                            required:
                                - firstname
                                - lastname
                                - email
                                - password
            responses:
                '201':
                    description: Successful registration of a new user
                '422':
                    description: Failed to register new user
                    content:
                        application/json:
                            schema:
                                type: 'object'
                                properties:
                                    message:
                                        type: 'string'
                                        example: 'firstname is required'

Run prism proxy --errors example.yaml http://localhost:8080

Run this curl command to confirm it works

curl --location --request POST 'localhost:4010/api/users/register' \
--header 'Content-Type: application/json' \
--data-raw '{
	"firstname": "First",
	"lastname": "Last",
	"email": "example@example.com",
	"password": "This_is_4_str0ng_password!"
}'

If you want pretty printing you can add json_pp

curl --location --request POST 'localhost:4010/api/users/register' \
--header 'Content-Type: application/json' \
--data-raw '{
	"firstname": "First",
	"lastname": "Last",
	"email": "example@example.com",
	"password": "This_is_4_str0ng_password!"
}'

Run this command to confirm it fails

curl --location --request POST 'localhost:4010/api/users/register' \
--header 'Content-Type: application/json' \
--data-raw '{}'

Same, but with pretty printing

curl --location --request POST 'localhost:4010/api/users/register' \
--header 'Content-Type: application/json' \
--data-raw '{}' | json_pp

And the final output is this

~/Repositories/example   feature/api_spec_and_test_reconciliation ±  ⬡ v12.17.0  curl --location --request POST 'localhost:4010/api/users/register' \
--header 'Content-Type: application/json' \
--data-raw '{}' | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   702  100   700  100     2   113k    333 --:--:-- --:--:-- --:--:--  114k
{
   "detail" : "Your request is not valid and no HTTP validation response was found in the spec, so Prism is generating this error for you.",
   "title" : "Invalid request",
   "type" : "https://stoplight.io/prism/errors#UNPROCESSABLE_ENTITY",
   "status" : 422,
   "validation" : [
      {
         "severity" : "Error",
         "location" : [
            "body"
         ],
         "message" : "should have required property 'firstname'",
         "code" : "required"
      },
      {
         "message" : "should have required property 'lastname'",
         "location" : [
            "body"
         ],
         "severity" : "Error",
         "code" : "required"
      },
      {
         "code" : "required",
         "severity" : "Error",
         "location" : [
            "body"
         ],
         "message" : "should have required property 'email'"
      },
      {
         "message" : "should have required property 'password'",
         "location" : [
            "body"
         ],
         "severity" : "Error",
         "code" : "required"
      }
   ]
}

Expected behavior

I expect the responses I created for 422 to be used so that I don’t have to worry about Prism returning different payloads than what my server does.

Questions

How do I get Prism to not return a generated 422 override?

Because it keeps generating that validation response I can’t integrate it with my tests, because the tests expect a message property to be there.

I’ve read through the documentation here:

And none of it provides an explicit example on how to make it so that Prism does not return an auto-generated response.

Additional context

  • Prism: 4.1.2
  • OS: MacOS

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (7 by maintainers)

Most upvoted comments

Hello @chohmann!

I really wish there was a higher bandwith form of communication we could use 😂 I think we may be getting our wires crossed together and I would love the opportunity to understand the thought process as well as contrast it with the goals I’m trying to achieve.

RE: --errors Flag

I want to use prism in my CI/CD in order to call out drift between my spec and implementation. I need this on a spiritual level, and on an organizational level 😂

With the --errors flag the CI/CD will fail because Prism will return an error code when the spec does not match the request or response payload. I want this feature

My understanding is that if I drop the --errors flag that will not cause an error to be generated and instead will just be receiving the exact response from my server back, which is fine. But not what I’m looking for.

What I want is prism to allow opting out of the request validation in order to allow the opportunity for the response to be validated against the spec BEFORE I receive the response back and run my validations on it.

In this situation there would be a response error generated just like the request error.

What I would like it look like is this

  1. Test Request -> Prism -> Server
    • No request validation
  2. Server Response -> Prism -> Test
    • Response validation is performed
  3. Prism screams about a response and spec mismatch
    • I fix mismatch, and run it again
    • Now response is valid
  4. Test Request -> Prism -> Server
    • No request validation
  5. Server Response -> Prism -> Test
    • Response validation is performed, but no errors were found, so original server response is returned

In this situation I can rely on Prism to fail when the spec doesn’t match and succeed when it does. And then, I get an addition layer of validation that’s easy to layer on top of existing tests without having to rewrite anything, because it will check

  1. Specification validity AND when the spec is valid, the correct response will get through so now I also get
  2. My tests to validate the response as well

Reiterating Desired Feature

I would love love love it if Prism generated errors when the response payload does not match the response in the specification.

In order to get to this step, I NEED Prism to allow the ability to skip request validation otherwise it generates a response saying the request is invalid, which is intended.

I see in your Prism Client documents here, that this type of logic is already supported https://meta.stoplight.io/docs/prism/docs/guides/http-client.md#override-the-configuration-object-on-the-request-level

HOWEVER, I do not want to code this in the Prism Client because I already have tests set up and configured. What I do want though, is the ability to pepper in something like a header to skip requestValidation

I want to have Prism as an intermediate layer that I spin up and run requests/responses through in my CI/CD

Ideally, just like the Prefer header you all have provided, it would be radical if you had something like

  • RequestValidation: False
  • ResponseValidation: False

For example - See line 4

// ...
it("It should fail to update project with a payload that has a name with an empty string", (done) => {
    chai.request(server)
        .patch(`/api/projects/${id}`)
        .set("RequestValidation", "false") // Required for Prism integration
        .send({
            name: "",
            description: "editing project name"
        })
        .end((err, res) => {
            if (err) done(err);
            assert.equal(res.status, 422);
            assert.equal(res.body.message, '"name" is not allowed to be empty');
            done();
        });
});

Which would then let me simply change my tests to add headers in order to skip the steps.

In this way, I can integrate Prism on top of my tests for free AND I don’t have to write any of my tests over again. I just add headers to indicate the validation I want Prism to scream about.

By dropping the --errors flag I am missing the best part about Prism which is the screaming about a spec mismatch. That is why I am so excited about the tool - Contract validation.

Prefer Header Usage

@chohmann Can you please clarify the issues I was having with the Prefer keyword and example as well as combining the usage of code and example

Or, would it be better to open a separate ticket for that issue?

@loganknecht I’m going to try to answer all of your questions.

If it’s possible, when using proxy mode it would be absolutely fantastic if there was a way to allow invalid requests through, instead of generating an error.

This is possible, just leave off the --errors flag and the request will be forwarded to the actual server. Prism will also still validate the request matches the spec and log out any validation errors.

Here’s how I verified this use case:

  1. rather than building an api server to proxy requests to, I spun up a mock server using your OAS Spec
prism mock /Users/chelseapace/stoplight/prism/examples/sample-openAPI.yaml -h localhost -p 4090

so now I have my “actual” server running at http://localhost:4090

  1. spin up prism in proxy mode and proxy requests to our “actual” server created in step 1. Leave off the --errors flag to make sure the requests get forwarded to our actual server
prism proxy /Users/chelseapace/stoplight/prism/examples/sample-openAPI.yaml http://localhost:4090

our prism proxy now accepts request at http://localhost:4010 and forwards them to http://localhost:4090

  1. Make a request to the prism proxy and see it get forwarded to our actual server at http://localhost:4090.
curl --location --request POST 'localhost:4010/api/users/register' \
--header 'Content-Type: application/json' \
--data-raw '{}'
{"message":"firstname is required"}%

The response we get is the one returned by our actual server: image

And looking at the logs of the prism proxy, you can see it logged out validation errors the request had: image

Based on the use case you described, I don’t think Prism will add any value. You want to hit the actual server, receive the actual response, and then validate the string like expect(response).toBe('message defined in spec'). You can do that without prism proxy, you’ll just have to keep the test code up to date with what response to expect.

Hi @loganknecht,

I was taking a closer look at this and realized you’re using prism in proxy mode, and not in mock mode. When in proxy mode a request that does not match the schema will either be forwarded to the existing api server, or prism will blow up the request based on the use of the --errors flag. It will not return a response defined in the spec in proxy mode. More information on proxy mode can be found here.

The error message is absolutely misleading when read in the context of proxy mode. The error object is shared across both mock and proxy modes, but is worded primarily for the mock context. We could look into making this error message more clear when used in the proxy context, but probably won’t be able to do so in the short term.

If you run prism in mock mode, the example defined for the 422 response will be returned if an invalid request is sent to prism: image

I hope that answers your questions!