Ghostwriter: GraphQL: Unable to interface with endpoint

I’m following the documentation (https://www.ghostwriter.wiki/features/graphql-api) and trying to interface with the GraphQL endpoint

Using the example request:

import json
import requests

headers = {"Content-Type": "application/json", }

def prepare_query(query, operation):
  return json.dumps({
    "query": query,
    "operationName": operation
  })

def post_query(headers, data):
  return requests.post(
    "https://127.0.0.1/v1/graphql",
    headers=headers,
    data=data
  )

# Stacked query with `Login` and `Whoami` operations
query = """
  mutation Login {
    login(password:"<redacted>", username:"<redacted>") {
      token expires
    }
  }

  query Whoami {
    whoami {
      username role expires
    }
  }
  """

# Send query and set `Login` as the `operationName`
response = post_query(headers, prepare_query(query, "Login"))
# Get the JWT from the response and add it to the headers
token = response.json()["data"]["login"]["token"]
headers["Authorization"] = f"Bearer {token}"
# Send the query again but execute the `Whoami` operation this time
response = post_query(headers, prepare_query(query, "Whoami"))
# Print our JWT's whoami informaiton
print(response.json())

When using the example request, I receive the following response as part of a 200: {'errors': [{'extensions': {'path': '$', 'code': 'unexpected'}, 'message': 'Invalid response from authorization hook'}]}

I created an API key and used that directly as the Bearer token, but receive the same Invalid response from authorization hook error as above.

I enabled Hasura, and similarly receive an error when attempting to perform the login query (Hasura POST’s to https://127.0.0.1/v1/graphql): image

However, I can perform ‘other’ types of queries (when authenticated with the x-hasura-admin-secret) without an issue: image

Can someone please assist with the issue and let me know how I can interface with the API via python without relying on Hasura?

Does the documentation need to be updated

About this issue

Most upvoted comments

@zachfey Thanks for confirming that helped! I appreciate the offer, but I think I’ve got everything I need right now. If new certificates help the others that will confirm the certificates are the root cause. I’ll leave this open for a while to collect feedback.

@chrismaddalena I have this same issue, however I was unable to solve this with adding a CA. As a workaround I used x-hasura-admin-secret. GraphQL queries work. However mutation queries don’t.

Performing the generateReport mutation results in a http exception when calling webhook. With error message Proxy connection to nginx:443 returned response with status code that indicated failure: 502

The error above is the same error which I receive when using the mutation Login query. I don’t understand why queries work, but mutations don’t.

Additional info: I enabled debug in the nginx.conf as mentioned in https://github.com/GhostManager/Ghostwriter/issues/238#issuecomment-1225016285. This didn’t result in any info logging in the error logging sadly.

Last edit and solution: This might be important information for anyone working from within an enterprise environment. Our Docker environment needs to pass through a webproxy. So we edited our docker config.json to include this:

/root/.docker/config.json
{
  "proxies": {
    "default": {
      "httpProxy": "http://PROXY:8080",
      "httpsProxy": "http://PROXY:8080
    }
  }
}

However the config above causes all the internal containers to go over the proxy. Causing nginx webhook calls to fail. In order to fix this the noProxy must be set. I’ve added the following line to the config.json: "noProxy": "django,postgres,redis,nginx,queue,graphql_engine"

@chrismaddalena I think this can also be fixed by expanding the no_proxy variables set in the https://github.com/GhostManager/Ghostwriter/blob/master/production.yml

After fixing the proxy issue I used these instructions to generate a certificate with the CN nginx https://github.com/GhostManager/Ghostwriter/issues/238#issuecomment-1252889629

@zachfey Thanks for adding your information to this! I setup an EC2 instance to see if I could experience the issue.

I reproduced the issue, but that was to be expected using an AWS IP and the public DNS name with the default self-signed Ghostwriter certificate. I added the public DNS name (ec2-x-x-x-x.compute-1.amazonaws.com) to Hasura’s Insecure TLS Allowlist and the issue was unchanged. I added the IP addresses and private DNS name (ip-x-x-x-x.ec2.internal) to cover everything, but nothing changed.

These changes always resolved the problem for me in my test environments, so deploying in AWS changed something and allowed me to troubleshoot this first hand.

It seems like Hasura’s Insecure TLS Allowlist might not work for every configuration. Based on the documentation, I expect it to ignore all SSL warnings/errors (e.g., sslv3 alert certificate unknown:SSL alert number 46) and establish the connection to the authentication webhook, but we know that adding your server’s hostname or the Nginx container’s hostname to that list doesn’t work for everyone.

Theoretically, our Ghostwriter setups should be nearly identical because we’re all using the Docker containers, so I’ve focused on the configuration differences. Those results are inconsistent, so I started to suspect the certificates.

I created a new self-signed certificate for the EC2 instance:

$ cd ssl
$ openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -keyout ghostwriter.key -out ghostwriter.crt
Generating a 4096 bit RSA private key
............++
............................................++
writing new private key to 'ghostwriter.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:WA
Locality Name (eg, city) [Default City]:Seattle
Organization Name (eg, company) [Default Company Ltd]:SpecterOps
Organizational Unit Name (eg, section) []:Ghostwriter
Common Name (eg, your name or your server's hostname) []:ec2-x-x-x-x.compute-1.amazonaws.com
Email Address []:admin@ghostwriter.local

Using installation defaults (i.e., default cofngiuration values set when running ./ghostwriter-cli install), the webhook authentication error disappeared. I didn’t even add the EC2 public DNS name to Hasura’s Insecure TLS Allowlist.

I need to look into this more to see if it would help to change to how Ghostwriter CLI generates the default certificate. For now, could someone else try generating a new certificate and checking their webhook?

Generate the certificate for whatever hostname you use to connect to Ghostwriter. If it’s in EC2, use the EC2 public DNS name.

I was having the same issue and generating a new self-signed certificate worked! Thanks @chrismaddalena

I also tried a new certificate with the CN set to nginx. That also worked without issue. The issue returned when I restored the original certificate files. The default certificate seems to be the problem.

To avoid issues with openssl not being in the PATH, Ghostwriter CLI generates the certificate entirely through Golang. Perhaps this causes a variance in the output base don the OS or OS version. I didn’t encounter this issue with the certificate on macOS or Debian 11, but see it on Amazon Linux.

Please let me know if a new certificate helps you.

Hey @SecurityJon, thanks for letting me know! I was traveling last week for Black Hat USA, so I couldn’t look into this much. I am diving back into this now and am dedicated to finding an answer. I’m trying to work with the Hasura developers to see if I can learn more about the exception.

There is a workaround if you need to interact with the API right now. You can bypass the authentication by using the x-hasura-admin-secret header. You will authenticate as an administrator and Hasura won’t use the webhook to authenticate your requests. If you do this, please be very careful. An administrator can do anything. You could delete or change things that should not be deleted or changed or override values that should be set automatically or auto-increment (e.g., id fields).

One other thing to look at:

https://www.ghostwriter.wiki/getting-started/quickstart#customizing-the-domain-name-or-ip-address

Hasura needs to talk to https://{NGINX_HOST}:{NGINX_PORT}. The default value for {NGINX_HOST} is nginx. That works as long as nginx appears in the list of allowed hostnames.

I did have a situation where nginx did not work because of the TLS certificate. In that case, setting NGINX_HOST to the domain name used for DNS and the certificate resolved the issue–i.e., there is DNS and a cert for ghostwriter.foo.bar, so NGINX_HOST changes to ghostwriter.foo.bar.

It’s been a while, but in that situation, there were no logs because a connection was never properly established. That made it difficult to troubleshoot. If you aren’t seeing anything in the logs, that may be the solution for you.