rancher: Rancher catalog: "invalid character '\x00' in string literal" for large catalog entries (>3k)

Rancher Version: 1.1.4

Docker Version: 1.10.3

OS and where are the hosts located? (cloud, bare metal, etc): Ubuntu / cloud

Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB) Multi-node, internal-DB, no-HA

Environment Type: (Cattle/Kubernetes/Swarm/Mesos) Kubernetes

Steps to Reproduce: Create a custom catalog entry which contains enough content in its set of files (ie rancher-compose.yml, service.yaml, deployment.yaml) such that the buffer sent to /r/projects/1a5/kubectld:8091/v1-kubectl/catalog?defaultNamespace=production is over 3096 characters.

I was testing with this adminMongo catalog entry which also contained a fairly lengthy deployment.yaml and service.yaml:

.catalog: 
  description: "A user interface for managing MongoDB"
  name: adminMongo
  version: 1.0
  questions: 
    - description: "Domain to deploy the service to"
      label: "Domain name to deploy on"
      required: true
      type: string
      variable: domain
    - description: "A secret containing 'realm' name and 'users' file (htpasswd format) for basic authentication"
      label: "Basic-Auth Secret"
      required: false
      type: string
      variable: basicauth_secret
    - default: MongoDB
      description: "A user-readable name to attach to the default MongoDB connection"
      label: "Name of default MongoDB connection"
      required: false
      type: string
      variable: conn_name
    - default: mongo
      description: "Mongo host to connect to"
      label: "MongoDB host"
      required: false
      type: string
      variable: mongo_host
    - default: "27017"
      description: "Mongo port to connect to"
      label: "MongoDB port"
      required: false
      type: string
      variable: mongo_port
    - default: ""
      description: "Username to use to connect to MongoDB"
      label: "MongoDB user"
      required: false
      type: string
      variable: mongo_user
    - default: ""
      description: "Password to use to connect to MongoDB"
      label: "MongoDB password"
      required: false
      type: string
      variable: mongo_pass
    - default: "true"
      description: "Whether to run monitoring at regular intervals"
      label: "Monitoring"
      required: true
      type: string
      variable: monitoring

Note that this is not over 3096 characters so I don’t think it will trigger the bug. However, because this is purely a file length limitation, you can trigger it with the following:

.catalog: 
  description: "A user interface for managing MongoDB"
  name: adminMongo
  version: 1.0
  questions: 
    - description: "Domain to deploy the service to ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
    "
      label: "Domain name to deploy on"
      required: true
      type: string
      variable: domain

Now, attempt to start the service from the catalog entry, providing whatever values you want for the questions.

Results:

This error is shown with no additional information as to where: invalid character '\x00' in string literal

In the browser’s inspector, there is slightly more information provided in the response to POST /r/projects/1a5/kubectld:8091/v1-kubectl/catalog?defaultNamespace=production:

{"exitCode":-1,"stdOut":"","stdErr":"invalid character '\\x00' in string literal","error":{"Offset":3096}}

From this, it’s clear that the limit being hit is 3096 characters. I’m not sure if the content is truncated before being processed, or if this is a buffer overflow.

Expected: Rancher should spin up the service, and provide more information when failures like this occur. Or, if the maximum length of a rancher-compose file is 3096 characters, then the frontend should indicate that the file is too long.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 2
  • Comments: 16

Most upvoted comments

After debugging for a while, I found this issue and can report that the problem is still there, and that changing the client_max_body_size on nginx does not do anything.

After hearing that this is isolated to nginx (which I am using) I realized that this sounded awfully familiar. Sure enough I tried my broken test catalog and it didn’t fail, because I had already hit and solved the client-max-body-size limitation due to the Docker registry that is also behind this nginx installation.

You need to use (in your nginx config in the appropriate block):

client_max_body_size 25m;

This sets a maximum request body size of 25 MB. This setting protects your backend servers from attempts to flood them with huge requests, so make sure you understand what it’s doing and why the default limitation is in place. You can probably get by with a much smaller value here.

Cheers!

EDIT: Changed it to 25MB just in case anyone doesn’t heed my caution above.

Moving to 1.6. Confirmed it is caused by size limitation on nginx, yet to figure out the steps on how to tweak to make it work. The current workaround is to use another LB provider in front of Rancher.

Is there a workaround or change one can do to get rid of this problem? I want to use kubectl from remote in build pipelines, so the UI workaround will definitely not work.

I was facing same issue, So tried with HAProxy instead of Nginx in front of Rancher Server and working as expected.

@rezonant Is this setting added to resolve issue “invalid character ‘\x00’ in string literal” ? I added parameter ‘client_max_body_size 64m;’ to nginx, but still have this issue.

$ ll total 28 drwxrwxr-x 2 devops devops 4096 Mar 3 15:10 ./ drwxrwxr-x 3 devops devops 4096 Mar 3 15:09 …/ -rw-rw-r-- 1 devops devops 1464 Feb 26 16:28 cacert.crt -rw-rw-r-- 1 devops devops 11298 Feb 26 16:28 grafana.ini -rw-rw-r-- 1 devops devops 3572 Mar 3 15:10 ldap.toml $ kubectl create configmap grafana-config --from-file . Error from server (BadRequest): the object provided is unrecognized (must be of type ConfigMap): couldn’t get version/kind; json parse error: invalid character ‘\x00’ in string literal ({“kind”:“ConfigMap”,"apiVersio …)

I get this issues when I use kubectl to add a secret that is greater than 3kb this why is it capped at 3 kb ? @will-chan @mrajashree

Is there any work around this ?