magento2: Multisite installation, default website slow (X-Magento-Vary)

Hi. We have a multisite installation. The default website is fast, but the non-default website is really slow. When we switch the default website to the slow one, that ones becomes fast. What I’ve found out so far is that the slow site always get a cookie named X-Magento-Vary, while the fast one get a PHPSESSID cookie.

Preconditions

Ubuntu 16.04 PHP 7.1.16 Aurora MySQL 5.6.10a (Aws RDS) Magento 2.2.2 Nginx 1.14.0

Running on multiple AWS instances in OpsWorks with continuous integration (Jenkins). Media folder is shared on EFS.

Steps to reproduce

The website codes are “base” and “exampleone”. In the tests below we have “exampleone” set as the default website.

Nginx configs:

upstream fastcgi_backend {
  server   unix:/run/php/php7.1-fpm.sock;
}

fastcgi_cache_path /var/cache/nginx levels=1:2 keys_zone=magento:100m inactive=60m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";

map $http_host $MAGE_RUN_CODE {
    default base;
    www.example1.com exampleone;
    example1.com exampleone;
}

server {
    listen 80 default_server;
    server_name _;
    set $MAGE_ROOT /srv/www/production/current/;
    set $MAGE_MODE production; # or developer
    index  index.php index.html index.htm;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    include /srv/www/production/current/nginx.conf;
}

Relevant part of /srv/www/production/current/nginx.conf:

# PHP entry point for main application
location ~ (index|get|static|report|404|503|autocomplete|health_check)\.php$ {
    try_files $uri =404;
    fastcgi_pass   fastcgi_backend;
    fastcgi_buffers 1024 4k;
    fastcgi_param  MAGE_RUN_CODE $MAGE_RUN_CODE;
    fastcgi_param  MAGE_RUN_TYPE $MAGE_RUN_TYPE;
    fastcgi_param https on;
    fastcgi_param  PHP_FLAG  "session.auto_start=off \n suhosin.session.cryptua=off";
    fastcgi_param  PHP_VALUE "memory_limit=1024M \n max_execution_time=60";
    fastcgi_read_timeout 60s;
    fastcgi_connect_timeout 60s;

    fastcgi_param REMOTE_ADDR $http_x_forwarded_for;
    fastcgi_param Host $http_host;
    fastcgi_param X-Forwarded-Proto https;
    fastcgi_param X-Forwarded-Port 443;
    fastcgi_param X-Forwarded-Host $http_host;
    fastcgi_param X-Real-IP $remote_addr;
    fastcgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
    add_header mage-run-code $MAGE_RUN_CODE;
    add_header mage-run-type $MAGE_RUN_TYPE;

    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    include        fastcgi_params;
}

1. Just by visiting the website, we can clearly tell that one website is much slower than the other one. TTFB is really high (8+ seconds).

2. By CURL’ing the websites, we can see the X-Magento-Vary and PHPSESSID cookies

curl -I https://www.example1.com/

HTTP/2 200
date: Tue, 05 Jun 2018 08:16:59 GMT
content-type: text/html; charset=UTF-8
server: nginx/1.14.0
vary: Accept-Encoding
set-cookie: PHPSESSID=kn4lppbu32dgfrd6ejnit2b307; expires=Wed, 06-Jun-2018 08:16:59 GMT; Max-Age=86400; path=/; domain=www.example1.com; secure; HttpOnly
pragma: no-cache
cache-control: max-age=0, must-revalidate, no-cache, no-store
expires: Mon, 05 Jun 2017 08:09:10 GMT
x-cache-warmer: HIT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
mage-run-code: exampleone
mage-run-type: website

curl -I https://www.example2.com/

HTTP/2 200
date: Tue, 05 Jun 2018 08:17:16 GMT
content-type: text/html; charset=UTF-8
server: nginx/1.14.0
vary: Accept-Encoding
set-cookie: X-Magento-Vary=4dc58702c807c010b15fff2913b3c86682a31e1e; path=/; secure; HttpOnly
pragma: no-cache
cache-control: max-age=0, must-revalidate, no-cache, no-store
expires: Mon, 05 Jun 2017 08:17:16 GMT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN

Expected result

Both websites should be fast. I’m not sure what X-Magento-Vary does, but every time the website is slow, this cookie is set. I was thinking it could be related to user specific caches, but we are testing on the index page as a non-logged in user so the same content should be delivered every time.

Actual result

Non-default website is slow. As a note, magento does set the correct website and everything works, it’s just slow.