magento2: Varnish ESI blocks don't load when HTTPS is used site-wide
This issue is related to #2657 that the OP closed.
Steps to reproduce
- Install Magento 2.0.2 with sample data
- Configure all urls to load from HTTPS (e.g., set “Base URL” and “Secure Base URL” to https://example.dev)
- Load frontend and you’ll see the navigation present.
- Configure and enable Varnish (see documentation)
- Load the frontend and you’ll see the main navigation is missing:
Potential Solutions
1. Modify Varnish ESI to ignore HTTPS
Per this Stack Exchange answer, if you modify your /etc/sysconfig/varnish file and add -p feature=+esi_ignore_https to the arguments passed to Varnish, the problem will be solved. For example:
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
-f ${VARNISH_VCL_CONF} \
-T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
-p feature=+esi_ignore_https \
-p thread_pool_min=${VARNISH_MIN_THREADS} \
-p thread_pool_max=${VARNISH_MAX_THREADS} \
-S ${VARNISH_SECRET_FILE} \
-s ${VARNISH_STORAGE}"
This solution is merely a workaround for the fact that Magento 2 returns HTTPS urls as ESI urls. Also, this option is only available in Varnish 4.
2. Modify Magento’s \Magento\PageCache\Observer\ProcessLayoutRenderElement::_wrapEsi method to never return HTTP urls
Per the #2657 issue raised by @boldhedgehog, the proper solution is to modify Magento to only return HTTP urls to Varnish. How exactly this is handled is up for discussion, as I don’t think it would be proper to simply replace “https” with “https”.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 4
- Comments: 52 (11 by maintainers)
Commits related to this issue
- add varnish edge side includes toggle for https://github.com/magento/magento2/issues/3897 — committed to ByteInternet/community.hypernode.io by vdloo 6 years ago
An update, I don’t think that it is caused by core Magento in our case. What I did:
Change php.ini, to display errors:
display_errors = on
Then I noticed an exception error where normally the menu is. It was from a plugin.
I disabled the plugin, ran setup:di:compile and rechecked.
The error was gone, while ttl=“3600” still was enabled in default.xml. Also I didn’t need to use the Varnish 'p feature=+esi_ignore_https ' option.
When I enabled the plugin again, the problem came back. Then, when I removed the ttl=“3600” from default.xml the plugin was working.
In my case the Bss_MetaTagManager module (BSS Commerce Meta Tags Template for Magento 2) was the module that was causing the issues.
I am seeing this with Varnish 6.x and Magento 2.3.x and 2.4.0. I followed @joshdavenport 's advice to filter the varnish logs with
varnishlog -q "ReqURL ~ '/page_cache/block/'"And browsed to the ESI block
/page_cache/block/esi/blocks/%5B"catalog.topnav"%5D/handles/WyJkZWZhd...This returned a 503 and the following errors in the varnish log
The too many restarts error lead me to issue https://github.com/magento/magento2/issues/24353 where another possible solution is to add
if (req.restarts > 0) { set req.hash_always_miss = true; }To the varnish vcl_recv.
I am reverse proxying to varnish with nginx using full https for Magento 2 and already implemented feature=+esi_ignore_https. It’s worth mentioning that I only see the “mysterious disappearing menu” with Varnish 6.x not with Varnish 5.
Fix should be delivered to github shortly, already done internally and will be released with 2.1
@rip057 There is certainly a learning curve with Magento 2, but after having worked with it for ~9 months (after 7 years on M1), I think it is a great platform that will server merchants well for many years to come.
Magento 2 decided to ship with Varnish support natively as many merchants and solutions partners implemented Varnish on M1 to build highly scalable and performant sites. However using Varnish as a cache system, while “recommended”, is optional. For merchants that don’t have large amounts of traffic (especially burst traffic), they can just use the native Magento caching system and forego Varnish altogether.
@erikhansen Thank you for reporting, we have created MAGETWO-51149 to investigate and fix.
Thanks for the link @gcampedelli, that was helpful. You’re right, just found the controller in use was triggering a 500 as a result of something else.
Perhaps this will be useful for those with this issue and stumbling upon this ticket:
/page_cache/block/esi/blocks/requests. Useful to runvarnishlog -q 'ReqURL ~ "^/page_cache/block"'so you only see what you’re interested in.RespStatusandRespReason. Chances are you’re seeing a 500 Internal Server Error.ReqURL. Behold, no results! This is the crux of the problem.IMO the real problem here is the obfuscation of this error during the ESI. It absolutely explains my despair at this issue, and I bet it also explains the frustration expressed by many in this ticket.
Is there a varnish limitation here that some alternative output can’t be grabbed in this case? There really should be one of the following as some kind of hint here, otherwise it fails silently:
<esi:remove>?<esi:include>is handled outside of control of Magento and if there is no log because of the error at the URL requested, that’s no different to there being no error on some other page where there is no error in exception.log - i.e. not Magentos problem<!-- ESI failed: 500 -->+esi_ignore_https doesn’t work for me, but removing ttl did. That is not a good solution.
Magento any ideas? @piotrekkaminski
Hey, even if this thread isn’t directly related to ssl setup - we’re running setups like this without problems… HaProxy terminates ssl -> varnish -> nginx -> php7-fpm/hhvm. The only thing related to categories not coming up when using varnish is to setup
on the running varnish instance like @erikhansen wrote. You can easily verify this by temporary bypassing or disabling varnish. Does the navigation show up on routes that will hit for pass? F.e. on my account or login?
All other things related to ssl should be ok.
Also sorry for being harsh, I just spent quite a lot of time getting magneto 2.0 in, then scratching and starting over, then again, only to find out about varnish at the end, and the only reason I found out about varnish is my effing navigational bar took a vacation, and I spent another several hours trying to figure out if I caused it by disabling something. Anyhow has anyone tried mod_mem_cache? Have any xp with it? On Apr 18, 2016 12:05 PM, “Chris Clark” rip057@gmail.com wrote:
@rip057 You can use a server like Nginx (example config) or Pound as an SSL terminator to pass through requests to Varnish so that HTTPS requests can still take advantage of Varnish’s memory-based cache system. So while it doesn’t natively support SSL, it can still work with SSL by having a system sitting in front of it handling the SSL connection.