magento2: After upgrade 2.1.2 to 2.1.7, Redis cache ends up in bad state intermittently, then all requests return 503

Preconditions

1. Magento CE 2.1.7
2. OS: RHEL 7.2
3. Apache 2.4.6-45
4. PHP 7.0.19-1 (IUS)
5. Redis 3.2.4-1 (IUS)
6. Percona MySQL 5.7.15-9
7. 2 identical servers behind an F5 load balancer, plus a separate admin server.
8. Media is shared between the servers via NFS.

Steps to reproduce

1. Run production traffic for hours to days until the 503s begin.
2. We have been unable to reproduce on staging, even replaying captured production traffic. However, capturing an rdb dump from the Redis server while in the error state and transplanting it to a test server has allowed us to at least reproduce the error state.

Expected result

1. Home page should load without issue

Actual result

1. User receives error processing request page with error number.
2. var/report contains file with error number, exception contained therein is one of the following, but during a particular error state the same exception occurs on every request:

a:4:{i:0;s:134:"Warning: array_merge(): Argument #1 is not an array in /data/www/magento2/vendor/magento/framework/Image/AdapterFactory.php on line 37";i:1;s:7726:"#0 [internal function]: Magento\Framework\App\ErrorHandler->handler(2, 'array_merge(): ...', '/data/www/magen...', 37, Array)
#1 /data/www/magento2/vendor/magento/framework/Image/AdapterFactory.php(37): array_merge(NULL, Array)
#2 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(93): Magento\Framework\Image\AdapterFactory->__construct(Object(Magento\Framework\ObjectManager\ObjectManager), Object(Magento\Framework\Image\Adapter\Config), Array)
#3 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(89): Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject('Magento\\Framewo...', Array)
#4 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('Magento\\Framewo...')
#5 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('Magento\\Framewo...')
#6 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'Magento\\Framewo...', NULL, 'adapterFactory', 'Magento\\Framewo...')
#7 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('Magento\\Framewo...', Array, Array)
#8 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('Magento\\Framewo...')
#9 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('Magento\\Framewo...')
#10 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'Magento\\Framewo...', NULL, 'imageFactory', 'MageArray\\Blog\\...')
#11 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('MageArray\\Blog\\...', Array, Array)
#12 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('MageArray\\Blog\\...')
#13 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('MageArray\\Blog\\...')
#14 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'MageArray\\Blog\\...', NULL, 'dataHelper', 'MageArray\\Blog\\...')
#15 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('MageArray\\Blog\\...', Array, Array)
#16 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(57): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('MageArray\\Blog\\...', Array)
#17 /data/www/magento2/var/generation/MageArray/Blog/Model/BlogpostFactory.php(43): Magento\Framework\ObjectManager\ObjectManager->create('MageArray\\Blog\\...', Array)
#18 /data/www/magento2/app/code/MageArray/Blog/Controller/Router.php(154): MageArray\Blog\Model\BlogpostFactory->create()
#19 /data/www/magento2/vendor/magento/framework/App/FrontController.php(50): MageArray\Blog\Controller\Router->match(Object(Magento\Framework\App\Request\Http))
#20 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\FrontController->dispatch(Object(Magento\Framework\App\Request\Http))
#21 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Framework\App\FrontController\Interceptor->___callParent('dispatch', Array)
#22 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'requestPreproce...')
#23 /data/www/magento2/vendor/magento/module-store/App/FrontController/Plugin/RequestPreprocessor.php(94): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#24 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\FrontController\Plugin\RequestPreprocessor->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#25 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'install')
#26 /data/www/magento2/vendor/magento/framework/Module/Plugin/DbStatusValidator.php(69): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#27 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Framework\Module\Plugin\DbStatusValidator->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#28 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#29 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/VarnishPlugin.php(55): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#30 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\PageCache\Model\App\FrontController\VarnishPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#31 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#32 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/BuiltinPlugin.php(68): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#33 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\PageCache\Model\App\FrontController\BuiltinPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#34 /data/www/magento2/var/generation/Magento/Framework/App/FrontController/Interceptor.php(26): Magento\Framework\App\FrontController\Interceptor->___callPlugins('dispatch', Array, Array)
#35 /data/www/magento2/vendor/magento/framework/App/Http.php(135): Magento\Framework\App\FrontController\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#36 /data/www/magento2/vendor/magento/framework/App/Bootstrap.php(258): Magento\Framework\App\Http->launch()
#37 /data/www/magento2/pub/index.php(37): Magento\Framework\App\Bootstrap->run(Object(Magento\Framework\App\Http))

a:4:{i:0;s:60:"Time to live is a mandatory parameter for set public headers";i:1;s:9650:"#0 /data/www/magento2/var/generation/Magento/Framework/App/Response/Http/Interceptor.php(50): Magento\Framework\App\Response\Http->setPublicHeaders(NULL)
#1 /data/www/magento2/vendor/magento/module-page-cache/Model/Layout/LayoutPlugin.php(48): Magento\Framework\App\Response\Http\Interceptor->setPublicHeaders(NULL)
#2 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(152): Magento\PageCache\Model\Layout\LayoutPlugin->afterGenerateXml(Object(Magento\Framework\View\Layout\Interceptor), Object(Magento\Framework\View\Layout\Interceptor))
#3 /data/www/magento2/var/generation/Magento/Framework/View/Layout/Interceptor.php(78): Magento\Framework\View\Layout\Interceptor->___callPlugins('generateXml', Array, Array)
#4 /data/www/magento2/vendor/magento/framework/View/Layout/Builder.php(104): Magento\Framework\View\Layout\Interceptor->generateXml()
#5 /data/www/magento2/vendor/magento/framework/View/Layout/Builder.php(64): Magento\Framework\View\Layout\Builder->generateLayoutXml()
#6 /data/www/magento2/vendor/magento/framework/View/Layout.php(244): Magento\Framework\View\Layout\Builder->build()
#7 /data/www/magento2/vendor/magento/framework/View/Layout.php(859): Magento\Framework\View\Layout->build()
#8 /data/www/magento2/var/generation/Magento/Framework/View/Layout/Interceptor.php(414): Magento\Framework\View\Layout->getBlock('page_content_he...')
#9 /data/www/magento2/vendor/magento/module-cms/Helper/Page.php(171): Magento\Framework\View\Layout\Interceptor->getBlock('page_content_he...')
#10 /data/www/magento2/vendor/magento/module-cms/Controller/Index/Index.php(43): Magento\Cms\Helper\Page->prepareResultPage(Object(Magento\Cms\Controller\Index\Index\Interceptor), 'home')
#11 /data/www/magento2/var/generation/Magento/Cms/Controller/Index/Index/Interceptor.php(24): Magento\Cms\Controller\Index\Index->execute(NULL)
#12 /data/www/magento2/vendor/magento/framework/App/Action/Action.php(102): Magento\Cms\Controller\Index\Index\Interceptor->execute()
#13 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\Action\Action->dispatch(Object(Magento\Framework\App\Request\Http))
#14 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Cms\Controller\Index\Index\Interceptor->___callParent('dispatch', Array)
#15 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'contextPlugin')
#16 /data/www/magento2/vendor/magento/module-store/App/Action/Plugin/Context.php(106): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#17 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\Action\Plugin\Context->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#18 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'customer-app-ac...')
#19 /data/www/magento2/vendor/magento/module-customer/Model/App/Action/ContextPlugin.php(61): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#20 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Customer\Model\App\Action\ContextPlugin->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#21 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'storeCheck')
#22 /data/www/magento2/vendor/magento/module-store/App/Action/Plugin/StoreCheck.php(44): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#23 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\Action\Plugin\StoreCheck->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#24 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'tax-app-action-...')
#25 /data/www/magento2/vendor/magento/module-tax/Model/App/Action/ContextPlugin.php(91): Magento\Cms\Controller\Index\Index\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#26 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\Tax\Model\App\Action\ContextPlugin->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#27 /data/www/magento2/var/generation/Magento/Cms/Controller/Index/Index/Interceptor.php(39): Magento\Cms\Controller\Index\Index\Interceptor->___callPlugins('dispatch', Array, Array)
#28 /data/www/magento2/vendor/magento/framework/App/FrontController.php(55): Magento\Cms\Controller\Index\Index\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#29 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\FrontController->dispatch(Object(Magento\Framework\App\Request\Http))
#30 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Framework\App\FrontController\Interceptor->___callParent('dispatch', Array)
#31 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'requestPreproce...')
#32 /data/www/magento2/vendor/magento/module-store/App/FrontController/Plugin/RequestPreprocessor.php(94): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#33 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\FrontController\Plugin\RequestPreprocessor->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#34 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'install')
#35 /data/www/magento2/vendor/magento/framework/Module/Plugin/DbStatusValidator.php(69): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#36 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Framework\Module\Plugin\DbStatusValidator->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#37 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#38 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/VarnishPlugin.php(55): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#39 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\PageCache\Model\App\FrontController\VarnishPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#40 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#41 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/BuiltinPlugin.php(68): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#42 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\PageCache\Model\App\FrontController\BuiltinPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#43 /data/www/magento2/var/generation/Magento/Framework/App/FrontController/Interceptor.php(26): Magento\Framework\App\FrontController\Interceptor->___callPlugins('dispatch', Array, Array)
#44 /data/www/magento2/vendor/magento/framework/App/Http.php(135): Magento\Framework\App\FrontController\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#45 /data/www/magento2/vendor/magento/framework/App/Bootstrap.php(258): Magento\Framework\App\Http->launch()
#46 /data/www/magento2/pub/index.php(37): Magento\Framework\App\Bootstrap->run(Object(Magento\Framework\App\Http))
#47 {main}";s:3:"url";s:1:"/";s:11:"script_name";s:10:"/index.php";}

Additional Information

What seems to happen is something corrupts the Redis cache, and from that point forward all requests start to return an error page with a 503. Deleting the bd5_SYSTEM_CACHE_EXISTS key from Redis, or issuing a FLUSHDB command, clears the error. The corrupted cache state even prevents the bin/magento CLI from functioning correctly.