puppet-elasticsearch: Elsticsearch 6.3 Upgrade Broken (in Debian based Systems)

Module version: 6.3.1
Puppet version: 4.10.12 and 5.5.4
OS and version: Debian 9 and Ubuntu 18.04

Bug description

Install any old version of elasticsearch before 6.3.0 e.g.

    include ::java

    class { 'elasticsearch':
      version => '6.2.0'
    }

    elasticsearch::instance { 'es-01': }

Then try to upgrade Elasticsearch to version 6.3.x. The upgrade will fail because it will try to create an initial keystore (this feature was introduced in version 6.3.0 https://github.com/elastic/elasticsearch/pull/28928 ). The creating of the keystore will fail because elasticsearch-keystore doesn’t find ES_PATH_CONF and exists with an errorcode.

==> puppet-test01.local: ES_PATH_CONF must be set to the configuration path
==> puppet-test01.local: dpkg: error processing package elasticsearch (--configure):
==> puppet-test01.local:  installed elasticsearch package post-installation script subprocess returned error exit status 1

The problem does not happen if the initial version is already 6.3.x because then the keystore is created before ES_PATH_CONF is removed from /etc/default/elasticsearch.

About this issue

Original URL
State: open
Created 6 years ago
Reactions: 7
Comments: 22 (10 by maintainers)

Most upvoted comments

Based upon the feedback here, my hunch is that taking the approach of:

Adding the /etc/environment value as a short-term fix, and
Long-term, dropping support for multiple Elasticsearch instances

Is probably the best approach. I’ve discussed the potential of removing multiple-instance support in the module with people both inside and outside Elastic, and it seems like doing so is really the only way to keep the module highly stable and reliable for people (there’s so many issues that it causes - this is just one, but out-of-date logging config files, jvm options, init scripts, and so many more are also plagued by this).

+11

tylerjl on Jan 31, 2019

Okay, new ElasticSearch update today, explicitly setting ES_PATH_CONF in the environment, and all went well.

# env ES_PATH_CONF=/etc/elasticsearch apt upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  elasticsearch-oss kibana-oss
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 110 MB of archives.
After this operation, 8192 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 elasticsearch-oss all 6.4.2 [33.7 MB]
Get:2 https://artifacts.elastic.co/packages/oss-6.x/apt stable/main amd64 kibana-oss amd64 6.4.2 [76.2 MB]
Fetched 110 MB in 6s (17.4 MB/s)                                                                                                                                                                                                          
(Reading database ... 237851 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.4.2_all.deb ...
Unpacking elasticsearch-oss (6.4.2) over (6.4.1) ...
Preparing to unpack .../kibana-oss_6.4.2_amd64.deb ...
Stopping kibana service... OK
Unpacking kibana-oss (6.4.2) over (6.4.1) ...
Setting up elasticsearch-oss (6.4.2) ...
Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
Created elasticsearch keystore in /etc/elasticsearch
Processing triggers for systemd (232-25+deb9u4) ...
Setting up kibana-oss (6.4.2) ...
#

This is probably the simplest workaround 🎉.

Puppet still fixes a lot of things after the update.

smortex on Oct 4, 2018

same issue.

I used to for work-around global env:

file_line { 'es-path-conf':
          path  => '/etc/environment',
          line  => "ES_PATH_CONF=/etc/elasticsearch",
          match => 'ES_PATH_CONF=',
}

olevole on Aug 28, 2018

At least for 6.6 just

touch /etc/elasticsearch/elasticsearch.keystore

still worked fine for me, as reported in https://github.com/elastic/puppet-elasticsearch/issues/977#issuecomment-425073134

seidler2547 on Mar 18, 2019

I just checked the postinstall script again and in 6.4.x, it looks like this: 6.4

# the equivalent code for rpm is in posttrans
if [ "$PACKAGE" = "deb" -a ! -f /etc/elasticsearch/elasticsearch.keystore ]; then
    /usr/share/elasticsearch/bin/elasticsearch-keystore create
    chown root:elasticsearch /etc/elasticsearch/elasticsearch.keystore
    chmod 660 /etc/elasticsearch/elasticsearch.keystore
    md5sum /etc/elasticsearch/elasticsearch.keystore > /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum
fi

It ssems that it would also be sufficient to have the puppet module create an empty file at /etc/elasticsearch/elasticsearch.keystore to fix the issue.

roock on Sep 27, 2018