Description

Provide a brief description of your issue here. For example:

On a multi master install deploy_cluster.yml terminates after task “openshift_master_certificates : Generate the loopback master client config” with the error message “ERROR! |failed expects hostvars is a dict”

Version

• oc verions

# oc version
oc v3.7.1+c2ce2c0-1
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

• Ansible verions

# ansible --version
ansible 2.6.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]

• Ansible Openshift Playbooks

# cd /etc/ansible/
# mkdir playbooks
# cd playbooks
# git clone https://github.com/openshift/openshift-ansible

• CentOS version

# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

• Openshift version

# openshift version
openshift v3.7.1+c2ce2c0-1
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

Inventory Hosts Files

#---- Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
debug_level='2'
containerized='false'
openshift_deployment_type=origin
os_firewall_use_firewalld='true'

#-- Configure networking
#openshift_master_default_subdomain=fossil.com
openshift_use_openshift_sdn=true
#openshift_use_flannel=true
#flannel_interface=bond0

# Uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider.
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# Native high availbility cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
#-- When we get to production use an external LB
openshift_master_cluster_method=native
#openshift_master_cluster_hostname=uusrcdkub00.fossil.com
#openshift_master_cluster_public_hostname=uusrcdkub00.fossil.com

# apply updated node defaults
openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}

# enable ntp on masters to ensure proper failover
openshift_clock_enabled=true

# Setup rolling updates at the service level
openshift_rolling_restart_mode='services'

# Change all the certificates to last for 10 years.
openshift_hosted_registry_cert_expire_days='720'
openshift_ca_cert_expire_days='720'
openshift_node_cert_expire_days='720'
openshift_master_cert_expire_days='720'
etcd_ca_default_days='720'


#---- host group for masters
[masters]
uusrcdkub01.fossil.com
uusrcdkub02.fossil.com
uusrcdkub03.fossil.com

#---- host group for etcd
[etcd]
uusrcdkub01.fossil.com
uusrcdkub02.fossil.com
uusrcdkub03.fossil.com

#---- Specify load balancer host
#[lb]
#uusrcdkub00.fossil.com

#---- host group for nodes, includes region info
[nodes]
uusrcdkub01.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
uusrcdkub02.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
uusrcdkub03.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"


# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
#lb

Steps To Reproduce

• After base CentOS 7.3 install

1. Install pre-reqs and update the OS

yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct curl docker etcd atomic pyOpenSSL.x86_64 python-cryptography python-lxml
yum update
systemctl reboot

1. Increase the root volume group to use the complete disk

fdisk /dev/sda
d
2
n
primary
2
default
default

systemctl reboot
pvresize /dev/sda2

1. Update the docker config to use a LV for the docker storage

systemctl stop docker
rm -rf /var/lib/docker
vi /etc/sysconfig/docker-storage-setup
# File contents below
/bin/docker-storage-setup
systemctl start docker
systemctl enbale docker
docker info
lvs

File: /etc/sysconfig/docker-storage-setup

# Edit this file to override any configuration options specified in
# /usr/lib/docker-storage-setup/docker-storage-setup.
#
# For more details refer to "man docker-storage-setup"
#STORAGE_DRIVER
#DEVS=/dev/sda
VG=centos
GROWPART=enable
AUTO_EXTEND_POOL=enable
MIN_DATA_SIZE=8G
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=10

• Install Ansible on the first master We need at least Ansible 2.4.3. At the time of writing this doc the latest published was 2.4.2. So, we need to role our own RPM. To build Ansible we need:

	yum -y install asciidoc rpm-build python2-devel

• Build and install the latest Ansible

	mkdir Development
	git clone https://github.com/ansible/ansible.git
	cd ansible
	make rpm
	cd /root/Development/ansible/rpm-build
	rpm --install ansible-2.6.0-100.git201803142047.0cf2ecb.devel.el7.centos.noarch.rpm

• Pull the OpenShift Ansible Source

	cd /etc/ansible/
	mkdir playbooks
	cd playbooks
	git clone https://github.com/openshift/openshift-ansible

• Create the Ansible Host File

  vi /etc/ansbile/hosts

File contents listed above.

• Run the pre-reqs and deploy

  cd /etc/ansible/playbooks/openshift-ansible/playbooks/
  ansible-playbook -i /etc/ansible/hosts $PWD/prerequisites.yml
	
  cd /etc/ansible/playbooks/openshift-ansible/playbooks/openshift-checks
  vi roles/openshift_health_checker/openshift_checks/package_version.py

I updated the allowed docker versions for (3,7) to include 1.13 and 1.13.1

  ansible-playbook -i /etc/ansible/hosts $PWD/pre-install.yml
    
  cd /etc/ansible/playbooks/openshift-ansible/playbooks/
  ansible-playbook -vvv -i /etc/ansible/hosts $PWD/deploy_cluster.yml

Expected Results

Describe what you expected to happen.

I expected the install and configuration to complete

Observed Results

Everything is chugging along find. The verbose output before the abend is listed below. I don’t see any other errors in the output until this.

TASK [openshift_master_certificates : Generate the loopback master client config] ***************************************************************************************
task path: /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml:63
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '/root\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884 `" && echo ansible-tmp-1521658222.4-93760209595884="` echo /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884 `" ) && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, 'ansible-tmp-1521658222.4-93760209595884=/root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884\n', '')
<uusrcdkub01.fossil.com> PUT /root/.ansible/tmp/ansible-local-20837_E4d2Q/tmpbS7CiZ TO /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884/command.py
<uusrcdkub01.fossil.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 '[uusrcdkub01.fossil.com]'
...
Skip a lot
...
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '/root\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726 `" && echo ansible-tmp-1521656686.1-214052715328726="` echo /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726 `" ) && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, 'ansible-tmp-1521656686.1-214052715328726=/root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726\n', '')
<uusrcdkub01.fossil.com> PUT /root/.ansible/tmp/ansible-local-115241x1n7s/tmp4R_seg TO /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py
<uusrcdkub01.fossil.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 '[uusrcdkub01.fossil.com]'
<uusrcdkub01.fossil.com> (0, 'sftp> put /root/.ansible/tmp/ansible-local-115241x1n7s/tmp4R_seg /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/ /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 -tt uusrcdkub01.fossil.com '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '\r\n{"invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "oc adm create-api-client-config\\n --certificate-authority=/etc/origin/master/ca.crt\\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\\n --groups=system:masters,system:openshift-master\\n --master=https://uusrcdkub03.fossil.com:8443\\n --public-master=https://uusrcdkub03.fossil.com:8443\\n --signer-cert=/etc/origin/master/ca.crt\\n --signer-key=/etc/origin/master/ca.key\\n --signer-serial=/etc/origin/master/ca.serial.txt\\n --user=system:openshift-master\\n --basename=openshift-master\\n --expire-days=720", "removes": null, "creates": "/etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig", "chdir": null, "stdin": null}}, "cmd": "oc adm create-api-client-config\\n --certificate-authority=/etc/origin/master/ca.crt\\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\\n --groups=system:masters,system:openshift-master\\n --master=https://uusrcdkub03.fossil.com:8443\\n --public-master=https://uusrcdkub03.fossil.com:8443\\n --signer-cert=/etc/origin/master/ca.crt\\n --signer-key=/etc/origin/master/ca.key\\n --signer-serial=/etc/origin/master/ca.serial.txt\\n --user=system:openshift-master\\n --basename=openshift-master\\n --expire-days=720", "stdout": "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists", "rc": 0, "changed": false}\r\n', 'Shared connection to uusrcdkub01.fossil.com closed.\r\n')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/ > /dev/null 2>&1 && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '', '')
ok: [uusrcdkub01.fossil.com -> uusrcdkub01.fossil.com] => (item=uusrcdkub03.fossil.com) => {
    "changed": false,
    "cmd": "oc adm create-api-client-config\n --certificate-authority=/etc/origin/master/ca.crt\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\n --groups=system:masters,system:openshift-master\n --master=https://uusrcdkub03.fossil.com:8443\n --public-master=https://uusrcdkub03.fossil.com:8443\n --signer-cert=/etc/origin/master/ca.crt\n --signer-key=/etc/origin/master/ca.key\n --signer-serial=/etc/origin/master/ca.serial.txt\n --user=system:openshift-master\n --basename=openshift-master\n --expire-days=720",
    "invocation": {
        "module_args": {
            "_raw_params": "oc adm create-api-client-config\n --certificate-authority=/etc/origin/master/ca.crt\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\n --groups=system:masters,system:openshift-master\n --master=https://uusrcdkub03.fossil.com:8443\n --public-master=https://uusrcdkub03.fossil.com:8443\n --signer-cert=/etc/origin/master/ca.crt\n --signer-key=/etc/origin/master/ca.key\n --signer-serial=/etc/origin/master/ca.serial.txt\n --user=system:openshift-master\n --basename=openshift-master\n --expire-days=720",
            "_uses_shell": false,
            "chdir": null,
            "creates": "/etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig",
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": "uusrcdkub03.fossil.com",
    "rc": 0,
    "stdout": "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists",
    "stdout_lines": [
        "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists"
    ]
}
ERROR! |failed expects hostvars is a dict