openshift-ansible: Failures with ansible 2.2.1.0

We’re seeing yaml rendering issues with our code and 2.2.1.0. Need to sort out if we need to fix something or if ansible will be fixing things.

Upstream references https://github.com/ansible/ansible/issues/20290 https://github.com/ansible/ansible/issues/20253

    TASK [openshift_master : set_fact] *********************************************
    task path: /home/rdu/sdodson/git/Openshift/openshift-ansible/roles/openshift_master/tasks/main.yml:146
    An exception occurred during task execution. The full traceback is:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
        res = self._execute()
      File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 443, in _execute
        self._task.post_validate(templar=templar)
      File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 248, in post_validate
        super(Task, self).post_validate(templar)
      File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 373, in post_validate
        value = templar.template(getattr(self, name))
      File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 427, in template
        disable_lookups=disable_lookups,
      File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 383, in template
        disable_lookups=disable_lookups,
      File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 583, in do_template
        res = j2_concat(rf)
      File "<template>", line 9, in root
      File "/home/rdu/sdodson/git/Openshift/openshift-ansible/filter_plugins/openshift_master.py", line 488, in translate_idps
        return yaml.safe_dump([idp.to_dict() for idp in idp_list], default_flow_style=False)
      File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 218, in safe_dump
        return dump_all([data], stream, Dumper=SafeDumper, **kwds)
      File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 190, in dump_all
        dumper.represent(data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 28, in represent
        node = self.represent_data(data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 57, in represent_data
        node = self.yaml_representers[data_types[0]](self, data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 215, in represent_list
        return self.represent_sequence(u'tag:yaml.org,2002:seq', data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 101, in represent_sequence
        node_item = self.represent_data(item)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 57, in represent_data
        node = self.yaml_representers[data_types[0]](self, data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 223, in represent_dict
        return self.represent_mapping(u'tag:yaml.org,2002:map', data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping
        node_value = self.represent_data(item_value)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 67, in represent_data
        node = self.yaml_representers[None](self, data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 247, in represent_undefined
        raise RepresenterError("cannot represent an object: %s" % data)
    RepresenterError: cannot represent an object: htpasswd_auth
     
    fatal: [ose3-master.example.com]: FAILED! => {
        "failed": true,
        "msg": "Unexpected failure during module execution.",
        "stdout": ""
    }

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 2
Comments: 51 (25 by maintainers)

Commits related to this issue

Temp ansible version change to 2.2.0.0 due to bug: https://github.com/openshift/openshift-ansible/issues/3111 — committed to BorisB2015/azure-quickstart-templates by deleted user 7 years ago
openshift: Use ansible 2.2.1.0-2 from the paas sig repo This build contains a fix for ansible that lets the installation of openshift (via openshift-ansible) pass. See: https://github.com/openshift/... — committed to obnoxxx/vagrant-heketi by obnoxxx 7 years ago

Most upvoted comments

Well, personally, I think we should ship an Openshift-ansible container which has all the bits.

jberkus on Mar 4, 2017

same problem with ansible 2.2.2.0

An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud02.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<c4-container-cloud06.bj> ESTABLISH SSH CONNECTION FOR USER: root
<c4-container-cloud06.bj> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r c4-container-cloud06.bj '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<c4-container-cloud07.bj> ESTABLISH SSH CONNECTION FOR USER: root
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud04.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
<c4-container-cloud07.bj> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r c4-container-cloud07.bj '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud03.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<c4-container-cloud08.bj> ESTABLISH SSH CONNECTION FOR USER: root
<c4-container-cloud08.bj> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r c4-container-cloud08.bj '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud05.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud07.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud06.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud08.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud01.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.

x1957 on Apr 15, 2017

Looking forward to it. When it’s available, we can close this.

jberkus on Mar 30, 2017

Ansible 2.2.2.0 has been released, it should be available in F25 and EPEL shortly. OCP channels will be updated after we’ve validated this version.

sdodson on Mar 28, 2017

@Delmonte3161 That’s the same problem. You can get ansible-2.2.1.0-2.el7 from the Centos PaaS SIG repo at http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/

sdodson on Mar 7, 2017

@jberkus @maxamillion the current situation is far from ideal by any means.

To give a bit more context:

After the initial issue was discovered, we started playing whack-a-mole with trying to work around the issue in openshift-ansible before realizing it was a bug in Ansible.
The Ansible team provided us with a patched 2.2.1 build that resolved the issues we were seeing and we expected a 2.2.1.1 release.
As an interim fix for RHEL and CentOS releases (which are both shipped via separate repos from the main OS repo) we worked out an agreement with the Ansible team to ship the patched 2.2.1 release to the OpenShift repos.
Because 2.2.2.0 appeared to be imminent and contained the same fix, both the OpenShift and the Ansible teams agreed to wait on 2.2.2
Ansible 2.2.2.0 has been delayed while trying to address other issues that have surfaced as a result of the security fixes contained in 2.2.1.0. While these did not affect openshift-ansible, but we’re still breaking production uses of Ansible.

I’m wondering if it would make sense to deliver OpenShift and openshift-ansible through a separate repo (or ideally repos, where each version is a separate repo) similar to how we are shipping in RHEL and CentOS. I think there are quite a few different reasons for why, but in this particular case, it would have allowed us to ship the patched Ansible version without affecting non-OpenShift users.

detiber on Mar 3, 2017

Just downgrade Ansible to from 2.2.1 to 2.2.0 using the YUM command below. (no other commands required)

# yum downgrade ansible

jospradlin on Feb 7, 2017

We’ve also got ansible-2.2.0.0 in the centos paas sig common repo http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/common/ansible-2.2.0.0-1.el7.noarch.rpm If you choose to downgrade to 2.2.0.0 it should be noted that version is vulnerable to CVE-2016-9587, see https://bugzilla.redhat.com/show_bug.cgi?id=1404378

sdodson on Jan 22, 2017

I can confirm that downgrading ansible-2.2.1.0-1.el7.noarch.rpm from epel-release-7-9 to ansible-2.2.0.0-4.el7.noarch.rpm from epel-release-7-8 is a workaround for a successful openshift-ansible v1.4 installation. Regards, Armin.

ArminMW on Jan 21, 2017