openshift-ansible: Openshift 3.9 provision fails

Description

I have single master and two nodes cluster, all 3 instances are at AWS, all ports opened, amazon ami is in used.

Version

• Your ansible version per ansible --version

ansible 2.4.3.0
  config file = /home/instana/openshift-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

If you’re operating from a git clone:

• The output of git describe

openshift-ansible-3.9.28-1

Error:

failed: [xxxxxxxx] (item={u'name': u'origin-node-3.9.0'}) => {"attempts": 2, "changed": false, "item": {"name": "origin-node-3.9.0"}, "msg": "Failure talking to yum: Could not contact any CDS load balancers: rhui2-cds01.us-east-1.aws.ce.redhat.com, rhui2-cds02.us-east-1.aws.ce.redhat.com."}
failed: [54.204.197.172] (item={u'name': u'origin-node-3.9.0'}) => {"attempts": 2, "changed": false, "item": {"name": "origin-node-3.9.0"}, "msg": "Failure talking to yum: Could not contact any CDS load balancers: rhui2-cds01.us-east-1.aws.ce.redhat.com, rhui2-cds02.us-east-1.aws.ce.redhat.com."}
FAILED - RETRYING: Install Node package, sdn-ovs, conntrack packages (2 retries left).
FAILED - RETRYING: Install Node package, sdn-ovs, conntrack packages (2 retries left).
FAILED - RETRYING: Install Node package, sdn-ovs, conntrack packages (1 retries left).
FAILED - RETRYING: Install Node package, sdn-ovs, conntrack packages (1 retries left).
failed: [xxxxxxxx] (item={u'name': u'origin-sdn-ovs-3.9.0', u'install': True}) => {"attempts": 2, "changed": false, "item": {"install": true, "name": "origin-sdn-ovs-3.9.0"}, "msg": "Failure talking to yum: Could not contact any CDS load balancers: rhui2-cds01.us-east-1.aws.ce.redhat.com, rhui2-cds02.us-east-1.aws.ce.redhat.com."}
failed: [xxxxxxxx] (item={u'name': u'origin-sdn-ovs-3.9.0', u'install': True}) => {"attempts": 2, "changed": false, "item": {"install": true, "name": "origin-sdn-ovs-3.9.0"}, "msg": "Failure talking to yum: Could not contact any CDS load balancers: rhui2-cds01.us-east-1.aws.ce.redhat.com, rhui2-cds02.us-east-1.aws.ce.redhat.com."}
FAILED - RETRYING: Install Node package, sdn-ovs, conntrack packages (2 retries left).
FAILED - RETRYING: Install Node package, sdn-ovs, conntrack packages (2 retries left).
FAILED - RETRYING: Install Node package, sdn-ovs, conntrack packages (1 retries left).
FAILED - RETRYING: Install Node package, sdn-ovs, conntrack packages (1 retries left).
failed: [xxxxxxxx] (item={u'name': u'conntrack-tools'}) => {"attempts": 2, "changed": false, "item": {"name": "conntrack-tools"}, "msg": "Failure talking to yum: Could not contact any CDS load balancers: rhui2-cds01.us-east-1.aws.ce.redhat.com, rhui2-cds02.us-east-1.aws.ce.redhat.com."}
failed: [xxxxxxxxxx] (item={u'name': u'conntrack-tools'}) => {"attempts": 2, "changed": false, "item": {"name": "conntrack-tools"}, "msg": "Failure talking to yum: Could not contact any CDS load balancers: rhui2-cds01.us-east-1.aws.ce.redhat.com, rhui2-cds02.us-east-1.aws.ce.redhat.com."}

Steps To Reproduce

1. [step 1] Run ansible-playbook -i inventory/instanatest playbooks/prerequisites.yml
2. [step 2] Run ansible-playbook -i inventory/instanatest playbooks/deploy_cluster.yml

Expected Results

To get working Openshift 3.9

The important thing that I noticed is that ansible is chaning nameserver and the content of the etc/resolv.conf is:

search cluster.local ec2.internal
nameserver 172.31.39.24

if I try to resolve anything on internet, like nslookup google.com i will get no answer. Something is wrong with DNS since instances cannot resolve queries.

Inventory file:

[masters]
master_aws_eip

[etcd]
master_aws_eip

[nodes]
node1_aws_eip
node2_aws_eip

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
#openshift_enable_unsupported_configurations=false
ansible_ssh_user=dusansusic
ansible_become=yes
#debug_level=2
openshift_disable_check=memory_availability,disk_availability,docker_storage
ansible_ssh_private_key_file=/home/dusansusic/.ssh/id_rsa
openshift_deployment_type=origin
openshift_release=v3.9
#openshift_image_tag=v3.6.0
#openshift_pkg_version=-3.6.0
#openshift_use_system_containers=False
#openshift_use_node_system_container=False
#openshift_use_master_system_container=False
#openshift_use_etcd_system_container=False
#openshift_install_examples=true
#openshift_master_logout_url=http://example.com
#openshift_master_extension_scripts=['/path/to/script1.js','/path/to/script2.js']
#openshift_master_extension_stylesheets=['/path/to/stylesheet1.css','/path/to/stylesheet2.css']
#openshift_master_extensions=[{'name': 'images', 'sourceDirectory': '/path/to/my_images'}]
#openshift_master_oauth_template=/path/to/login-template.html
#openshift_master_image_policy_config={"maxImagesBulkImportedPerRepository": 3, "disableScheduledImport": true}
#openshift_docker_additional_registries=registry.example.com
#openshift_docker_insecure_registries=registry.example.com
#openshift_docker_blocked_registries=registry.hacker.com
#openshift_docker_disable_push_dockerhub=True
#openshift_docker_use_system_container=False
#openshift_docker_systemcontainer_image_registry_override="registry.example.com"
#openshift_docker_options="-l warn --ipv6=false"
#openshift_docker_selinux_enabled=False
#oreg_url_master=example.com/openshift3/ose-${component}:${version}
#oreg_url_node=example.com/openshift3/ose-${component}:${version}
#oreg_url=example.com/openshift3/ose-${component}:${version}
#openshift_examples_modify_imagestreams=true
#openshift_additional_repos=[{'id': 'openshift-origin-copr', 'name': 'OpenShift Origin COPR', 'baseurl': 'https://copr-be.cloud.fedoraproject.org/results/maxamillion/origin-next/epel-7-$basearch/', 'enabled': 1, 'gpgcheck': 1, 'gpgkey': 'https://copr-be.cloud.fedoraproject.org/results/maxamillion/origin-next/pubkey.gpg'}]
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
#openshift_master_htpasswd_users={'user1': '<pre-hashed password>', 'user2': '<pre-hashed password>'}
#openshift_master_htpasswd_file=<path to local pre-generated htpasswd file>
#openshift_master_identity_providers=[{'name': 'allow_all', 'login': 'true', 'challenge': 'true', 'kind': 'AllowAllPasswordIdentityProvider'}]
#openshift_master_identity_providers=[{'name': 'my_ldap_provider', 'challenge': 'true', 'login': 'true', 'kind': 'LDAPPasswordIdentityProvider', 'attributes': {'id': ['dn'], 'email': ['mail'], 'name': ['cn'], 'preferredUsername': ['uid']}, 'bindDN': '', 'bindPassword': '', 'ca': 'my-ldap-ca.crt', 'insecure': 'false', 'url': 'ldap://ldap.example.com:389/ou=users,dc=example,dc=com?uid'}]
#openshift_master_ldap_ca_file=<path to local ca file to use>
#openshift_master_identity_providers=[{"name": "openid_auth", "login": "true", "challenge": "false", "kind": "OpenIDIdentityProvider", "client_id": "my_client_id", "client_secret": "my_client_secret", "claims": {"id": ["sub"], "preferredUsername": ["preferred_username"], "name": ["name"], "email": ["email"]}, "urls": {"authorize": "https://myidp.example.com/oauth2/authorize", "token": "https://myidp.example.com/oauth2/token"}, "ca": "my-openid-ca-bundle.crt"}]
#openshift_master_openid_ca_file=<path to local ca file to use>
#openshift_master_identity_providers=[{"name": "my_request_header_provider", "challenge": "true", "login": "true", "kind": "RequestHeaderIdentityProvider", "challengeURL": "https://www.example.com/challenging-proxy/oauth/authorize?${query}", "loginURL": "https://www.example.com/login-proxy/oauth/authorize?${query}", "clientCA": "my-request-header-ca.crt", "clientCommonNames": ["my-auth-proxy"], "headers": ["X-Remote-User", "SSO-User"], "emailHeaders": ["X-Remote-User-Email"], "nameHeaders": ["X-Remote-User-Display-Name"], "preferredUsernameHeaders": ["X-Remote-User-Login"]}]
#openshift_master_request_header_ca_file=<path to local ca file to use>
#openshift_cfme_install_app=False
openshift_cloudprovider_aws_access_key=
openshift_cloudprovider_aws_secret_key=+b
openshift_cloudprovider_kind=aws
#openshift_cloudprovider_aws_access_key=aws_access_key_id
#openshift_cloudprovider_aws_secret_key=aws_secret_access_key
#openshift_cloudprovider_kind=openstack
#openshift_cloudprovider_openstack_auth_url=http://openstack.example.com:35357/v2.0/
#openshift_cloudprovider_openstack_username=username
#openshift_cloudprovider_openstack_password=password
#openshift_cloudprovider_openstack_domain_id=domain_id
#openshift_cloudprovider_openstack_domain_name=domain_name
#openshift_cloudprovider_openstack_tenant_id=tenant_id
#openshift_cloudprovider_openstack_tenant_name=tenant_name
#openshift_cloudprovider_openstack_region=region
#openshift_cloudprovider_openstack_lb_subnet_id=subnet_id
#openshift_cloudprovider_openstack_blockstorage_version=v2
#openshift_cloudprovider_kind=gce
#osm_project_request_message=''
#osm_project_request_template=''
#osm_mcs_allocator_range='s0:/2'
#osm_mcs_labels_per_project=5
#osm_uid_allocator_range='1000000000-1999999999/10000'
#openshift_additional_projects={'my-project': {'default_node_selector': 'label=value'}}
#osm_use_cockpit=true
#osm_cockpit_plugins=['cockpit-kubernetes']
#openshift_master_cluster_method=native
#openshift_master_cluster_hostname=openshift-ansible.test.example.com
#openshift_master_cluster_public_hostname=openshift-ansible.test.example.com
#openshift_master_cluster_method=pacemaker
#openshift_master_cluster_password=openshift_cluster
#openshift_master_cluster_vip=192.168.133.25
#openshift_master_cluster_public_vip=192.168.133.25
#openshift_master_cluster_hostname=openshift-ansible.test.example.com
#openshift_master_cluster_public_hostname=openshift-ansible.test.example.com
#osm_controller_lease_ttl=30
#osm_controller_args={'resource-quota-sync-period': ['10s']}
#osm_api_server_args={'max-requests-inflight': ['400']}
#openshift_master_default_subdomain=apps.test.example.com
#osm_custom_cors_origins=['foo.example.com', 'bar.example.com']
#osm_default_node_selector='region=primary'
#openshift_master_pod_eviction_timeout=5m
#openshift_master_max_requests_inflight=500
#openshift_master_min_tls_version=VersionTLS12
#openshift_master_cipher_suites=['TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', '...']
#openshift_node_cipher_suites=['TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', '...']
#osn_storage_plugin_deps=['ceph','glusterfs','iscsi']
#openshift_hosted_router_selector='region=infra'
#openshift_hosted_router_replicas=2
#openshift_hosted_router_force_subdomain='${name}-${namespace}.apps.example.com'
#openshift_hosted_router_certificate={"certfile": "/path/to/router.crt", "keyfile": "/path/to/router.key", "cafile": "/path/to/router-ca.crt"}
#openshift_hosted_manage_router=true
#openshift_hosted_routers:
#- name: router1
#- name: router2
#openshift_cockpit_deployer_prefix=registry.example.com/myrepo/
#openshift_cockpit_deployer_version=1.4.1
#openshift_hosted_registry_selector='region=infra'
#openshift_hosted_registry_replicas=2
#openshift_hosted_registry_cert_expire_days=730
#openshift_hosted_manage_registry=true
#openshift_hosted_registry_storage_kind=nfs
#openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
#openshift_hosted_registry_storage_nfs_directory=/exports
#openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
#openshift_hosted_registry_storage_volume_name=registry
#openshift_hosted_registry_storage_volume_size=10Gi
#openshift_hosted_registry_storage_kind=nfs
#openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
#openshift_hosted_registry_storage_host=nfs.example.com
#openshift_hosted_registry_storage_nfs_directory=/exports
#openshift_hosted_registry_storage_volume_name=registry
#openshift_hosted_registry_storage_volume_size=10Gi
#openshift_hosted_registry_storage_kind=openstack
#openshift_hosted_registry_storage_access_modes=['ReadWriteOnce']
#openshift_hosted_registry_storage_openstack_filesystem=ext4
#openshift_hosted_registry_storage_openstack_volumeID=3a650b4f-c8c5-4e0a-8ca5-eaee11f16c57
#openshift_hosted_registry_storage_volume_size=10Gi
#openshift_hosted_registry_storage_kind=object
#openshift_hosted_registry_storage_provider=s3
#openshift_hosted_registry_storage_s3_encrypt=false
#openshift_hosted_registry_storage_s3_kmskeyid=aws_kms_key_id
#openshift_hosted_registry_storage_s3_accesskey=aws_access_key_id
#openshift_hosted_registry_storage_s3_secretkey=aws_secret_access_key
#openshift_hosted_registry_storage_s3_bucket=bucket_name
#openshift_hosted_registry_storage_s3_region=bucket_region
#openshift_hosted_registry_storage_s3_chunksize=26214400
#openshift_hosted_registry_storage_s3_rootdirectory=/registry
#openshift_hosted_registry_pullthrough=true
#openshift_hosted_registry_acceptschema2=true
#openshift_hosted_registry_enforcequota=true
#openshift_hosted_registry_storage_kind=object
#openshift_hosted_registry_storage_provider=s3
#openshift_hosted_registry_storage_s3_accesskey=access_key_id
#openshift_hosted_registry_storage_s3_secretkey=secret_access_key
#openshift_hosted_registry_storage_s3_regionendpoint=https://myendpoint.example.com/
#openshift_hosted_registry_storage_s3_bucket=bucket_name
#openshift_hosted_registry_storage_s3_region=bucket_region
#openshift_hosted_registry_storage_s3_chunksize=26214400
#openshift_hosted_registry_storage_s3_rootdirectory=/registry
#openshift_hosted_registry_pullthrough=true
#openshift_hosted_registry_acceptschema2=true
#openshift_hosted_registry_enforcequota=true
#openshift_hosted_registry_storage_s3_cloudfront_baseurl=https://myendpoint.cloudfront.net/
#openshift_hosted_registry_storage_s3_cloudfront_privatekeyfile=/full/path/to/secret.pem
#openshift_hosted_registry_storage_s3_cloudfront_keypairid=yourpairid
#openshift_hosted_metrics_storage_kind=nfs
#openshift_hosted_metrics_storage_access_modes=['ReadWriteOnce']
#openshift_hosted_metrics_storage_nfs_directory=/exports
#openshift_hosted_metrics_storage_nfs_options='*(rw,root_squash)'
#openshift_hosted_metrics_storage_volume_name=metrics
#openshift_hosted_metrics_storage_volume_size=10Gi
#openshift_hosted_metrics_storage_labels={'storage': 'metrics'}
#openshift_hosted_metrics_storage_kind=nfs
#openshift_hosted_metrics_storage_access_modes=['ReadWriteOnce']
#openshift_hosted_metrics_storage_host=nfs.example.com
#openshift_hosted_metrics_storage_nfs_directory=/exports
#openshift_hosted_metrics_storage_volume_name=metrics
#openshift_hosted_metrics_storage_volume_size=10Gi
#openshift_hosted_metrics_storage_labels={'storage': 'metrics'}
#openshift_hosted_metrics_storage_kind=dynamic
#openshift_hosted_metrics_public_url=https://hawkular-metrics.example.com/hawkular/metrics
#openshift_hosted_metrics_deployer_prefix=docker.io/openshift/origin-
#openshift_hosted_metrics_deployer_version=3.6.0
#openshift_hosted_logging_deploy=true
#openshift_hosted_logging_storage_kind=nfs
#openshift_hosted_logging_storage_access_modes=['ReadWriteOnce']
#openshift_hosted_logging_storage_nfs_directory=/exports
#openshift_hosted_logging_storage_nfs_options='*(rw,root_squash)'
#openshift_hosted_logging_storage_volume_name=logging
#openshift_hosted_logging_storage_volume_size=10Gi
#openshift_hosted_logging_storage_labels={'storage': 'logging'}
#openshift_hosted_logging_storage_kind=nfs
#openshift_hosted_logging_storage_access_modes=['ReadWriteOnce']
#openshift_hosted_logging_storage_host=nfs.example.com
#openshift_hosted_logging_storage_nfs_directory=/exports
#openshift_hosted_logging_storage_volume_name=logging
#openshift_hosted_logging_storage_volume_size=10Gi
#openshift_hosted_logging_storage_labels={'storage': 'logging'}
#openshift_hosted_logging_storage_kind=dynamic
#openshift_hosted_logging_hostname=logging.apps.example.com
#openshift_hosted_logging_elasticsearch_cluster_size=1
#openshift_hosted_logging_deployer_prefix=docker.io/openshift/origin-
#openshift_hosted_logging_deployer_version=3.6.0
#osm_cluster_network_cidr=10.128.0.0/14
#openshift_portal_net=172.30.0.0/16
#openshift_master_external_ip_network_cidrs=['0.0.0.0/0']
#openshift_master_ingress_ip_network_cidr=172.46.0.0/16
#osm_host_subnet_length=9
#openshift_master_api_port=8443
#openshift_master_console_port=8443
#openshift_pkg_version=-1.1
#openshift_master_ca_certificate={'certfile': '/path/to/ca.crt', 'keyfile': '/path/to/ca.key'}
#openshift_master_overwrite_named_certificates=true
#openshift_master_named_certificates=[{"certfile": "/path/to/custom1.crt", "keyfile": "/path/to/custom1.key", "cafile": "/path/to/custom-ca1.crt"}]
#openshift_master_named_certificates=[{"certfile": "/path/to/custom1.crt", "keyfile": "/path/to/custom1.key", "names": ["public-master-host.com"], "cafile": "/path/to/custom-ca1.crt"}]
#openshift_master_session_name=ssn
#openshift_master_session_max_seconds=3600
#openshift_master_session_auth_secrets=['DONT+USE+THIS+SECRET+']
#openshift_master_session_encryption_secrets=['DONT+USE+THIS+SECRET+b4NV+']
#openshift_node_iptables_sync_period=5s
#openshift_set_node_ip=True
#openshift_set_hostname=True
#openshift_dns_ip=172.30.0.1
#openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['85'], 'image-gc-low-threshold': ['80']}
#logrotate_scripts=[{"name": "syslog", "path": "/var/log/cron\n/var/log/maillog\n/var/log/messages\n/var/log/secure\n/var/log/spooler\n", "options": ["daily", "rotate 7", "compress", "sharedscripts", "missingok"], "scripts": {"postrotate": "/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true"}}]
#openshift_override_hostname_check=true
#openshift_use_dnsmasq=False
#openshift_node_dnsmasq_additional_config_file=/home/bob/ose-dnsmasq.conf
#openshift_http_proxy=http://USER:PASSWORD@IPADDR:PORT
#openshift_https_proxy=https://USER:PASSWORD@IPADDR:PORT
#openshift_no_proxy='.hosts.example.com,some-host.com'
#openshift_generate_no_proxy_hosts=True
#openshift_builddefaults_http_proxy=http://USER:PASSWORD@HOST:PORT
#openshift_builddefaults_https_proxy=https://USER:PASSWORD@HOST:PORT
#openshift_builddefaults_no_proxy=mycorp.com
#openshift_builddefaults_git_http_proxy=http://USER:PASSWORD@HOST:PORT
#openshift_builddefaults_git_https_proxy=https://USER:PASSWORD@HOST:PORT
#openshift_builddefaults_git_no_proxy=mycorp.com
#openshift_builddefaults_image_labels=[{'name':'imagelabelname1','value':'imagelabelvalue1'}]
#openshift_builddefaults_nodeselectors={'nodelabel1':'nodelabelvalue1'}
#openshift_builddefaults_annotations={'annotationkey1':'annotationvalue1'}
#openshift_builddefaults_resources_requests_cpu=100m
#openshift_builddefaults_resources_requests_memory=256m
#openshift_builddefaults_resources_limits_cpu=1000m
#openshift_builddefaults_resources_limits_memory=512m
#openshift_builddefaults_json='{"BuildDefaults":{"configuration":{"apiVersion":"v1","env":[{"name":"HTTP_PROXY","value":"http://proxy.example.com.redhat.com:3128"},{"name":"NO_PROXY","value":"ose3-master.example.com"}],"gitHTTPProxy":"http://proxy.example.com:3128","gitNoProxy":"ose3-master.example.com","kind":"BuildDefaultsConfig"}}}'
#openshift_buildoverrides_force_pull=true
#openshift_buildoverrides_image_labels=[{'name':'imagelabelname1','value':'imagelabelvalue1'}]
#openshift_buildoverrides_nodeselectors={'nodelabel1':'nodelabelvalue1'}
#openshift_buildoverrides_annotations={'annotationkey1':'annotationvalue1'}
#openshift_buildoverrides_json='{"BuildOverrides":{"configuration":{"apiVersion":"v1","kind":"BuildDefaultsConfig","forcePull":"true"}}}'
#openshift_template_service_broker_namespaces=['openshift']
#openshift_master_dynamic_provisioning_enabled=False
#openshift_master_admission_plugin_config={"ProjectRequestLimit":{"configuration":{"apiVersion":"v1","kind":"ProjectRequestLimitConfig","limits":[{"selector":{"admin":"true"}},{"maxProjects":"1"}]}},"PodNodeConstraints":{"configuration":{"apiVersion":"v1","kind":"PodNodeConstraintsConfig"}}}
#openshift_clock_enabled=true
#openshift_master_api_env_vars={"ENABLE_HTTP2": "true"}
#openshift_master_controllers_env_vars={"ENABLE_HTTP2": "true"}
#openshift_node_env_vars={"ENABLE_HTTP2": "true"}
#openshift_master_audit_config={"enabled": true}
#openshift_master_audit_config={"enabled": true, "auditFilePath": "/var/log/openpaas-oscp-audit/openpaas-oscp-audit.log", "maximumFileRetentionDays": 14, "maximumFileSizeMegabytes": 500, "maximumRetainedFiles": 5}
#openshift_enable_origin_repo=false
#openshift_node_cert_expire_days=730
#openshift_master_cert_expire_days=730
#openshift_upgrade_nodes_serial=1
#openshift_upgrade_nodes_max_fail_percentage=0
#openshift_upgrade_control_plane_nodes_serial=1
#openshift_upgrade_control_plane_nodes_max_fail_percentage=0
openshift_clusterid=qa
openshift_repos_enable_testing=true