airflow: CLI 'kubernetes cleanup-pods' fails on invalid label key
Apache Airflow version: 2.0.2 Helm chart version: 1.0.0 Kubernetes version: 1.20
What happened: Airflow airflow-cleanup cronjob is failing with the error below. When I run the same command form the webserver or scheduler pod I got the same error.
> airflow@airflow-webserver-7f9f7954c-p9vv9:/opt/airflow$ airflow kubernetes cleanup-pods --namespace airflow
Loading Kubernetes configuration
Listing pods in namespace airflow
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 89, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/kubernetes_command.py", line 111, in cleanup_pods
pod_list = kube_client.list_namespaced_pod(**list_kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py", line 12803, in list_namespaced_pod
(data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py", line 12905, in list_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 345, in call_api
_preload_content, _request_timeout)
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
_request_timeout=_request_timeout)
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 366, in request
headers=headers)
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
query_params=query_params)
File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '53ee7655-f595-42a5-bdfb-689067a7fe02', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'e14ece85-9601-4034-9a43-7872ebabcbc5', 'X-Kubernetes-Pf-Prioritylevel-Uid': '72601873-fd48-4405-99dc-b7c4cac03b5c', 'Date': 'Sun, 23 May 2021 16:07:37 GMT', 'Content-Length': '428'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable to parse requirement: invalid label key \"{'matchExpressions':\": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')","reason":"BadRequest","code":400}
How to reproduce it: Create and airflow deployment with Helm chart Enable automatic cleanup
cleanup:
enabled: true
Run command airflow kubernetes cleanup-pods --namespace airflow
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 16 (12 by maintainers)
Commits related to this issue
- Fixed issue #16013 with kubernetes cleanup-pods CLI — committed to dlampa/airflow by dlampa 3 years ago
- Fix CLI 'kubernetes cleanup-pods' which fails on invalid label key (#17298) Fix for #16013 - CLI 'kubernetes cleanup-pods' fails on invalid label key — committed to apache/airflow by dlampa 3 years ago
- Fix CLI 'kubernetes cleanup-pods' which fails on invalid label key (#17298) Fix for #16013 - CLI 'kubernetes cleanup-pods' fails on invalid label key (cherry picked from commit 36bdfe8d0ef7e5fc42843... — committed to apache/airflow by dlampa 3 years ago
- Fix CLI 'kubernetes cleanup-pods' which fails on invalid label key (#17298) Fix for #16013 - CLI 'kubernetes cleanup-pods' fails on invalid label key (cherry picked from commit 36bdfe8d0ef7e5fc42843... — committed to apache/airflow by dlampa 3 years ago
- Fix CLI 'kubernetes cleanup-pods' which fails on invalid label key (#17298) Fix for #16013 - CLI 'kubernetes cleanup-pods' fails on invalid label key (cherry picked from commit 36bdfe8d0ef7e5fc42843... — committed to apache/airflow by dlampa 3 years ago
- Fix CLI 'kubernetes cleanup-pods' which fails on invalid label key (#17298) Fix for #16013 - CLI 'kubernetes cleanup-pods' fails on invalid label key (cherry picked from commit 36bdfe8d0ef7e5fc42843... — committed to apache/airflow by dlampa 3 years ago
Hi all, thanks to @dlampa , we have a fix for this issue now (https://github.com/apache/airflow/pull/17298).
The fix is merged into
main
branch now. I have marked it for milestone2.1.3
, so you should be able to expect having this fix in the next release2.1.3
.Hi @jean-malo , let’s continue the discussion at @dlampa 's PR #17298 ?
I will add my comments there later.
Thanks!
Hey @jean-malo I’ve reached the same conclusion re.
V1LabelSelector
and had the solution ready for a few days - please take a look at #17298 . I’ve created a simple script using the same approach as in the CLI outside the Airflow environment and had exactly the same error - this definitely looks like akubernetes-client
issue.Thanks @andormarkus
Something I have in mind may be a sample pod yaml in your environment, i.e. the output of
kubectl get pod your_pod_name -o yaml
, something like that (status
part in the yaml is not necessary; if you can provide it, please do remember to mask/remove the potentially sensitive information inside)(I’m a bit busy recently. If I don’t get back here in the coming few days, please feel free to ping me here for reminding. Thanks😄)
Hi @XD-DENG, Im more than happy to provide related information however I need guidance how to provide it.