python: very slow performance due to excessive reconfiguration in modells
What happened (please include outputs or screenshots):
Running this simple script in a big cluster takes about 30 seconds to execute:
import kubernetes as k8s
k8s.config.load_kube_config()
apps = k8s.client.AppsV1Api()
print(k8s.__version__)
print(len(apps.list_replica_set_for_all_namespaces().items))
time python3 script:
24.2.0
3661
real 0m32.394s
user 0m17.764s
sys 0m11.235s
as you see it has very high system cpu usage running this under a profiler shows following interesting things:
957/1 0.002 0.000 32.177 32.177 {built-in method builtins.exec}
1 0.173 0.173 32.177 32.177 test.py:1(<module>)
1 0.000 0.000 31.563 31.563 apps_v1_api.py:3804(list_replica_set_for_all_namespaces)
1 0.000 0.000 31.563 31.563 apps_v1_api.py:3838(list_replica_set_for_all_namespaces_with_http_info)
1 0.000 0.000 31.563 31.563 api_client.py:305(call_api)
1 0.018 0.018 31.563 31.563 api_client.py:120(__call_api)
1 0.000 0.000 28.090 28.090 api_client.py:244(deserialize)
780711/1 0.997 0.000 27.281 27.281 api_client.py:266(__deserialize)
190593/1 0.934 0.000 27.281 27.281 api_client.py:620(__deserialize_model)
36658/1 0.078 0.000 27.281 27.281 api_client.py:280(<listcomp>)
190594 0.839 0.000 22.939 0.000 configuration.py:75(__init__)
190594 0.108 0.000 10.911 0.000 context.py:41(cpu_count)
190594 10.803 0.000 10.803 0.000 {built-in method posix.cpu_count}
190597 0.300 0.000 8.327 0.000 configuration.py:253(debug)
381195 0.216 0.000 7.443 0.000 __init__.py:1448(setLevel)
381195 4.364 0.000 7.117 0.000 __init__.py:1403(_clear_cache)
...
190594 10.803 0.000 10.803 0.000 {built-in method posix.cpu_count}
10 seconds are spent running multiprocessing.cpu_count which accounts for most of the system usage
381195 0.216 0.000 7.443 0.000 __init__.py:1448(setLevel)
7 seconds are spent configuring logging
looking at what causes this appears to be following line in every model:
if local_vars_configuration is None:
local_vars_configuration = Configuration()
This runs the configuration function which sets up logging and calls multiprocessing.cpu_count
commenting the multiprocessing call confirms this, i then runs significantly faster:
real 0m11.964s
user 0m8.162s
sys 0m0.338s
Is there a way to avoid calling Configuration on every model init?
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 18 (13 by maintainers)
Commits related to this issue
- [Python] pass api_client configuration to model deserialize The if not passed the models create a new configuration object which configures logging and determines cpu count every time. This causes ex... — committed to juliantaylor/openapi-generator by juliantaylor 2 years ago
- [Python] pass api_client configuration to model deserialize (#13922) The if not passed the models create a new configuration object which configures logging and determines cpu count every time. Thi... — committed to OpenAPITools/openapi-generator by juliantaylor 2 years ago
this problem still exists in the latest version 28.1.0
if you have no plans to update the generator to the fixed version can you please add this patch locally? this is wasting loads of cpu cycles for every user.