python: very slow performance due to excessive reconfiguration in modells
What happened (please include outputs or screenshots):
Running this simple script in a big cluster takes about 30 seconds to execute:
import kubernetes as k8s
k8s.config.load_kube_config()
apps = k8s.client.AppsV1Api()
print(k8s.__version__)
print(len(apps.list_replica_set_for_all_namespaces().items))
time python3 script:
24.2.0
3661
real	0m32.394s
user	0m17.764s
sys	0m11.235s
as you see it has very high system cpu usage running this under a profiler shows following interesting things:
    957/1    0.002    0.000   32.177   32.177 {built-in method builtins.exec}
        1    0.173    0.173   32.177   32.177 test.py:1(<module>)
        1    0.000    0.000   31.563   31.563 apps_v1_api.py:3804(list_replica_set_for_all_namespaces)
        1    0.000    0.000   31.563   31.563 apps_v1_api.py:3838(list_replica_set_for_all_namespaces_with_http_info)
        1    0.000    0.000   31.563   31.563 api_client.py:305(call_api)
        1    0.018    0.018   31.563   31.563 api_client.py:120(__call_api)
        1    0.000    0.000   28.090   28.090 api_client.py:244(deserialize)
 780711/1    0.997    0.000   27.281   27.281 api_client.py:266(__deserialize)
 190593/1    0.934    0.000   27.281   27.281 api_client.py:620(__deserialize_model)
  36658/1    0.078    0.000   27.281   27.281 api_client.py:280(<listcomp>)
   190594    0.839    0.000   22.939    0.000 configuration.py:75(__init__)
   190594    0.108    0.000   10.911    0.000 context.py:41(cpu_count)
   190594   10.803    0.000   10.803    0.000 {built-in method posix.cpu_count}
   190597    0.300    0.000    8.327    0.000 configuration.py:253(debug)
   381195    0.216    0.000    7.443    0.000 __init__.py:1448(setLevel)
   381195    4.364    0.000    7.117    0.000 __init__.py:1403(_clear_cache)
...
   190594   10.803    0.000   10.803    0.000 {built-in method posix.cpu_count}
10 seconds are spent running multiprocessing.cpu_count which accounts for most of the system usage
   381195    0.216    0.000    7.443    0.000 __init__.py:1448(setLevel)
7 seconds are spent configuring logging
looking at what causes this appears to be following line in every model:
        if local_vars_configuration is None:
            local_vars_configuration = Configuration()
This runs the configuration function which sets up logging and calls multiprocessing.cpu_count
commenting the multiprocessing call confirms this, i then runs significantly faster:
real	0m11.964s
user	0m8.162s
sys	0m0.338s
Is there a way to avoid calling Configuration on every model init?
About this issue
- Original URL
 - State: open
 - Created 2 years ago
 - Comments: 18 (13 by maintainers)
 
Commits related to this issue
- [Python] pass api_client configuration to model deserialize The if not passed the models create a new configuration object which configures logging and determines cpu count every time. This causes ex... — committed to juliantaylor/openapi-generator by juliantaylor 2 years ago
 - [Python] pass api_client configuration to model deserialize (#13922) The if not passed the models create a new configuration object which configures logging and determines cpu count every time. Thi... — committed to OpenAPITools/openapi-generator by juliantaylor 2 years ago
 
this problem still exists in the latest version 28.1.0
if you have no plans to update the generator to the fixed version can you please add this patch locally? this is wasting loads of cpu cycles for every user.