linkerd2: high cpu and memory usage

Bug Report

What is the issue?

I’m not exactly sure what is going on but it appears that the linkerd proxy on some of our endpoints does not respond. We can also watch the cpu and memory usage start to increase until the pod is eventually killed for being out of resources

How can it be reproduced?

I’m not exactly sure how to reproduce this bug on an external location

Logs, error output, etc

This is the last bit of logs that happen in the proxy:

DBUG [    79.314223s] proxy={server=out listen=127.0.0.1:4140 remote=10.28.5.99:49000} linkerd2_proxy::app::main outbound addr=Ok(Name(NameAddr { name: DNSName("voting-service-p-tzbnm.voting-service-p-tzbnm.svc.cluster.local"), port: 8080 }))
DBUG [    79.314470s] voting-service-p-tzbnm.voting-service-p-tzbnm.svc.cluster.local:8080 linkerd2_proxy::app::main outbound dst=Some(DstAddr { addr: Name(NameAddr { name: DNSName("voting-service-p-tzbnm.voting-service-p-tzbnm.svc.cluster.local"), port: 8080 }), direction: Out })
DBUG [    79.314689s] proxy={server=out listen=127.0.0.1:4140 remote=10.28.5.99:49000} linkerd2_proxy::proxy::http::orig_proto upgrading HTTP/1.1 to HTTP2 with orig-proto
DBUG [    79.314798s] 10.28.4.195:8080 linkerd2_proxy::proxy::http::client client request: method=GET uri=http://voting-service-p-tzbnm.voting-service-p-tzbnm.svc.cluster.local:8080/admin/challenges/overviews?challenge_ids[]=523168a8-19bf-461f-a464-117bf3f807a2&challenge_ids[]=9e77c18c-82a2-4654-9c1f-31fc073bbf15&challenge_ids[]=845ab9b1-fb7e-4df8-8966-b317f48ba221&challenge_ids[]=f4448c84-6a84-4067-bfc3-b637d69f1a78&challenge_ids[]=0f19007d-c169-498a-84a8-aba2a3262aa5&challenge_ids[]=cc1ef53b-1710-47be-9e76-2d557a6c653f&challenge_ids[]=d082314e-e774-44b8-8efe-401527b65190&challenge_ids[]=3244178e-96f0-4a6a-99ab-53aa562ca574&challenge_ids[]=7465272f-8137-4a63-a39d-e817158c4b70&challenge_ids[]=a90b495e-6f2b-4ec8-b3c3-545e96311d25&challenge_ids[]=8359afc0-9ebc-4e8a-ba0e-f1561d9ce28b&challenge_ids[]=8b07b6c0-e141-4eb0-a677-709e9de14e9c&challenge_ids[]=017cd879-22d5-4308-8a17-d8c596d0b7bc&challenge_ids[]=27b8b4b2-e6ca-49b7-9440-d526e4f3e354&challenge_ids[]=283a8086-1ce8-48a9-acde-69ca03d6658d&challenge_ids[]=7f8e5d49-e1e5-4dd6-80cf-65f968977397&challenge_ids[]=af3e2852-2e42-40f4-b0a9-84b1815a3cd0&challenge_ids[]=5e680bac-bcff-4127-8571-0abddc5db8b6&challenge_ids[]=07ef30c6-776d-4cdd-aad9-57effa6d5e9e&challenge_ids[]=d0cd26e7-f8b6-4667-b8b1-3bd84dc59139&challenge_ids[]=62a15068-3c44-4a27-86d3-59bfa036604e&challenge_ids[]=17c617f1-4f8a-4b46-af20-2c16b4133cce&challenge_ids[]=e641d2d9-9b93-4049-86cb-1512a1d74f9c&challenge_ids[]=3b04155c-d968-4917-8b80-c3ee362b15ef&challenge_ids[]=451bd1d5-8903-4f34-a3ab-4600e8d29c16&challenge_ids[]=9b91194b-5a15-43df-ac01-2711110b7a0a&challenge_ids[]=01b8b5e3-5a96-4613-8dfa-68fc09472848&challenge_ids[]=7468d674-a536-43b9-b803-489606ab7144&challenge_ids[]=21092330-3c76-4d62-a68b-b484172ae3d8&challenge_ids[]=b750fccf-74a1-414a-b929-6aab3ed0b4e3&challenge_ids[]=38211bd5-026c-44eb-924b-77adfeff2589&challenge_ids[]=97e6a821-e80c-46d5-80ec-92106ffe8654&challenge_ids[]=20fe83a2-0b7b-4828-9d52-262d96d4917a&challenge_ids[]=7fdf6567-c21b-4d7f-abda-09551f65c6e0&challenge_ids[]=e1e2d72a-94c1-48c3-9625-d95321ba2edf&challenge_ids[]=17707ea6-56db-453f-b740-25fe3a6edc59&challenge_ids[]=9e252c71-b681-4bbc-be71-2d30ee2fb227&challenge_ids[]=bd5d87f7-5617-418c-a53c-946f285ec182&challenge_ids[]=5c66bcec-8c62-4b5f-8632-d57cdbe09e31&challenge_ids[]=73e0555e-3783-4a6d-8b1d-62f7a5006e01&challenge_ids[]=024e8749-5b53-4a5b-a8d3-e6f6794490fb&challenge_ids[]=f26de7af-67fb-4c71-89d1-38ff87fc67be&challenge_ids[]=538bd87d-0e8f-4441-ac36-cbd2ce171510&challenge_ids[]=b9582251-1a76-4aea-b076-8292d457f43e&challenge_ids[]=79a3c310-e087-4280-a168-8cad3c57a4dd&challenge_ids[]=f108944f-528a-4274-a94a-34098e7fada9&challenge_ids[]=75af7478-9ec4-43d3-a51f-8fa85d15c007&challenge_ids[]=b4d5a8e9-52e2-41ec-93bc-049455044a55&challenge_ids[]=16109ba0-b4d3-4c2f-afe8-562fe288d34f&challenge_ids[]=05a5aaa0-e1ae-4bfe-b69b-ca81420b3ac8&challenge_ids[]=ed6d46bc-9593-4f8b-9060-b3cd1f5f0bfa&challenge_ids[]=94a88b98-bf3e-4eb6-aae9-236d858a20c1&challenge_ids[]=8a586f87-0c59-40ec-bf39-b8afc76814d7&challenge_ids[]=28cdb303-cfcf-43f7-aa8c-8bf3488b612e&challenge_ids[]=020bd43a-7b8d-419f-b228-f28fb4dafb46&challenge_ids[]=4645974b-da9b-4b25-87e4-1330edd82e26&challenge_ids[]=13fef78f-b7a6-4373-85d9-adeab87d1e3a&challenge_ids[]=3e9c7694-9409-41cb-865c-571b08d3ebec&challenge_ids[]=dd6851bd-9cad-4658-8ccb-dea9b41c20df&challenge_ids[]=ffa24252-46b2-428f-a6c5-1fa401df5223&challenge_ids[]=51ad00e1-8814-43a6-9bd1-757a304f6225&challenge_ids[]=b40508d1-4ce8-4d20-bd6b-404280887f37&challenge_ids[]=9298b1c0-7872-4691-91e4-54f36a30bdeb&challenge_ids[]=f653a4b2-0626-46f8-aacb-038d27ac3ea5&challenge_ids[]=45114614-27f6-46c5-8113-7dbba60841bf&challenge_ids[]=b017670b-e50e-4318-a9ba-fd90160beafe&challenge_ids[]=4898037d-3e15-4fca-9899-a8225dad7a32&challenge_ids[]=848307c8-0963-4d25-8b4c-24a09ed186c3&challenge_ids[]=20b4a923-fcec-4010-9317-ad871b3226d4&challenge_ids[]=b9d4de98-4abf-4ff0-96bc-d24515a70a93&challenge_ids[]=cf38497d-6af6-4481-baa5-dc3ab7c9cfb6&challenge_ids[]=f7ebc54a-dcb0-44c7-b2cf-578ac91943b3&challenge_ids[]=84c50ca3-ad52-47f9-b693-daffce9cd423&challenge_ids[]=fe788fe7-f508-4d67-8e29-f3eb438b8bca&challenge_ids[]=e97f5c59-364a-4267-b533-b8536fab83c2&challenge_ids[]=f5511549-d197-4a6a-aed7-22ad3807af6f&challenge_ids[]=aa096dcb-5cac-4549-af80-94a5999629c5&challenge_ids[]=ab3afcda-5b4e-45be-b606-7e905fc6a487&challenge_ids[]=3ab3a6cb-8972-4d1f-b18e-efef2fac54ae&challenge_ids[]=d272d515-9147-4f7a-ae50-60a3dc7a8e20&challenge_ids[]=719dcb64-67b7-4556-9d0d-97a53e451e04&challenge_ids[]=0d0bb51c-9bf5-4840-bc11-8304fd1530cf&challenge_ids[]=630ae519-f2fd-4456-ac39-981667a8b98f&challenge_ids[]=a3affa1f-979c-47bd-8fae-d8cda3728df0&challenge_ids[]=471bf787-5960-4df3-a032-93566b18c80a&challenge_ids[]=a7f8baa9-d50d-4fc8-a9af-3f6c8effcfae&challenge_ids[]=4aa9361e-55f9-4383-aa2b-8ece3ca7a5df&challenge_ids[]=66aea17b-b8a5-4d87-9ff6-1795fd2d209b&challenge_ids[]=ac2b5ba1-0b41-49b0-a738-0d0b7ae94511&challenge_ids[]=942d6e6e-1406-4036-b420-4e38307206d5&challenge_ids[]=d9f235cf-faf5-4a83-a86c-46c0e125b253&challenge_ids[]=5d64db7e-47c4-45cd-beba-6ffb34b9e7d6&challenge_ids[]=dad40e92-500b-4369-ab36-682d769094db&challenge_ids[]=b518e3bb-7598-4cf5-a226-9d7210f02c14&challenge_ids[]=6815220b-9bbd-47ee-a393-6cb068a62bad&challenge_ids[]=19845373-2ded-4405-bb31-7ac3c4f8bc21&challenge_ids[]=43b15547-1f5a-4d91-a0eb-04c0d5ae8867&challenge_ids[]=55002cf6-2424-4a9e-86e0-38c2490fe636&challenge_ids[]=7f427439-58ba-431a-b3cd-bedd4faab0a1&challenge_ids[]=968a9b7f-8c47-4d23-9ec2-f79d5dce74d3&challenge_ids[]=f23a0c62-3354-4c95-8b03-8a7605e1db9c&challenge_ids[]=2dd40992-86fe-4875-bc19-dce324e1546a&challenge_ids[]=e52eb763-0744-46a9-9481-03058a779f67&challenge_ids[]=4ddfa7b8-e522-462e-aead-3fa3e576ee6e&challenge_ids[]=793d6c5d-85f1-46a6-bfbe-250d9a388ced&challenge_ids[]=755df7d1-d9e3-4ae1-98f8-eab710b8f297&challenge_ids[]=58b5fe01-b3aa-4bfb-a34b-df067d162fcd&challenge_ids[]=b5306e68-07e4-4454-9eb1-24d741f9892a&challenge_ids[]=ea72d2a7-b137-4ddb-8429-77514f6db765&challenge_ids[]=03afe78f-a851-46b9-802d-dbaecd35cd5c&challenge_ids[]=4cdd00bc-c110-4651-b9a3-540dac87ddaf&challenge_ids[]=3cb7be77-2741-4adf-a66e-a63f63c6c028&challenge_ids[]=83ed68c1-b9b4-4e06-b206-4f5aee4061a8&challenge_ids[]=1d20db2e-033a-45b3-ab62-29fd6b83e04c&challenge_ids[]=bb4af7a8-cf08-4ab2-82f0-42c689073e36&challenge_ids[]=1e549c6f-71cd-4463-b2f2-9df70bfa0724&challenge_ids[]=5cc4a3e3-5778-4ced-8531-bb949ac34d1c&challenge_ids[]=d183be7f-8b92-45ce-8db3-ee9d7b996b79&challenge_ids[]=1abcaf25-5c55-4b95-940c-102140ea2054&challenge_ids[]=8519896e-1b41-4ee7-880b-5b9c99e930d5&challenge_ids[]=69e577a3-6782-4465-94c7-6cffd22b1cea&challenge_ids[]=f5fd0279-f8c1-407b-ae5e-b1f67d093d15&challenge_ids[]=e32d8b21-89c4-4d20-8919-76c10bf8101f&challenge_ids[]=2d12d171-5eba-46c2-93b9-a69727d7a854&challenge_ids[]=517d11a3-8316-468b-bcb3-72250c07bd60&challenge_ids[]=1d386f61-2ef8-46e9-93e0-6f29cfb88cd6&challenge_ids[]=a103a776-b2fe-4454-9043-91b648cf3e9a&challenge_ids[]=022e7766-0287-4057-9da0-c8edeae7c0ae&challenge_ids[]=169781a1-51c9-4562-a5c7-98ea4e77c42d&challenge_ids[]=fa3dcef6-432b-4833-aed5-feb361c7d5ad&challenge_ids[]=bf28d28e-53f7-4d9d-bd2b-ae3590cff5eb&challenge_ids[]=fc5b5d42-4c39-4575-bc24-74a8729921b6&challenge_ids[]=0e246eef-ebb1-4189-8b6e-409e9f4e6402&challenge_ids[]=08371020-9f6c-43ae-ad51-cc4fcbb529dc&challenge_ids[]=92dd8cbc-af3c-43b2-b1e2-169980aa7339&challenge_ids[]=e11f75ba-c3d6-48e8-b782-c4d3c81a8bd0&challenge_ids[]=759b461e-d356-4347-adda-d86ce56117ca&challenge_ids[]=f70a4ab8-8d99-4870-981d-255e10cd8135&challenge_ids[]=38954a37-3e05-4832-8b35-debb9d180f3b&challenge_ids[]=8f97799d-667a-47f1-a0e0-d3545242bffa&challenge_ids[]=fa0f0584-0368-4ae7-a34f-40f08c3934d6&challenge_ids[]=fedf775d-c4a5-429e-8598-58bdf67a1ead&challenge_ids[]=8f30cde8-1159-4375-a485-3a9180939828&challenge_ids[]=a561a2f2-ea19-4072-8036-2bcad479bd37&challenge_ids[]=b41b7ebd-59df-40e0-bc22-a1d77165f9fd&challenge_ids[]=ab244975-3b43-4a0e-87bd-a4ad33efdde1&challenge_ids[]=48296388-88e2-453b-b8ed-b98c947c16d7&challenge_ids[]=da94405a-5bf1-4196-940a-477758beded8&challenge_ids[]=f7086254-7c43-452d-956b-a9c8ef4861c7&challenge_ids[]=d10f1919-f789-4246-82e3-490714693c8f&challenge_ids[]=3671eb47-d8ec-4a04-9dfe-b27e46d7924c&challenge_ids[]=41187b28-11d2-4e36-ad9f-2a99bce9d43c&challenge_ids[]=45765a48-f3c9-4666-9d07-ce8aa4345a54&challenge_ids[]=d4ad41d7-1dbc-455e-a672-b85b7f483ebb&challenge_ids[]=624a9119-5d6b-4899-b89c-6f88fe7a8601&challenge_ids[]=8661e63a-46ed-420c-a303-bd0e331cc7fa&challenge_ids[]=d29a1dd9-89b6-4660-bd97-485bebb62220&challenge_ids[]=bf9c0010-25dc-42a9-8742-f4337bb0dea7&challenge_ids[]=59c01c22-a0b9-4f08-9ee9-c7ffa0cb9207&challenge_ids[]=59717e2a-e1be-41ea-9a60-2b1cabae70a2&challenge_ids[]=c5f6a573-5e18-49e2-9638-3832304530df&challenge_ids[]=60354f40-56b4-49e5-9b1f-77cc7c82c16a&challenge_ids[]=850cd7f1-ea3b-4602-af89-15e0c1638eba&challenge_ids[]=ec294be4-628d-4e6a-b408-6b69340a26f5&challenge_ids[]=4216a632-1266-4941-8781-446cde7f635f&challenge_ids[]=a2ca6bd5-692f-4303-9594-2819cdc396b8&challenge_ids[]=10967c5c-4afc-481f-ad7c-f79423fc021b&challenge_ids[]=d3f1769d-ac8a-4cc9-bde5-5abbe2598e2a&challenge_ids[]=0f95e44c-06e9-4b94-819e-0f0e90b4f3c6&challenge_ids[]=afef4e24-6bdf-40a6-af5d-88227b646342&challenge_ids[]=1076f825-60c2-407a-b90c-57c32e6e95b2&challenge_ids[]=56c25349-7b86-4212-844f-382b767679a1&challenge_ids[]=83f8d20b-a56f-4ed1-b3aa-b52d4cbcc82e&challenge_ids[]=40627f1c-deac-4b68-ae53-714c7e467899&challenge_ids[]=fbc389f0-3754-4704-ab24-c1ede174a19c&challenge_ids[]=a1f705b2-4a0b-4f93-9d92-14b8053e1fe3&challenge_ids[]=9f07cfd0-fc49-4486-b198-e87fcbff8c83&challenge_ids[]=12d8e82f-71b6-4a44-a08c-4396962c8f62&challenge_ids[]=35131686-9bf5-421b-8a53-50903bdda827&challenge_ids[]=84783332-5ad2-41d1-8f7a-bb98cbead75f&challenge_ids[]=3daf56b2-5e26-479d-83f4-bf47cb450a75&challenge_ids[]=21ff286e-ee09-4e18-914c-09e8f68d7b65&challenge_ids[]=37666906-48b8-472b-8cff-6d90b692bee9&challenge_ids[]=6a42ac8d-6762-4550-94f5-8a75881b0ba6&challenge_ids[]=dd08e07c-c3ff-4e32-917b-343dacce7552&challenge_ids[]=7f068c02-34dd-47d2-b9c2-2c1f58182c1d&challenge_ids[]=a511bb4b-d958-4539-8412-854fdc051133&challenge_ids[]=7ece8b78-f872-4cdf-8cae-4c036bd603f8&challenge_ids[]=7415aed2-c27a-4794-b229-e2639eeca702&challenge_ids[]=8aa90825-a013-4c57-a0f4-d9ce0c2556b8&challenge_ids[]=085e95d1-f27d-4a28-af76-75dcd9f48419&challenge_ids[]=33cbd776-f838-4955-934f-09c25748513a&challenge_ids[]=7e93c6de-0639-49e4-b6bf-3a14feb31b0c&challenge_ids[]=1c7eebe8-d2f9-437d-97f3-6325ff95409c&challenge_ids[]=f8f8ac83-8e67-4b6b-bd1c-78e502a47b09&challenge_ids[]=3a7afe49-a1b7-4873-b49f-38d3d580d22f&challenge_ids[]=0a4953bb-7c2b-49e4-a4fb-fb19e0e69142&challenge_ids[]=76d02b3d-92f9-4a5c-95d7-0a537168cd88&challenge_ids[]=5e4a08f4-37d5-42bc-badb-e40aef84c1a3&challenge_ids[]=6de865ff-8953-4bac-a495-eb04318b8e85&challenge_ids[]=5b9ac345-afce-4790-a666-2984dd865c91&challenge_ids[]=38ab3df2-64d5-43a6-bb8c-cf1dc16551e4&challenge_ids[]=e732b01e-ad61-4745-938a-a0b12e433532&challenge_ids[]=a287b066-3132-434c-8173-6b528b79aee5&challenge_ids[]=0dea78e7-43f2-42bf-b63f-bcf605fc4f0c version=HTTP/2.0 headers={"host": "voting-service-p-tzbnm.voting-service-p-tzbnm.svc.cluster.local:8080", "user-agent": "Go-http-client/1.1", "accept-encoding": "gzip", "l5d-dst-canonical": "voting-service-p-tzbnm.voting-service-p-tzbnm.svc.cluster.local:8080", "l5d-orig-proto": "HTTP/1.1"}
DBUG [    81.015448s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60552 to None; attempting TLS handshake
DBUG [    84.292052s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60580 to None; attempting TLS handshake
DBUG [    91.015260s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60650 to None; attempting TLS handshake
DBUG [    94.290975s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60680 to None; attempting TLS handshake
DBUG [   101.022533s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60756 to None; attempting TLS handshake
DBUG [   104.290440s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60778 to None; attempting TLS handshake
DBUG [   111.015406s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60840 to None; attempting TLS handshake
DBUG [   114.291024s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60862 to None; attempting TLS handshake
DBUG [   121.021278s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60930 to None; attempting TLS handshake
DBUG [   124.291432s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:60964 to None; attempting TLS handshake
DBUG [   131.015255s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:32802 to None; attempting TLS handshake
DBUG [   134.290846s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:32838 to None; attempting TLS handshake
DBUG [   141.015578s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:32910 to None; attempting TLS handshake
DBUG [   144.291986s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:32932 to None; attempting TLS handshake
DBUG [   151.015089s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:32988 to None; attempting TLS handshake
DBUG [   154.290850s] admin={server=admin listen=0.0.0.0:4191} linkerd2_proxy::transport::tls::listen accepted connection from 10.28.5.1:33010 to None; attempting TLS handshake

(If the output is long, please create a gist and paste the link here.)

linkerd check output

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ control plane namespace exists
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ no invalid service profiles

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

Status check results are √

Environment

• Kubernetes Version: 1.12.6-gke.10
• Cluster Environment: (GKE, AKS, kops, …) GKE
• Host OS: alpine
• Linkerd version: 2.2.1 and edge-19.4.4

Possible solution

Additional context

This only seems to happen on long running requests possibly? The output data should only be about 100kb so it isn’t all that large I don’t think? Happy to help however I can!