aws-sdk-ruby: RuntimeError: can't add a new key into hash during iteration

On our production application we are receiving intermittent stack traces on S3 requests- the errors arrive in bursts of a couple hundred over an hour or two, then nothing for a few days, then another burst (6 over the last 30 days). I suspect this might be some sort of concurrency issue related to network-error retries somewhere deep in the seahorse stack, but could use some help figuring out what exactly is going on and how to fix.

Using aws-sdk-core 2.6.1.

Relevant part of the stack trace (net_http):

Full aws-sdk stack trace with all handlers:

[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:40 :in `block in initialize`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:89 :in `yield`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:89 :in `block in session_for`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:87 :in `synchronize`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:87 :in `session_for`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/handler.rb:116 :in `session`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/handler.rb:68 :in `transmit`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/handler.rb:42 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/plugins/content_length.rb:12 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_request_signer.rb:88 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_request_signer.rb:23 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/xml/error_handler.rb:8 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_request_signer.rb:65 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_redirects.rb:15 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/retry_errors.rb:87 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_dualstack.rb:31 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_accelerate.rb:42 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_md5s.rb:31 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/rest/handler.rb:7 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/user_agent.rb:12 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/plugins/endpoint.rb:41 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_url_encoded_keys.rb:39 :in `manage_keys`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_url_encoded_keys.rb:31 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/param_validator.rb:21 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/plugins/raise_response_errors.rb:14 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_dualstack.rb:23 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_accelerate.rb:33 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/param_converter.rb:20 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/response_paging.rb:26 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/plugins/response_target.rb:21 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/request.rb:70 :in `send_request`
[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/base.rb:207 :in `block (2 levels) in define_operation_methods`
[...Application code stacktrace...]

Reproducing the issue with debug logging would be challenging (since the issue only occurs intermittently in production), so I’m hoping to track this down by inspecting the codebase directly.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 18 (10 by maintainers)

Links to this issue

ruby on rails - RuntimeError: can't add a new key into hash during iteration in Rack - Stack Overflow

Commits related to this issue

Minor changes in connection pool. See #1331 — committed to aws/aws-sdk-ruby by trevorrowe 8 years ago

Most upvoted comments

Hi, I just started seeing this issue with version aws-sdk-core (3.180.3) while ramping up concurrency with puma. This is the stack trace that we got:

at block in session_for(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/net_http/connection_pool.rb:111)
at synchronize(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/net_http/connection_pool.rb:110)
at session_for(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/net_http/connection_pool.rb:110)
at session(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/net_http/handler.rb:128)
at transmit(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/net_http/handler.rb:76)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/net_http/handler.rb:50)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/plugins/content_length.rb:24)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/plugins/request_callback.rb:87)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-dynamodb-1.93.1/lib/aws-sdk-dynamodb/plugins/crc32_validation.rb:32)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/json/error_handler.rb:10)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/sign.rb:49)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/transfer_encoding.rb:26)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/retry_errors.rb:360)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/user_agent.rb:37)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/http_checksum.rb:19)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/request_compression.rb:94)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/json/handler.rb:13)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/recursion_detection.rb:18)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-dynamodb-1.93.1/lib/aws-sdk-dynamodb/plugins/endpoints.rb:41)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/plugins/endpoint.rb:47)
at block in call(/usr/local/bundle/ruby/3.0.0/gems/ddtrace-0.54.2/lib/ddtrace/contrib/aws/instrumentation.rb:21)
at trace(/usr/local/bundle/ruby/3.0.0/gems/ddtrace-0.54.2/lib/ddtrace/tracer.rb:283)
at call(/usr/local/bundle/ruby/3.0.0/gems/ddtrace-0.54.2/lib/ddtrace/contrib/aws/instrumentation.rb:20)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/param_validator.rb:26)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/logging.rb:41)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/plugins/raise_response_errors.rb:16)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-dynamodb-1.93.1/lib/aws-sdk-dynamodb/plugins/simple_attributes.rb:119)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/param_converter.rb:26)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/plugins/request_callback.rb:71)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/aws-sdk-core/plugins/response_paging.rb:12)
at call(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/plugins/response_target.rb:24)
at send_request(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-core-3.180.3/lib/seahorse/client/request.rb:72)
at describe_table(/usr/local/bundle/ruby/3.0.0/gems/aws-sdk-dynamodb-1.93.1/lib/aws-sdk-dynamodb/client.rb:3008)

Are you still seeing this issue? Are there any ideas on how to fix it?

rottgoth on Sep 28, 2023