google-cloud-ruby: Memory leak with datastore using grpc version 1.2.2
During a recent bundle update of our app the version of grpc changed from 1.2.0 to 1.2.2. With the new version of grpc I am seeing runaway memory use. Here is a screenshot from a test I ran on Heroku. The initial flat spot is the Rails app running happily after deploy and the start of the large ramp was caused by a single datastore query. I then left the app alone and didn’t make any more requests. Memory use continued to climb until 8:10am when Heroku killed the dyno. The app restarted and once again ran happily until I made another datastore request at 11:30 am and the ramp started over. The deploy of v 223 was with a single change of gem 'grpc', '1.2.0' to the Gemfile and the app has been running happily since.
I can also reproduce this locally using the derailed gem. Has anything changed with version 1.2.2 of the grpc gem that needs a change to the datastore code? Or should this issue be moved to the grpc repo?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 21 (13 by maintainers)
In the meantime, going to release a 1.2.5 gem that reverts the connectivity fix (in https://github.com/grpc/grpc/pull/9986), which is causing this issue.
24 hours later and memory is still holding steady. Nice work @apolcyn! Are we ok to close this issue?
@apolcyn I deployed
grpcversion 1.3.4 to our staging environment today and so far everything looks fine.I have created a simplified Rails 5 app that uses the
derailedgem to show the memory usage. It uses an actual cloud datastore instance (so needs a project ID), authenticated locally through gcloud auth login. Clone the grpc-1.2.2-memory branch from here. Note that when running derailed to profile the memory usage all tests are run without a webserver, as it uses Rack directly. The Rails app is set to use grpc 1.2.0 initially.cd test/support/datastore_example_rails_appbundleCheck if the app is connecting to datastore by running:
RAILS_ENV=production GCLOUD_PROJECT=project-id-goes-here rails serverNavigate to localhost:3000 with a browser and create a few users. Execute the following command to run the derailed benchmark (substitute your datastore project ID).
DERAILED_SKIP_ACTIVE_RECORD=true PATH_TO_HIT=/users GCLOUD_PROJECT=project-id-goes-here TEST_COUNT=5000 bundle exec derailed exec perf:mem_over_timeThe profile will take about 15 minutes. Then change the version of grpc to 1.2.2 in the Gemfile.
bundle updateRun the derailed command again to compare.