activerecord-jdbc-adapter: Error: out of shared memory on 'large' rails migration

I get this error when running a migration that does a bit of ETL with ~30k records. I don’t know if this issue belongs here of with the java jdbc.

I had to adjust max_locks_per_transaction to a much larger number than it’s default: 64. When i run this same migration on CRuby with the pg gem I do not have to make this change.

postgres (PostgreSQL) 12.1 (Debian 12.1-1.pgdg100+1) Rails 6.0.1 activerecord-jdbc-adapter 60.0 activerecord-jdbcpostgresql-adapter 60.0 jdbc-postgres 42.2.6

About this issue

Original URL
State: open
Created 5 years ago
Comments: 15 (7 by maintainers)

Most upvoted comments

btw. setting config.active_record.verbose_query_logs = false brings down the runtime to 18s for me. Reason is that ActiveRecord::LogSubscriber logs the caller (and then not show it in the log 🙄). To do this JRuby uses, depending on Java version, java.lang.Thread.getStrackTrace() which is slow. Java >= 9 should be faster there

dr-itz on Nov 30, 2019

Just tested…runtime is around 45-49s either way…but I actually might be able to optimize the way we do inserts with what I learnt today.

dr-itz on Nov 29, 2019

errm…an ancient one. 9.6.something. I increased the number of rows now and it crashes exactly like yours. So that’s “good” news 😉

dr-itz on Nov 29, 2019

well now it has no line endings…

rwilliams on Nov 28, 2019

@dr-itz I updated the gem versions used in the main comment. I’ll try and work on a simplified version of the migration for testing.

rwilliams on Nov 28, 2019