mongify: Error migrating data from sql server to mongo

Hi, I’m trying to migrate data from sql server to mongodb, and I have this error during data transfer…

the error is:

Copying t_messages (54/59): (20000/20000) 100% |ooooo| Time: 00:00:03 /Users/elena/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/activerecord-sqlserver-adapter-3.2.0/lib/active_record/connection_adapters/sqlserver/database_statements.rb:414:in `each’: TinyTds::Error: Adaptive Server connection timed out: EXEC sp_executesql N’SELECT * FROM (ActiveRecord::StatementInvalid) ( SELECT TOP 1100000 *, ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS rnum FROM t_messages ) t_messages WHERE rnum > 1080000’

the table t_messages has 1.174.000 record…

Can you help me? Thank you and sorry for my poor english… Elena.

About this issue

  • Original URL
  • State: open
  • Created 8 years ago
  • Comments: 17 (6 by maintainers)

Most upvoted comments

May I know if the above issue is resolved? I was trying to pull data from SQL Server to MongoDB and sees the same error.

lrsa@SERVER5:~$ mongify check database.config /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/lib/mongify/configuration.rb:14:in instance_eval': no block given (yield) (LocalJumpError) from /home/lrsa/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb:33:in block in catch’ from /home/lrsa/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb:33:in catch' from /home/lrsa/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb:33:in catch’ from /home/lrsa/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb:108:in timeout' from (eval):8:in block in parse’ from /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/lib/mongify/configuration.rb:34:in instance_exec' from /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/lib/mongify/configuration.rb:34:in sql_connection’ from (eval):1:in parse' from /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/lib/mongify/configuration.rb:14:in instance_eval’ from /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/lib/mongify/configuration.rb:14:in parse' from /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/lib/mongify/cli/options.rb:78:in config_file’ from /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/lib/mongify/cli/options.rb:58:in parse' from /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/lib/mongify/cli/application.rb:27:in execute!’ from /home/lrsa/.rvm/gems/ruby-2.4.0/gems/mongify-1.3.1/bin/mongify:15:in <top (required)>' from /home/lrsa/.rvm/gems/ruby-2.4.0/bin/mongify:22:in load’ from /home/lrsa/.rvm/gems/ruby-2.4.0/bin/mongify:22:in <main>' from /home/lrsa/.rvm/gems/ruby-2.4.0/bin/ruby_executable_hooks:15:in eval’ from /home/lrsa/.rvm/gems/ruby-2.4.0/bin/ruby_executable_hooks:15:in `<main>’

I’ve just experienced this. I have to first admit general ignorance in regard to ruby. I know enough to be dangerous. Please forgive me if I get all the nomenclature, syntax, and everything else ruby specific, completely wrong.

Some part of the code that deals with configuration somehow executes the value “timeout” in the config file as a method instead of just passing it on to be used in the ActiveRecord connection. Ruby happens to have one such method, and I suppose it is somehow available to run in this context, so it does, but the call is incorrect, and shouldn’t have happened anyhow.

I fixed this locally by using a different name entirely, “dbtimeout”, and then in BaseConnection->to_hash I added a gsub to change dbtimeout into timeout. I am acutely aware that this is the wrong way to do this.

There is a larger problem here. ActiveRecord plugins don’t all use 100% standard variable names. One may use timeout, another may be query_timeout. The way the configuration handling for mongify is written doesn’t account for that.

Hi, yes, I tried and got errors

if I write “timeout” in database.config

sql_connection do
  adapter   "sqlserver"
  host      "<my host IP>"
  port      3408
  username  "<my un>"
  password  "<my pw>"
  database  "ASM_DB"
  timeout 10000
end

mongodb_connection do
  host      "localhost"
  database  "ASM_BIG_MODB"
end

and run: mongify process database.config translation.rb I get this error:

bash-3.2$ mongify process database.config translation.rb
(eval):8:in `block in parse': Object#timeout is deprecated, use Timeout.timeout instead.
/Users/elena/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/mongify-1.2.4/lib/mongify/configuration.rb:14:in `instance_eval': no block given (yield) (LocalJumpError)...

if I write Timeout.timeout, as suggested by the above error

sql_connection do
  adapter   "sqlserver"
  host      "<my host IP>"
  port      3408
  username  "<my un>"
  password  "<my pw>"
  database  "ASM_DB"
  Timeout.timeout 10000
end

mongodb_connection do
  host      "localhost"
  database  "ASM_BIG_MODB"
end

and run: mongify process database.config translation.rb I get this error:

bash-3.2$  mongify process database.config translation.rb
/Users/elena/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/mongify-1.2.4/lib/mongify/configuration.rb:14:in `instance_eval': no block given (yield) (LocalJumpError)
    from /Users/elena/.rbenv/versions/2.3.0/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'...

What can I try? Any other ideas? Thank you, Elena.