rails: When I run ENV=development rake db:create I don't expect it to try and create the test db too

You are assuming that this behavior is a bug. It is not, it is the behavior that we (the Rails core) wants to the framework. I’m sorry that the behavior don’t work for you but it is not going to change.

Thank you for the issue

@rafaelfranca Fair enough. It’s odd, unexpected, behaviour though, and it’s tripping people up.

Yes. The reason is rails new foo; rails g scaffold user name; rails db:create db:migrate; rails test have to pass without having to rails db:test:create.

Rails started as a great framework in 2005. And now we have this.

On Rails 6 it still exists. I’ve spent an hour trying to fix that for docker container unless found this ticket. Can this be fixed?

I do not understand why this is considered not an issue. It obviously is, it is super surprising that tasks ran in development environment try to touch the test database at all.

Just hit a wall when trying to get a correct docker-compose with MySQL and a non-root user, where only one MYSQL_DATABASE can be specified at a time.

Thanks for fighting the good fight @ioquatix .

well, I guess I’m not the only one facing this ‘issue’.

@edwardmp IMHO, the existing infrastructure was broken the moment it decided to run tests without setting up a database first, and then Rails some how worked around that (or was buggy from the start) and now we have this huge mess.

In these situations, my response is: fix the bug/issue correctly (whatever that may be), and then fix the software which was depending on that behaviour. You can implement a more elaborate solution, e.g. issuing deprecation notices.

Making layers on top of layers like this is simply not maintainable. If ENV=development, trying to make the test database is rediculous - I think we can all agree on that. So, the solution is clear, that function should be fixed.

Yes, this breaks backwards compatibility with existing tools. That’s the nature of depending on buggy, unspecified, implicit, behaviour.

Wow, actually I’m able to prevent rake from trying to find some_test by removing config/database.yml completely

In this case, all rake db:* commands are working only with database specified in DATABASE_URL

@rafaelfranca @schneems any chance of revisiting this for Rails 6? If the tests create their own database and we add support for Minitest and RSpec by default, is there anything else holding this back?

@rafaelfranca

this behavior is causing a log of troubles for those who using DATABASE_URL without config/database.yml

Example:

I’m using docker-compose with DATABASE_URL: postgres://some_u:some_p@postgres:5432/some_development and when running

recreate_db_dev:
	docker-compose \
		-f docker-compose.dev.yml \
		run --rm be bash -c '\
		bin/rails db:environment:set RAILS_ENV=development && \
		(rake db:drop RAILS_ENV=development || true) && \
		rake db:create && \
		rake db:migrate && \
		rake db:seed && \
		rake db:schema:dump'

I’m getting

Dropped database 'some_development'
could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Couldn't drop database 'some_test'
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
/usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in `initialize'
/usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in `new'
/usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in `connect'

it is trying to drop some_test but doesn’t have password and username for some_test, only for some_development

Just ran into this today and was definitely flummoxed. Your desired behavior is unexpected to your users.

Bitten by this today. Super disappointing that I can’t resolve this without a monkey patch gem.

But, I’m open to an environment variable to disable that if that is what people want.

@edwardmp Thanks for your continued input and it is a valuable discussion to have.

I’m glad we can agree on this behaviour being troublesome. Whatever way you slice this cake, it’s got problems. We couldn’t deploy our database on development postgres because it insisted on creating a “test” database.

But realistically I don’t see this getting accepted as it would probably break a lot of existing apps.

Making a new release of ActiveRecord doesn’t retroactively break existing apps. Semantic versioning, a decent changeling, etc, all help solve problems like this.

No. This is by design.

this should be configurable.

It’s pretty strange to me there is no way to override this behavior.

I understand the Rails Core team expects this behavior, and wants new users to hit the ground running with simple commands, etc. That’s fine.

But we’re using AWS’ ParameterStore to manage our credentials, so every time we make an unnecessary, and unexpected query, it costs more money and creates more issues for us.

Even if we accept that this behavior is “expected,” despite it being surprising, I still believe there should be a way to override this behavior with a switch or something.

These are far too much pain for a problem that shouldn’t exist in the first place, sorry to say.

So, here is the odd behyavior from db:drop

export RACK_ENV=development
rake db:drop
{"adapter"=>"mysql2", "database"=>"vmail_development", "strict"=>true}
Dropped database 'vmail_development'
Dropped database 'vmail_test'

I don’t think that’s good behaviour.

Would it be possible to get the best of both worlds?

Keep rails new foo; rails g scaffold user name; rails db:create db:migrate; rails test passing by default without having to rails db:test:create, but also respect RAILS_ENV if passed explicitly to rails db:create.

It might require some internal refactoring, but from an end user point of view, it should be possible, right?

Maybe when DATABASE_ENV or RAILS_ENV is specified explicitly, it can be respected (a flag as it were). The default with no ENV specified, which implies development, can also include test.

How about that?

Just a --skip-test-database flag or a TEST_DATABASE_URL environment variable would be nice features that could be added even without breaking stable rails 5 releases

@MmKolodziej I appreciate your sentiment but making this into a fight puts me (and us) at odds with the entire Rails development team who support this design decision. I’d like to think we can solve this problem more constructively. Unfortunately this issue has been closed, and the Rails core team have made a decision, so we have to respect that. Even if we disagree with it.

In that sense, a while ago, I did actually implement https://github.com/ioquatix/activerecord-configurations, which along with https://github.com/ioquatix/activerecord-migrations solves a lot of my own pain points with ActiveRecord. If you have the same pain points, feel free to invest that energy into using and maintaining the above gems. All the pain points outlined in this issue (and more) are resolved by those gems.

@JCSHoosier take a look at https://github.com/ioquatix/activerecord-migrations it fixes those issues by monkey patch.

Looks like a violation of the least surprise principle to me as well. Experienced that issue many times

Here is some example output:

% RACK_ENV=development rake db:create                                                                       
{"adapter"=>"mysql2", "database"=>"vmail_development", "strict"=>true}
Database 'vmail_development' already exists
Created database 'vmail_test'

An example of where this is a bit odd, is when test is an sqlite3 in-memory database. Therefore, get this output every time.

If you landed here like me

https://github.com/rails/rails/pull/39027 has seem to fixed this with

SKIP_TEST_DATABASE=true or ENV["SKIP_TEST_DATABASE"]=true

Is this fixed in any newer versions? On Rails 5.1 it’s still annoying like hell.

Similar to @MmKolodziej, this is also causing issues for me and my docker-compose script.

I for one would appreciate a fix. I realise that’s not going to happen in 5.x, but it would be great to see in 6.x

I just published a post detailing this issue and showcasing the gem dotenv_rails_db_tasks_fix where I attempt to solve the issues this behaviour causes with dotenv.

It’s not a bug it’s a feature 😃

How about optionally disable creating the test base, e.g. by setting an environment var (e.g. RAILS_SKIP_TEST_DATABASE_CREATION. This would keep backward compatibility but allow skipping the test database creation without using monkey patches or dirty hacks.

For instance:

def each_current_configuration(environment)
   environments = [environment]
   environments << 'test' if environment == 'development' && !ENV.fetch('RAILS_SKIP_TEST_DATABASE_CREATION').present?

Why wouldn’t rails test just create the test database if required? Wouldn’t that make more sense?

Honestly, the db:* tasks seem like a bit of a mess. For all the above reasons… and the way they tie into Rails by default, which breaks unless you override a magical set of variables on “module DatabaseTasks” which in the documentation is described as a class.

What’s even more… inconsistent… is the fact that once you actually connect, all the keys are symbols:

> Account.connection_config
=> {:adapter=>"mysql2", :database=>"vmail_development", :mail_root=>"/tmp/mail_root"}

In the same vein, I’d like to point out the following inconsistencies:

task :environment do
    # This must be a symbol... or establish_connection thinks it's a URL.
    DATABASE_ENV = :development

    ActiveRecord::Base.configurations = {
        # This key must be a string or it will not be matched by ActiveRecord:
        'development' => {
            'adapter' => 'sqlite3',
            # This key must be a string or rake tasks will fail (e.g. each_current_configuration fails):
            'database' => 'db/development.db'
        }
    }

    # Connect to database:
    unless ActiveRecord::Base.connected?
        ActiveRecord::Base.establish_connection(DATABASE_ENV)
    end
end

I’d be happy to take a look at these issues, but I’d need to know if the Rails core team consider these to be problems or not.

I don’t want this to be configurable. I’m not against people using this, but I also don’t want this becoming mainstream feature.

There are clearly some use cases like creating test database in paid machine where creating those databases are expensive but this is not something everyone needs.

For those applications an environment variable is enough.

I came across this yesterday too.

@rafaelfranca I tried the environment variable approach you described: https://github.com/rails/rails/pull/39027

I got bitten by this today as well… Wasted about 4 hours looking into this particular issue until I found this thread.

If you add up all the hours everyone has invested in this not so obvious behavior you might be impressed by the amount of money that is being wasted because of an ego.

Quick advise: Listen to your users.

If we have different database connections for test vs development, it appears this new scheme doesn’t work at all. Whatever login we use for test (using a ramdisk for speed) doesn’t work for development (using persistent disk for large amounts of test data). And there’s no easy workaround that I can see, apart from patching the source.

The workaround is removing the test config from config/database.yml, correct? If it’s the one, I don’t call that a workaround. Or I miss the one you’re talking about.

@fredericgermain Rails is still great, this is just a minor, if surprising, issue that can be mostly worked around as suggested above. It would be nice if this was less surprising and if there was a first class solution though, but no need to be dramatic.

@edwardmp That’s an interesting idea.

The real issue, here, IMHO, is broken behaviour to support existing testing infrastructure. Your suggestion just adds another layer to the onion.

This issue still can be occurred in rails 5.1.0.rc1

The ENV=development rails db:create execution produces the following result: Created database 'db/development.sqlite3' Created database 'db/test.sqlite3'

But it is more concerning that dropping the test database drops the development one as well. ENV=test rails db:drop Dropped database 'db/development.sqlite3' Dropped database 'db/test.sqlite3'

Rails version: 5.1.0.rc1 Ruby version: 2.4.0

So, I’ve been working on a gem which fixes all these issues (and more). https://github.com/ioquatix/activerecord-migrations

I’m not really sure if this is a solution, it’s more of a monkey patch. But it solves a lot of pain points for us w.r.t. deployment with active record, which to be honest leaves a lot to be desired.

I came across this yesterday too.

@rafaelfranca I tried the environment variable approach you described: #39027

This will be available in the next rails release?

@JCSHoosier Actually this behaviour is well documented.

bin/rake -D db:create
rake db:create
    Creates the database from DATABASE_URL or config/database.yml for the current RAILS_ENV (use db:create:all to create all databases in the config). Without RAILS_ENV or when RAILS_ENV is development, it defaults to creating the development and test databases.

Anyway creating only specified database should be somehow supported. Adding ENV variables is not systematic solution. If we should to make this configurable, I think it should be added as config option for active record and you can tweak this config using ENV variable on your own the same we do for example for static files:https://github.com/rails/rails/blob/7ee8d7ea9c371a420cb180cef5061049e379ae62/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt#L25

I think that would definitely help. When I ran into this years ago it was such a pain due to how we had everything configured. @tinco 's suggestion above would help. This also should be documented as an expected behavior somewhere (with the flag to disable).

@rafaelfranca Do you think there is any way we can revisit the decision here?

@fredericgermain - the following are the suggested workarounds so far:

• https://github.com/rails/rails/issues/27299#issuecomment-266386876
• https://github.com/rails/rails/issues/27299#issuecomment-354320563
• https://github.com/rails/rails/issues/27299#issuecomment-394356237

Just to clarify another inconsistency:

ActiveRecord::Base.configurations = {
	'production' => Database::APP_PRODUCTION,
	'development' => Database::APP_DEVELOPMENT,
	'test' => Database::TEST,
}

DATABASE_ENV = :production

# Access key.. via string:
ActiveRecord::Base.configurations[DATABASE_ENV.to_s]

# Actually connect.. need to use symbol:
ActiveRecord::Base.establish_connection[DATABASE_ENV]

If you try to use symbols in configurations, it breaks in other ways. If you try to use a string in establish_connection it thinks it’s a URL.

I’ve been hacking up a solution. Something like this:

				def each_current_configuration(environment)
					unless configuration = ActiveRecord::Base.configurations[environment]
						raise ArgumentError.new("Cannot find configuration for environment #{environment}")
					end
					
					if configuration['database'].blank?
						raise ArgumentError.new("Configuration for environment #{environment} doesn't specify database")
					end
					
					yield configuration
				end