manageiq: Sporadic failures with fix_auth

Awesome find @jrafanie … no ones been able to get a reproducer.

I thought all v2 keys were in memory tests. don’t remember writing anything to disk for these tests

I don’t know if you’ve identified the issue causing this failure and if it’s fixed in https://github.com/ManageIQ/manageiq/pull/21178 but it’s easy enough to recreate in the auth_config_model_spec.rb locally without PARALLEL or anything. Just crank up the count of times it runs the test and it fails most times, even multiple times.

Note, I tried the same thing with the manageiq-password test suite and couldn’t get it to fail no matter how many times I forced each test to run.

It does occur fairly easy with the fix_auth test though so perhaps it’s specific with this test or setup. See below.

diff --git a/spec/tools/fix_auth/auth_config_model_spec.rb b/spec/tools/fix_auth/auth_config_model_spec.rb
index e3b9086476..67ffac3737 100644
--- a/spec/tools/fix_auth/auth_config_model_spec.rb
+++ b/spec/tools/fix_auth/auth_config_model_spec.rb
@@ -48,7 +48,8 @@ RSpec.describe FixAuth::AuthConfigModel do
     end
   end

-  context "#request_tasks" do
+1000.times do |i|
+  context "#request_tasks #{i}" do
     subject { FixAuth::FixMiqRequestTask }
     let(:request) do
       subject.create(
@@ -76,3 +77,4 @@ RSpec.describe FixAuth::AuthConfigModel do
     end
   end
 end
+end

joerafaniello@Joes-MBP-2 manageiq % be rspec spec/tools/fix_auth/auth_config_model_spec.rb
** override_gem("manageiq-gems-pending", :path=>"/Users/joerafaniello/Code/manageiq-gems-pending") at /Users/joerafaniello/Code/manageiq/bundler.d/Gemfile.dev.rb:13
** Using session_store: ActionDispatch::Session::MemoryStore
** ManageIQ master, codename: Morphy

Randomized with seed 48240
.........................................................................................................................................................................................................F...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F...................................................................................................................................................................................................

Failures:

  1) FixAuth::AuthConfigModel#request_tasks 630 upgrades request (find with prefix, do not stringify keys)
     Failure/Error: expect(new_options[:sysprep_domain_password]).to be_encrypted(pass)
       expected: "v2:{xQQXbxWHMAarXYxni/o8fw==}" to be encrypted and decrypt to "password"
     # ./spec/tools/fix_auth/auth_config_model_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /Users/joerafaniello/.gem/ruby/2.6.6/gems/webmock-3.13.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  2) FixAuth::AuthConfigModel#request_tasks 334 upgrades request (find with prefix, do not stringify keys)
     Failure/Error: expect(new_options[:sysprep_domain_password]).to be_encrypted(pass)
       expected: "v2:{DFtIFBQDjOkXeeYfw2Zlhg==}" to be encrypted and decrypt to "password"
     # ./spec/tools/fix_auth/auth_config_model_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /Users/joerafaniello/.gem/ruby/2.6.6/gems/webmock-3.13.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

Finished in 8.17 seconds (files took 15.06 seconds to load)
1001 examples, 2 failures

Failed examples:

rspec './spec/tools/fix_auth/auth_config_model_spec.rb[1:632:1]' # FixAuth::AuthConfigModel#request_tasks 630 upgrades request (find with prefix, do not stringify keys)
rspec './spec/tools/fix_auth/auth_config_model_spec.rb[1:336:1]' # FixAuth::AuthConfigModel#request_tasks 334 upgrades request (find with prefix, do not stringify keys)

Randomized with seed 48240

joerafaniello@Joes-MBP-2 manageiq % be rspec spec/tools/fix_auth/auth_config_model_spec.rb
** override_gem("manageiq-gems-pending", :path=>"/Users/joerafaniello/Code/manageiq-gems-pending") at /Users/joerafaniello/Code/manageiq/bundler.d/Gemfile.dev.rb:13
** Using session_store: ActionDispatch::Session::MemoryStore
** ManageIQ master, codename: Morphy

Randomized with seed 50568
..........................................................................................................................................................................................................................................................F............................................................................................................................................................................................................................F.........................................................................................F..........................................................................................................................................F...................................................................................................F...................................................................................................................................................................................F....................

Failures:

  1) FixAuth::AuthConfigModel#request_tasks 192 upgrades request (find with prefix, do not stringify keys)
     Failure/Error: expect(new_options[:sysprep_domain_password]).to be_encrypted(pass)
       expected: "v2:{fmnC7KQaqxEeuOm7TsjXyw==}" to be encrypted and decrypt to "password"
     # ./spec/tools/fix_auth/auth_config_model_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /Users/joerafaniello/.gem/ruby/2.6.6/gems/webmock-3.13.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  2) FixAuth::AuthConfigModel#request_tasks 735 upgrades request (find with prefix, do not stringify keys)
     Failure/Error: expect(new_options[:sysprep_domain_password]).to be_encrypted(pass)
       expected: "v2:{gyN/UvEAA4zU03WttaF+VA==}" to be encrypted and decrypt to "password"
     # ./spec/tools/fix_auth/auth_config_model_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /Users/joerafaniello/.gem/ruby/2.6.6/gems/webmock-3.13.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  3) FixAuth::AuthConfigModel#request_tasks 522 upgrades request (find with prefix, do not stringify keys)
     Failure/Error: expect(new_options[:sysprep_domain_password]).to be_encrypted(pass)
       expected: "v2:{Q4aOWAlKccxrIU74vjpuHA==}" to be encrypted and decrypt to "password"
     # ./spec/tools/fix_auth/auth_config_model_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /Users/joerafaniello/.gem/ruby/2.6.6/gems/webmock-3.13.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  4) FixAuth::AuthConfigModel#request_tasks 483 upgrades request (find with prefix, do not stringify keys)
     Failure/Error: expect(new_options[:sysprep_domain_password]).to be_encrypted(pass)
       expected: "v2:{UM4wbTxyacWRrldMOTsbQg==}" to be encrypted and decrypt to "password"
     # ./spec/tools/fix_auth/auth_config_model_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /Users/joerafaniello/.gem/ruby/2.6.6/gems/webmock-3.13.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  5) FixAuth::AuthConfigModel#request_tasks 401 upgrades request (find with prefix, do not stringify keys)
     Failure/Error: expect(new_options[:sysprep_domain_password]).to be_encrypted(pass)
       expected: "v2:{djAYxZp9KvdlTF2PU1BneA==}" to be encrypted and decrypt to "password"
     # ./spec/tools/fix_auth/auth_config_model_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /Users/joerafaniello/.gem/ruby/2.6.6/gems/webmock-3.13.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  6) FixAuth::AuthConfigModel#request_tasks 687 upgrades request (find with prefix, do not stringify keys)
     Failure/Error: expect(new_options[:sysprep_domain_password]).to be_encrypted(pass)
       expected: "v2:{fKc5BiuJtenAlPzYiutg8g==}" to be encrypted and decrypt to "password"
     # ./spec/tools/fix_auth/auth_config_model_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /Users/joerafaniello/.gem/ruby/2.6.6/gems/webmock-3.13.0/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

Finished in 6.35 seconds (files took 12.92 seconds to load)
1001 examples, 6 failures

Failed examples:

rspec './spec/tools/fix_auth/auth_config_model_spec.rb[1:194:1]' # FixAuth::AuthConfigModel#request_tasks 192 upgrades request (find with prefix, do not stringify keys)
rspec './spec/tools/fix_auth/auth_config_model_spec.rb[1:737:1]' # FixAuth::AuthConfigModel#request_tasks 735 upgrades request (find with prefix, do not stringify keys)
rspec './spec/tools/fix_auth/auth_config_model_spec.rb[1:524:1]' # FixAuth::AuthConfigModel#request_tasks 522 upgrades request (find with prefix, do not stringify keys)
rspec './spec/tools/fix_auth/auth_config_model_spec.rb[1:485:1]' # FixAuth::AuthConfigModel#request_tasks 483 upgrades request (find with prefix, do not stringify keys)
rspec './spec/tools/fix_auth/auth_config_model_spec.rb[1:403:1]' # FixAuth::AuthConfigModel#request_tasks 401 upgrades request (find with prefix, do not stringify keys)
rspec './spec/tools/fix_auth/auth_config_model_spec.rb[1:689:1]' # FixAuth::AuthConfigModel#request_tasks 687 upgrades request (find with prefix, do not stringify keys)

Randomized with seed 50568