Behat: 3.4.0 breaks the ability to run individual tests

On a side note, I was able to run tests again from PhpStorm after in the Run/Debug configuration I set the Custom working directory option (in the default configuration for Behat) to the directory in which behat.yml lives (it needs to align with whatever in Behat is %paths.base%).

Revert to 3.3 for now, I’ll try to look into the bug ASAP.

We also can’t run a single feature anymore since the update to Behat 3.4. We start a single feature with bin/behat feature/natarch-continuous/ArchiveLandingPage.feature - I seem to have found the culprit within the Behat\Behat\Gherkin\Specification\Locator\FilesystemFeatureLocator::locateSpecifications() - That method instantiates the LazyFeatureIterator with the $locator = ‘features/natarch-continuous/ArchiveLandingPage.feature’ and $filters = array(new PathsFilter($suiteLocators)); If I set the $filters = []; Then the single feature is run. I can also run bin/behat features/natarch-continuous/ and it finds and runs the features inside the directory.

This is our behat.yml configuration.

default:
  suites:
    natarch_features:
      paths: [ features ]
      contexts:
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\DrushContext
        - Drupal\DrupalExtension\Context\MarkupContext
        - rdx\behatvars\BehatVariablesContext
        - FeatureContext
        - OtapDrushContext
        - SeleniumContext
        - CustomSoapContext
        - DebugContext:
           screen_shot_path: project/behat/screenshots
  formatters:
    pretty:
    junit:
      output_path: %paths.base%/build
  extensions:
    Behat\MinkExtension:
      goutte:
        guzzle_parameters:
          verify: false
      javascript_session: selenium2
      selenium2:
        wd_host: http://natarch.dev:8643/wd/hub
#        wd_host: http://<your-local-ip-address>:4444/wd/hub
      base_url: http://natarch.dev
#      default_session: selenium2
#      browser_name: 'firefox'
      files_path: %paths.base%/features/dummy
    Drupal\DrupalExtension:
      subcontexts:
        paths:
          - %paths.base%/features/bootstrap
      blackbox: ~
      api_driver: 'drupal'
      drush_driver: 'drush'
      drush:
        root: '/var/www/natarch'
      drupal:
        drupal_root: '/var/www/natarch'
      text:
        username_field: "Gebruikersnaam"
        password_field: "Wachtwoord"
        log_in: "Inloggen"
        log_out: "Uitloggen"
      region_map:
        breadcrumb: ".region--breadcrumb"
        content: ".region-content, .layout-content"
        footer: ".region--footer"
        header: ".region--header"
        highlighted: ".region--highlighted"
        main_links: ".region--main-links"
        primary_menu: ".region--primary-menu"
        secondary_menu: "#block-natarch-account-menu"
        sidebar_first: ".layout-sidebar-first"
        site_menu_first: ".region--site-menu-first"
        site_menu_second: ".region--site-menu-second"
        title_image: ".region--title-image"
      selectors:
        message_selector: ".messages"
        error_message_selector: ".messages.messages--error"
        success_message_selector: ".messages.messages--status"
    Behat\SoapExtension:
          # An associative array as second argument for \SoapClient::__soapCall().
          options: {}
          # An associative array as parameters for \DOMXpath::registerNamespace().
          namespaces: {}
    rdx\behatvars\BehatVariablesExtension: ~
testserver:
  extensions:
    Behat\MinkExtension:
      selenium2:
        wd_host: http://miauwmiauwmiauw.test.rad.lan:8643/wd/hub
      base_url: https://test.miauwmiauwmiauw.nl
    Drupal\DrupalExtension:
      drush:
        root: '/var/www/miauwmiauwmiauw-web'
      drupal:
        drupal_root: '/var/www/miauwmiauwmiauw-web'
testzap:
  extensions:
    Behat\MinkExtension:
      selenium2:
        #wd_host: http://miauwmiauwmiauw.test.rad.lan:8643/wd/hub
        wd_host: http://miauwmiauwmiauw:8643/wd/hub
        capabilities:
          proxy:
            proxyType: 'manual'
            httpProxy: 'localhost:6667'

Can we remove $filters = array(new PathsFilter($suiteLocators)); from the FilesystemFeatureLocator?

@milkovsky the fix was actually in behat/gherkin 4.6.1 which is what @ciaranmcnulty is trying to say here – i got confused there for a second, i thought there was a “behat/behat” 4.6.1 version 😃. Since it was a patch when you updated behat it probably updated gherkin too.

Thanks @ciaranmcnulty !

Update to behat 3.6.1 fixed the issue for me.

Still happening, here is how I work around it (I want to run Behat in PhpStorm):

default:
    ...

    extensions:
        Behat\Symfony2Extension:
            kernel:
                bootstrap: "tests/tests_bootstrap.php" # <-- add this
                class: App\Kernel

And I create the tests/tests_bootstrap.php file with:

<?php
// Workaround for https://github.com/Behat/Behat/issues/1076
chdir(__DIR__ . '/..');

I have updated the PR with the proposed fix: https://github.com/Behat/Gherkin/pull/129

This should be resolved in 4.6.1

@pfrenssen ignore my previous comment about tests being run per suite. apparently if you do not specify a specific suite, then Behat will loop through all the suites. it will perform any BeforeSuite/AfterSuite hooks even if it has no scenario’s to run in that suite. so that’s what I was seeing being run per suite, not the scenario itself, just the BeforeSuite/AfterSuite cycling. the scenario’s themselves were only executed once.

so your branch works fine for me 😃 it’s been two years since this bug was introduced, it’d be nice to have it fixed and no longer need to set a custom working directory

 "behat/behat": "3.5.0",
 "behat/mink-extension": "2.3.1"

and make sure that you clear Gherkin cache

This is still a problem for me with behat + gherkin. I was able to work around it by adding a unique tag to tests I want to run individually and just reference that with behat using --tags @MEDSTAT-PUBLICATION-BUCKETS. That seems to be an okay temporary solution until this is fixed.

Hi,

Is there any progress made on this? We’re stuk on Behat 3.3 as well because of this.

Using @warmwaterkruik his information I tried to dive deeper into it and it’s a real maze.

What I encountered was in PathsFilter the method isFeatureMatch checks if the feature file is in the correct path.

https://github.com/Behat/Gherkin/blob/e66888c99c97b85113822b9f37276c2f9c414c97/src/Behat/Gherkin/Filter/PathsFilter.php#L53

However $feature->getFile() resolves a relative path and not an absolute path, then if your working directory is not the base directory realpath() won’t resolve the file, the function returns false and the feature isn’t executed. Basically what’s discussed in the already referenced #1085 and possibly related to #1056 .

For linux, it is located in the /tmp dir I believe. Also, you can disable the cache in your behat.yml file, using this:

  gherkin:
    cache: ~

Full answer here.