great_expectations: [BUG] Expectations Raise "The column "xxx" in BatchData does not exist" Error

Describe the bug Hi team, we are working on integrating GX with snowflake datasource into our data validation system via GreatExpectations Version: 0.15.34.

We are planning to batch-run 3 expectations validation against a snowflake table named test_table. However, we got Error The column "my_column" in BatchData does not exist for expectations expect_column_values_to_not_be_null and expect_column_values_to_be_between. However, expectation expect_column_distinct_values_to_be_in_set worked as expected.

Detailed Error Msg:

great_expectations.exceptions.exceptions.MetricResolutionError: Error: The column "my_column" in BatchData does not exist.

Does anyone know what the issue could be? Or is there any configuration issue in my expectation_suite, data_context_config, or checkpoint_config? Any help would be greatly appreciated ~~

To Reproduce Steps to reproduce the behavior:

  1. expectation_suite:
{
  "data_asset_type": null,
  "expectation_suite_name": "test_table.demo",
  "expectations": [
    {
      "expectation_context": {
        "description": null
      },
      "expectation_type": "expect_column_values_to_not_be_null",
      "ge_cloud_id": null,
      "kwargs": {
        "column": "my_column"
      },
      "meta": {}
    },
    {
      "expectation_context": {
        "description": null
      },
      "expectation_type": "expect_column_distinct_values_to_be_in_set",
      "ge_cloud_id": null,
      "kwargs": {
        "column": "my_column",
        "value_set": [
          1,
          2,
          4
        ]
      },
      "meta": {}
    },
    {
      "expectation_context": {
        "description": null
      },
      "expectation_type": "expect_column_values_to_be_between",
      "ge_cloud_id": null,
      "kwargs": {
        "column": "my_column",
        "max_value": 6,
        "min_value": 1
      },
      "meta": {}
    }
  ],
  "ge_cloud_id": null,
  "meta": {
    "great_expectations_version": "0.15.34"
  }
}

Expected behavior Expectations expect_column_values_to_not_be_null and expect_column_values_to_be_between should be able to find the column in batch data

Environment (please complete the following information):

  • Operating System: MacOS
  • Great Expectations Version: 0.15.34
  • Python: 3.7

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (3 by maintainers)

Most upvoted comments

I’m having the same issue. Happy to provide another set of test data if needed. Interestingly, it only happens with some data files and not with others, so it could be non-deterministic or some kind of race condition

Closing as resolved by #6951 🚀

I’m having the same issue, a workaround that worked for me was to set the column name in lowercase, even if it was originally named in uppercase