altair: Faceting bug for categorical columns

What happened?

Faceting by pl.Categorical columns results in wrong facets

alt.Chart(
    pl.from_pandas(vega_datasets.data.cars()).with_columns(
        pl.col("Origin").cast(pl.Categorical),
        pl.col("Cylinders").cast(pl.String).cast(pl.Categorical),
    )
).mark_point().properties(width=150, height=150).encode(
    x="Horsepower",
    y="Miles_per_Gallon",
    shape="Cylinders",
    color=alt.Color("Origin").scale(scheme="category10"),
).facet(row="Origin", column="Cylinders")

facets-not-ok

I am not exactly sure what is going wrong, but suddenly all American cars are in the Europe facet, some European cars are in the Japan facet, the Japanese cars are in the correct facet, the 4-Cylinder cars are in the 5 and 6-Cylinder facets, etc. (There is probably some obvious pattern here which I am missing)

I checked the Vega-Lite output and I think the issue is the sort parameter of the resulting spec file.

What would you like to happen instead?

The same code with pl.String columns works as expected:

alt.Chart(
    pl.from_pandas(vega_datasets.data.cars()).with_columns(
        pl.col("Origin"), 
        pl.col("Cylinders").cast(pl.String)
    )
).mark_point().properties(width=150, height=150).encode(
    x="Horsepower",
    y="Miles_per_Gallon",
    shape="Cylinders",
    color=alt.Color("Origin").scale(scheme="category10"),
).facet(row="Origin", column="Cylinders")

facets-ok

Which version of Altair are you using?

5.4.1

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 29 (23 by maintainers)

Most upvoted comments

Yeah, The referenced VL issues are relevant here.

But just to be complete, what is happening. Altair tries to sort the fields in your column ascending when defined as type str on an encoding channel.

So when having this data:

import polars as pl
import altair as alt

df = pl.DataFrame({"value": [2, 5, 3], "month": ["jan", "feb", "mar"]})

And visualising it with the month on the x-axis channel and the values on the color channel using a rect-mark

chart = alt.Chart(df).mark_rect().encode(
    x='month',
    color='value'
)
chart
image

It can be seen that the x-axis is ordered by feb, jan, mar, since the f comes for j in the alphabet.

So by casting the month column in the dataframe as being a categorical in order of appearance (default of polars). We get the following:

df_catg = df.with_columns(pl.col("month").cast(pl.Categorical))
chart_catg = alt.Chart(df_catg).mark_rect().encode(
    x='month',
    color='value'
)
chart_catg
image

The x-axis is now ordered by jan, feb, mar, like the order as is defined in the dataframe.

By comparing the Vega-Lite specification of both charts we notice that the categorical column is serialised differently.

Top chart, column is of type str and it becomes:

chart.to_dict()['encoding']['x']
{'field': 'month', 'type': 'nominal'}

With categorical column defined, it becomes:

chart_catg.to_dict()['encoding']['x']
{'field': 'month', 'sort': ['jan', 'feb', 'mar'], 'type': 'ordinal'}

The sort order is serlialized from the categorical definition of the month column in the DataFrame. All good so far!


Sidenote Observe that the type is also different, ordinal for the dataframe with month column defined as categorical and nominal for the month column just defined as str. The effect of this is that when you use the categorical month column for the color encoding channel it is treated as an ordered categorical and therefor adding a sequential color scheme, versus the default which provides distinct color values for str values:

alt.vconcat(
    chart.encode(color="month"), 
    chart_catg.encode(color="month")
).resolve_scale(
    color="independent"
)
image

But upon adding encoding channels such as row and column this logic for sorting categorical columns in the DataFrame is breaking the rendering when there are combinations that contains no data.

The following goes well, but the order of the column may be seen as not right.

import polars as pl
import altair as alt

df = pl.DataFrame(
    {
        "time": [0, 1, 0, 1, 0, 1, 0, 1],
        "value": [0, 5, 0, 5, 0, 5, 0, 5],
        "choice": ["A", "A", "B", "B", "A", "A", "B", "B"],
        "month": ["jan", "jan", "feb", "feb", "feb", "feb", "mar", "mar"],
    }
)

chart = alt.Chart(df, height=100, width=100).mark_line().encode(
    x='time',
    y='value',
    color='choice',
    row='choice',
    column='month'
)
chart
image

So upon defining the column month as categorical, the order of the months in the column encoding is correctly sorted, but the data within the plots are incorrect.

df_catg = df.with_columns(pl.col("month").cast(pl.Categorical))
chart_catg = alt.Chart(df_catg, height=100, width=100).mark_line().encode(
    x='time',
    y='value',
    color='choice',
    row='choice',
    column='month'
)
chart_catg
image

Leading indeed to data being drawn within the wrong subplot! So, indeed, be very careful here!

One can use the following workaround when having a polars DataFrame as in OP:

df_complete = (
    df.select(pl.col(["choice", "month"]).unique().implode())
    .explode("choice")
    .explode("month")
    .join(df, how="left", on=["choice", "month"])
)

df_complete_sorted = df_complete.sort(pl.col("month").cast(pl.Enum(["jan", "feb", "mar"])))
df_complete_catg = df_complete_sorted.with_columns(pl.col("month").cast(pl.Categorical))
df_complete_catg
image
chart_complete_catg = alt.Chart(df_complete_catg, height=100, width=100).mark_line().encode(
    x='time',
    y='value',
    color='choice',
    row='choice',
    column='month'
)
chart_complete_catg
image

This workaround basically makes sure that all combinations that are possible to make with the row/column channel encoding, are actually existing in the DataFrame, albeit filled with a null value.

Pff, complicated. Altair assumes that the returned categories are in sorted order when it is defined as ordered, but this is an assumption that does not always hold.

  • Custom order with Enums is going OK in Altair. The column is seen as an ordered categorical. get_categories() returns the custom order as is defined. Result is that the custom order list from get_categories() is used to sort the y-encoding channel in the following chart.
my_order = ["k", "z", "b", "a"]

df = pl.from_dict({"cats": ['z', 'z', 'k', 'a', 'b'], "vals": [3, 1, 2, 2, 3]})
df = df.with_columns(pl.col("cats").cast(pl.Enum(my_order)))

nw_s = nw.from_native(df.get_column("cats"), allow_series=True)
print('ordered according to narwhals:', nw.is_ordered_categorical(nw_s))
print('sort order of get categories:', df.get_column("cats").cat.get_categories().to_list())

chart = alt.Chart(df, title='pl.Enum(my_order)').mark_bar().encode(
    x='vals',
    y='cats'
)
print('y-encoding sort definition:', chart.to_dict()['encoding']['y'])
chart
image
  • Physical categorical is going OK in Altair. The column is seen as an ordered categorical. get_categories() returns physical categorical sorted in physical order. Result is that the physical ordered list from get_categories() is used to sort the y-encoding channel in the following chart.
df = pl.from_dict({"cats": ['12', '4', '2'], "vals": [3, 1, 2]})
df = df.with_columns(pl.col("cats").cast(pl.Categorical()))  # 'physical'

nw_s = nw.from_native(df.get_column("cats"), allow_series=True)
print('ordered according to narwhals:', nw.is_ordered_categorical(nw_s))
print('sort order of get categories:', df.get_column("cats").cat.get_categories().to_list())

chart = alt.Chart(df, title='pl.Categorical()').mark_bar().encode(
    x='vals',
    y='cats'
)
print('y-encoding sort definition:', chart.to_dict()['encoding']['y'])
chart
image
  • Lexical categorical is ignored in Altair. The column is not seen as an ordered categorical. get_categories() does not return lexical categorical sorted in lexical order. Result is that the list from get_categories() is not used for the y-encoding channel. Since there is no sort defined it applies ascending sorting within Vega, making it look like that the lexical categorical has impact.
df = pl.from_dict({"cats": ['12', '4', '2'], "vals": [3, 1, 2]})
df = df.with_columns(pl.col("cats").cast(pl.Categorical('lexical')))

nw_s = nw.from_native(df.get_column("cats"), allow_series=True)
print('ordered according to narwhals:', nw.is_ordered_categorical(nw_s))
print('sort order of get categories:', df.get_column("cats").cat.get_categories().to_list())

chart = alt.Chart(df, title="pl.Categorical('lexical')").mark_bar().encode(
    x='vals',
    y='cats'
)
print('y-encoding sort definition:', chart.to_dict()['encoding']['y'])
chart
image

To support lexical categorical, it should

  1. Be considered as ordered by narwhals.
  2. The sort order of the get_categories() should be reflecting the lexical order.

Current implemention of nw.is_ordered_categorical only allows order to be defined based on numeric values and not on alphabet (lexical).

Apparently the situation is different for pandas ordered categorical. Since it does not always return the sorted physical ordered categorical.


Btw. When trying OP as you did in https://github.com/vega/altair/issues/3588#issuecomment-2347221524. I get this:

import polars as pl
import vega_datasets
import altair as alt
alt.Chart(
    pl.from_pandas(vega_datasets.data.cars()).with_columns(
        pl.col("Origin").cast(pl.Categorical),
        pl.col("Cylinders").cast(pl.String).cast(pl.Categorical),
    )
).mark_point().properties(width=150, height=150).encode(
    x="Horsepower",
    y="Miles_per_Gallon",
    shape="Cylinders",
    color=alt.Color("Origin").scale(scheme="category10"),
).facet(row="Origin", column="Cylinders").to_dict()['facet']
{'column': {'field': 'Cylinders',
  'sort': ['8', '4', '6', '3', '5'],
  'type': 'ordinal'},
 'row': {'field': 'Origin',
  'sort': ['USA', 'Europe', 'Japan'],
  'type': 'ordinal'}}

With ('5.5.0dev', '1.6.2') for alt.__version__, nw.__version__. Reflecting the behaviour you have when using Altair version 5.3.0…

Thanks for the ping!

Misunderstood that nw.Enum != pl.Enum

I think they should be the same? As in, pl.Enum should be recognised as nw.Enum:

In [21]: nw.from_native(pl.Series(['a', 'b', 'c'], dtype=pl.Enum(['b', 'a', 'c', 'd'])), allow_series=True).dtype == nw.Enum
Out[21]: True

Regarding physical vs lexical, I don’t think that get_categories reflects the order - but maybe it should? The difference can be seen if you compare the categories, e.g. in a sort:

In [22]: pl.Series(['b', 'a', 'c'], dtype=pl.Categorical('lexical')).sort()
Out[22]:
shape: (3,)
Series: '' [cat]
[
        "a"
        "b"
        "c"
]

In [23]: pl.Series(['b', 'a', 'c'], dtype=pl.Categorical('physical')).sort()
Out[23]:
shape: (3,)
Series: '' [cat]
[
        "b"
        "a"
        "c"
]

but they both return the same output for .cat.get_categories(). Do you need the output of .cat.get_categories to reflect the category ordering?


nw.is_ordered_categorical just does what Polars does in its dataframe interchange protocol definition:

https://github.com/pola-rs/polars/blob/501988ea1c2a114e4c28619727157354211af93a/py-polars/polars/interchange/column.py#L60-L78

        if dtype == Categorical:
            categories = self._col.cat.get_categories()
            is_ordered = dtype.ordering == "physical"  # type: ignore[attr-defined]
        elif dtype == Enum:
            categories = dtype.categories  # type: ignore[attr-defined]
            is_ordered = True
        else:
            msg = "`describe_categorical` only works on categorical columns"
            raise TypeError(msg)

the interchange protocol definition is a bit vague here, it just says “whether the ordering of dictionary indices is semantically meaningful”

Thanks for adding more info on the table! But I’m not sure if I can read an answer in this already. Or can I understand from here that a categorical with dtype.ordering == "lexical" is intentionaly not ordered? And therefor casting to pl.Categorical('lexcial') is correctly not adding a sort argument to the Vega-Lite specification?

@mattijn no worries, yeah you’ve understood that correctly

Thanks for adding more info on the table! But I’m not sure if I can read an answer in this already. Or can I understand from here that a categorical with dtype.ordering == "lexical" is intentionaly not ordered? And therefor casting to pl.Categorical('lexcial') is correctly not adding a sort argument to the Vega-Lite specification?