napari: Color mapping on label images using color dictionaries displays wrong for some objects

I think I have it; I’ll draft a PR shortly. If you want to try the branch before then, please let me know if it works.

Jules Vanaret’ example (from ImageSC) looks like this:

@Cryaaa’s example…

Unfortunately, @aeisenbarth’s example won’t be fixed. The underlying implementation uses a 1024-element 2D texture for storing colors. Your example creates 262144 unique colors… Not to say you’re forgotten – I believe this calls for some documentation and/or warning before we implement a proper fix.

I have a minor update. I was able to determine that color_dict_to_colormap is generating incorrect control points.

For example, with five colors, this function will generate control points like [0, 0.25, 0.5, 0.75, 1.0], while Colormap generates [0, 0.2, 0.4, 0.6, 0.8, 1.0]. The extra control point (1.0) is needed with ColormapInterpolationMode.ZERO.

I have a branch with this fix, but it’s insufficient. Something is still going wonky when this data is sent to the GPU.

More soon (hopefully).

I recently fell into this problem and created a test case, not knowing that this issue exists already.

• This happens for most colormaps with more than 2 control points (like viridis; not for gray and also not noticeable for gist_earth).
• It becomes more prominent with a large number of labels.
• I suspected floating point precision and checked that all relevant variables actually were float64 (my values, color map colors and controls), which would surprise me if it is still the cause.

My code may be useful since it has small, artificial data and assertions.

from typing import Tuple

import napari.viewer
import numpy as np
import pandas as pd
import pytest
from napari.layers import Labels
from napari.layers.labels._labels_constants import LabelColorMode
from napari.utils.colormaps import AVAILABLE_COLORMAPS

# ########## The function to test ##########


def create_colored_labels_layer(
    labels_image: np.ndarray, values_series: pd.Series, colormap_name: str
):
    # Create a labels layer
    layer = Labels(labels_image, name="actual labels")
    layer.color_mode = LabelColorMode.DIRECT
    # Create a color dictionary which assigns to each label an RGBA color based on the values.
    layer.colormap = colormap_name
    original_colormap = layer.colormap
    color = {label: original_colormap.map(value).ravel() for label, value in values_series.items()}
    layer.color = color
    return layer


# ########## Helper functions ##########


def create_random_labels_with_values(width: int) -> Tuple[np.ndarray, pd.Series]:
    # Create a randomized labels image
    labels = list(range(width**2))
    _labels = list(labels)
    np.random.default_rng().shuffle(_labels)
    labels_image = np.reshape(_labels, (width, width))
    # Create a horizontal gradient of values.
    # This makes it easy to visually find mismatching colors.
    values = np.linspace(start=0, stop=1.0, num=width)
    values_array = np.repeat(values, width).reshape((width, width)).T
    # Series mapping labels and values, as reference.
    values_series = pd.Series(values_array.ravel(), index=labels_image.ravel()).sort_index()
    return labels_image, values_series


def create_actual_and_expected_images(
    actual_layer: Labels, values_series: pd.Series, expected_colormap
):
    shape = actual_layer.data.shape
    labels_image = actual_layer.data
    # Compute the image array as Napari displays it
    actual_rgba_image = actual_layer.colormap.map(
        [actual_layer._label_color_index[label] for label in labels_image.ravel()]
    ).reshape(shape + (-1,))
    # Compute the image array as it should be displayed
    values_normalized = (values_series - values_series.min()) / (
        values_series.max() - values_series.min()
    )
    values_image = values_normalized.values[labels_image]
    expected_rgba_image = expected_colormap.map(values_image.ravel()).reshape(shape + (-1,))
    return actual_rgba_image, expected_rgba_image


# ########## Unit test ##########


@pytest.fixture(params=[2, 100])
def random_labels_with_values(request) -> Tuple[np.ndarray, pd.Series]:
    return create_random_labels_with_values(width=request.param)


@pytest.mark.parametrize("colormap_name", ["gray", "viridis"])
def test_napari_labels_coloring(random_labels_with_values, colormap_name):
    labels_image, values_series = random_labels_with_values
    actual_layer = create_colored_labels_layer(labels_image, values_series, colormap_name)

    # Extract data for assertions
    original_colormap = AVAILABLE_COLORMAPS[colormap_name]
    # Compute the colors mapped to the values
    # We compare the green component because for viridis, it is monotonically increasing.
    actual_colors_series = pd.Series(
        actual_layer.colormap.map(values_series.values)[:, 1], index=values_series.index
    )
    expected_colors_series = pd.Series(
        original_colormap.map(values_series.values)[:, 1], index=values_series.index
    )
    actual_colors2_series = pd.Series(
        actual_layer.colormap.map(
            [actual_layer._label_color_index[label] for label in values_series.index]
        )[:, 1],
        index=values_series.index,
    )
    expected_colors2_series = pd.Series(
        original_colormap.map(
            [actual_layer._label_color_index[label] for label in values_series.index]
        )[:, 1],
        index=values_series.index,
    )
    # Compute the image array as it should be displayed and as Napari displays it.
    actual_rgba_image, expected_rgba_image = create_actual_and_expected_images(
        actual_layer, values_series, AVAILABLE_COLORMAPS[colormap_name]
    )

    # Assertions
    np.testing.assert_almost_equal(actual_rgba_image, expected_rgba_image)
    pd.testing.assert_series_equal(actual_colors_series, expected_colors_series)
    pd.testing.assert_series_equal(actual_colors2_series, expected_colors2_series)
    # The mapped colors should not have discontinuities, that means have the same monotonic order.
    pd.testing.assert_series_equal(
        np.argsort(actual_colors_series.drop_duplicates()),
        np.argsort(expected_colors_series.drop_duplicates()),
    )


if __name__ == "__main__":
    # Interactively visualize the issue in Napari
    labels_image, values_series = create_random_labels_with_values(width=1000)
    layer = create_colored_labels_layer(labels_image, values_series, "viridis")
    actual_rgba_image, expected_rgba_image = create_actual_and_expected_images(
        layer, values_series, AVAILABLE_COLORMAPS["viridis"]
    )
    viewer = napari.viewer.Viewer()
    viewer.add_image(expected_rgba_image, name="expected image")
    viewer.add_image(actual_rgba_image, name="actual image")
    viewer.add_layer(layer)
    napari.run()

The promised example where it still fails at 1000 colors:

I made two test cases with 1000 colors (or an arbitrary number): Once with all the colors in order. And a slightly more complex test case where the label values are “shuffled”, such that they aren’t in order left to right. That way, rounding errors should stand out more, I thought. The results were surprising to me.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import napari

# Set the number of steps
nb_steps = 1000

# Create a dummy label image
base = np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint16')
label_img = np.repeat(base.reshape([1, base.shape[0]]), int(nb_steps/10), axis=0)

# Add a harder test case: Randomly order the label values. 
# But assign them monotonously increasing feature values (=> if it's off by one, the error is more visible)

shuffled = np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint16')
np.random.shuffle(shuffled)
label_img_shuffled = np.repeat(shuffled.reshape([1, shuffled.shape[0]]), int(nb_steps/10), axis=0)
df_shuffled = pd.DataFrame([shuffled, np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint16')]).T
df_shuffled.columns = ['label', 'feature']

# calculate the colormaps manually
lower_contrast_limit = 1
upper_contrast_limit = nb_steps

df = df_shuffled

df['feature_scaled_shuffled'] = (
    (df['feature'] - lower_contrast_limit) / (upper_contrast_limit - lower_contrast_limit)
)
colors = plt.cm.get_cmap('viridis')(df['feature_scaled_shuffled'])
colormap_shuffled = dict(zip(df['label'].astype(int), colors))

df['feature_scaled_not_shuffled'] = (
    (df['label'] - lower_contrast_limit) / (upper_contrast_limit - lower_contrast_limit)
)
colors_ordered = plt.cm.get_cmap('viridis')(df['feature_scaled_not_shuffled'])
colormap_ordered = dict(zip(df['label'].astype(int), colors_ordered))

# Add to napari
viewer = napari.Viewer()
viewer.add_image(label_img, colormap='viridis')
labels_layer_shuffled = viewer.add_labels(label_img_shuffled, opacity=100)
labels_layer_ordered = viewer.add_labels(label_img, opacity=100)

# Set the label image colormaps
labels_layer_shuffled.color = colormap_shuffled
labels_layer_ordered.color = colormap_ordered

When just using the viridis colormap on the intensity image, we get this nice gradient (based on the non-shuffled image)

When using napari 0.4.16, using the direct color mode on the label maps leads to this:

Whole blocks are assigned the wrong color. Interestingly enough, the ordered labels and the shuffled labels are the same, reinforcing again how it depends on underlying rounding somewhere of the color we provide to what is displayed, not e.g. on the matching of a color to its label value.

When using the branch with the fix for the colormaps, it looks way better (thanks a lot @perlman !). But it still makes a bit of a mistake on the left side. 4 consecutive labels (4 colors that should be close to each other) get assigned the same, wrong value

There are 2 interesting takeaways here:

1. The new implementation makes much fewer errors. When setting nb_steps = 100, I can’t see any errors in the viridis colormap anymore, while one can see multiple errors in the old implementation. When setting nb_steps = 1000, there is an error again though. So the limit where it starts to fail is much higher, but <1000.
2. There still seems to be some rounding going on. I just noticed thanks to the “stripe” in the nb_steps = 1000: This stripe is actually 4 label values broad. And, while it’s a big hard to distinguish, when zooming in on a good monitor, one can actually see that it’s always blocks of 4 labels that get the exact same color assigned. So there appears to be some rounding going on for very similar colors that get match together to the same underlying thing. And most of the time, that is close to being correct. But in rare cases, it apparently is not (=> 1, matching the 4 colors to something that is far away from what it should be)

EDIT: The binning of 4 colors together when there are 1000 different colors also already happened in 0.4.16, I just didn’t really notice there because there are much more problems there 😃

Thank you for the data & code.

As to the specific issue here with the layer colormap different from the picker, I think this is being caused by some floating point precision oddities. (The relevant code is here, mostly as a placeholder for myself to step through later.)

@jni We do need to come up with a strategy for handling Labels-as-LUT in #3308, as this seems to be a popular use for the labels!

I’m hoping that we can fix this as part of #3308, though we haven’t yet looked at manual color settings.