wandb: [App] Table not updating it at each call of log

I have a code for which I want to log some metadata at each step. I thought I could log this metadata in a Table, unfortunately the table is not updated on the app after the first call to log.

My code is something like

metadata = [["I love my phone", "1", "1"]]
table = wandb.Table(data=metadata, columns=["Text", "Predicted Label", "True Label"])
wandb.log({"metadata": table})

run = wandb.init(**config)
with run:
    for step in steps:
        ## some code
        table.add_data(*metadata)
        run.log({"examples": table})

On the app, I can only see the initial table with data no rows are added to it. Note that the corresponding artifact is also not updated. Maybe this is due to wandb assigning the same identity to initial and updated table and not logging it again?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 19
  • Comments: 62 (8 by maintainers)

Most upvoted comments

First of all, I’m sorry on behalf of W&B for not updating this thread in a while.

There are a few options here.

Use W&B History: If you’re just logging text data, not rich media like images, you can use the typical wandb.log({'predicted': 'hello', 'label': 'world'}) API and then use the UI to get a W&B Table.

  • In a workspace, click “Add Panel”
  • Add a Query panel by clicking Query Panel
  • Type runs.history.concat to configure it to concatenate all of your history logs
  • You can then configure the columns shown by clicking columns in the bottom right of the Table This will function as a Table that can continuously log to, regardless of from which run.

See Tables Update: Create a new wandb.Table object each iteration and log it to the same key with wandb.log. This should update in the UI as you train.

Compare 2 Tables: If you want to compare logged Tables:

  • Navigate to the Artifacts tab of your project
  • Click an Artifact
  • Click File
  • Click <table_name>.table.json That should show up your Table.

Now you can click compare with a different version of your artifact. You can either concatenate the two Tables so the table shown will have all the rows from both Tables, or join them using one of the columns, so all other columns are merged.

Concatenate Tables: Note: Advanced & may be limited by performance depending on the size of your Table You can create a Weave panel in your workspace and use runs[0].loggedArtifactVersions.map((row, index) => row.file("<table_key>.table.json")) where <table_key> is the key you’ve logged the table to. This will create one big table with all of the rows from your logged Tables (make sure to create a new Table object every iteration). This is using the Weave query language which is still in active development and may change in the future.

Hello! I am also experiencing the same issue. In the first log call, the table is logged, but any future log calls with an updated table (updated via add_data) does not display the updated data in the UI. I also do not find any of the updated rows in the artifacts section.

I am also experiencing this issue.

How is this still an issue? The feature was first proposed back in 2021, and it’s already 2024! Is it really such a difficult problem for the WandB team to solve?

A work around I’ve just found is creating a new instance of a Table which contains the same columns and data as the old one, and logging that.

new_table = wandb.Table(
    columns=self.wandb_table.columns, data=self.wandb_table.data
)
self.wandb.log({"predictions": new_table}, commit=False)

Before doing this, it would only create a single json for the table in the RUN_DIR/files/media/table folder, but now it’s making one for each time this command is run.

Maybe this can help narrowing down the problem?

As of March 14, 2024, this is THE MOST correct trick.

Although this trick seems to require repeated copies of memory and writes to files. But believe me, it’s the best solution “right now”.

Hi guys!

Apologies for the radio silence on this issue. It looks like our notification software has completely stopped updating us regarding any follow ups in here.

I see that the ticket, at first, has been marked as backlog. Because it was not an easy and timely fix.

I have then re-escalated it to our engineering team, and it was marked for the end of this month for the engineering team to debug and scope. From there, we should have some more solid timelines on this.

I’ll try to keep everyone posted here.

faced with the same issue, would be nice to have a fix

I am surprised this is still not fixed over a year after being raised.

The docs here are quite misleading, as they give the impression that it’s possible to incrementally log text during training. However, it’s not possible to do that with weights and biases at the moment. Having to set up the workarounds for this adds an annoying amount of friction when setting up WandB logging for training with LLMs.

+1 I am also facing this issue, for now I am defaulting to logging the tables after training.

I suffered from this problem too. Thanks to @amitkparekh’s solution, I can get it to work. I made a small tweak tho: I just log a shallow copy of the table:

from copy import copy
...

my_table.add_data(...)
wandb.log({"my_table": copy(my_table)})

+1 on this

So after the first log, any future logs do not do anything. After the finish, it still does not update.

It’s like for the given instance of table, it will only ever be uploaded once. That’s why my workaround solution was to continue adding rows to the table as described in the help, but I log a brand new instantiation of Table using the columns and data from the old one.

A work around I’ve just found is creating a new instance of a Table which contains the same columns and data as the old one, and logging that.

new_table = wandb.Table(
    columns=self.wandb_table.columns, data=self.wandb_table.data
)
self.wandb.log({"predictions": new_table}, commit=False)

Before doing this, it would only create a single json for the table in the RUN_DIR/files/media/table folder, but now it’s making one for each time this command is run.

Maybe this can help narrowing down the problem?

Quick update (I know it’s been a while).

This was assigned to an engineer and worked on this whole year (multiple changes had to take place to enable incremental logging to show in the UI). At the moment, if everything goes according to plan, we are planning on releasing this feature in the next version of our W&B SDK. I will let you know as soon as I hear any more news about this.

A work around I’ve just found is creating a new instance of a Table which contains the same columns and data as the old one, and logging that.

new_table = wandb.Table(
    columns=self.wandb_table.columns, data=self.wandb_table.data
)
self.wandb.log({"predictions": new_table}, commit=False)

Before doing this, it would only create a single json for the table in the RUN_DIR/files/media/table folder, but now it’s making one for each time this command is run. Maybe this can help narrowing down the problem?

As of March 14, 2024, this is THE MOST correct trick.

Although this trick seems to require repeated copies of memory and writes to files. But believe me, it’s the best solution “right now”.

This can be combined with table concatenation so that each time you log you only log a single row (which creates n tables) but then concatenate the results into a single table.

The above code snippet can be modified to:

...
 your_table = wandb.Table(columns=your_column_names)
...
in your training loop:
    do stuff...
    your_table.add_data(<your_data_2_add>)  # add the single row

    # reinit the table without the data ready for next loop
    your_table = wandb.Table(columns=results_table.columns)  

The concatenation can be done using the weave snippet provided by @scottire

runs[0].loggedArtifactVersions.map((row, index) => row.file("{table key>.table.json"))

Noting to update the <table key>

Upside of this work around is only need to update data for a single row rather than duplicating the entire table, downside is you get n tables

It would be nice to have a run.log_row(table_row) function which appends a single row to a single table.

Still facing the same issue.

Any updates on this 6 months later?

I experienced the same problem.

It turns out that the tables are being updated when I look at the artifacts section - there is a new version under the artifact, and I can view the table by going to Files --> {path_to_table} --> {table_name.json}.

But for some reason it is not updating in the UI.

this would be a super helpful feature - would appreciate any progress here

What happened when we call table.add() wandb.log(table) twice?

Here is what happened:

payload saves 2rows, but the file only have one row

After the file creation, payload[key] records the file path, ncols, nrows …, but in the picture we can see that the nrows=2 but the table_key_0_28...table.json only contains 1 row.

Which means there is no further file update except the first time we call wandb.log(table).

Same here

This is still a problem. And same with confusion matrices.

Still same issue. Why is it not resolved yet?

Following - i’m facing this same error

Hi guys! Quick follow up from our engineering team!

We’ve just landed a new feature for logging incremental tables in W&B UI — the feature is available on the main branch of wandb/wandb. If you’d like to try it out ahead of the official release, feel free to install W&B directly from main and experiment with it in your runs!

This update introduces two new logging modes for tables:

Mutable Logging

This allows you to overwrite the table each time it’s logged, replacing what you see in the UI. This is useful for adding columns or rows to existing Tables.

Trade-off: When a table gets very large, serializing it for logging can impact your workload if done frequently.

Usage Example:

import wandb
import numpy as np

run = wandb.init(project="mutable-table-demo")

# Create a table object with MUTABLE logging mode
table = wandb.Table(columns=["input", "label", "prediction"],
                    log_mode="MUTABLE")

# Load data and make predictions
inputs, labels = load_eval_data() # Placeholder function
raw_preds = model.predict(inputs) # Placeholder function

for inp, label, pred in zip(inputs, labels, raw_preds):
    table.add_data(inp, label, pred)

# Step 1: Log initial data
wandb.log({"eval_table": table})  # Log initial table

# Step 2: Add confidence scores (e.g. max softmax)
confidences = np.max(raw_preds, axis=1)
table.add_column("confidence", confidences)
run.log({"eval_table": table})  # Add confidence info

# Step 3: Add post-processed predictions
# (e.g., thresholded or smoothed outputs)
post_preds = (confidences > 0.7).astype(int)
table.add_column("final_prediction", post_preds)
wandb.log({"eval_table": table})  # Final update with another column

run.finish()

Incremental Logging

This table logging mode is used to update the table with batches of new rows. You can view the table while your run is ongoing in the run’s workspace.

Key things to know:

Due to system limitations, only the last 100 incremental updates will be visible in the Run Workspace, but you can still step through all the increments using the table stepper feature on runs.history

Trade-off: The 100-increment display limit, but this can be mitigated (see below).

Usage Example:

import wandb

run = wandb.init(project="incremental-table-demo")

# Create a table with INCREMENTAL logging mode
table = wandb.Table(columns=["step", "input", "label", "prediction"],
                    log_mode="INCREMENTAL")

# Training loop
for step in range(get_num_batches()): # Placeholder function
    # Load batch data
    inputs, labels = get_training_batch(step) # Placeholder function

    # Train and predict
    train_model_on_batch(inputs, labels) # Placeholder function
    predictions = predict_on_batch(inputs) # Placeholder function

    # Add batch data to table
    for input_item, label, prediction in zip(inputs, labels, predictions):
        table.add_data(step, input_item, label, prediction)

    # Log the table incrementally
    wandb.log({"training_table": table}, step=step)

run.finish()

Get the best of both worlds, use two Tables, one with log_mode=“INCREMENTAL” and one with the default (log_mode="IMMUTABLE"):

  • Log batches incrementally throughout your run.

  • At the end of the run, create a new wandb.Table object from your final data and log it one last time.

This gives you:

  • Updates on the UI during the run.

  • Full visibility of the table data in the Project & Run Workspace, without being limited by the 100-increment view cap in the Run Workspace.

Documentation is still in progress, but we wanted to share this early so you can try it out and provide feedback. Let us know how it works for your use case!

You can install wandb off main via pip install git+https://github.com/wandb/wandb.git@main

Thanks!

@ArtsiomWB any update?

Also facing this issue.

Hi! We are still working on this issue - this requires a large refactor to our logic of how we handle tables due to which this is something which has not been implemented yet. I’ll post back here once I have some some good news about the progress of this feature!

Hi @amitkparekh is this one of tables that weren’t updating properly?

Sorry, no it isn’t the same tables.

On my own project, I’ve been having the same issue where after logging an instance of wandb.Table once, all future “wandb.log’s” are ignored.

I am following the same logic outlined above by @oumarkaba: creating a table at the start, then adding data and logging it each epoch.

It seems that after the first time you log a table, logging that same instance of the table, with or without any data changes does nothing. Looking within the local files for the run, no new table artifacts are created after the first logging, so it might be something in the client-side code preventing it from being updated?

Using the workaround, where I just log a newly created wandb.Table instance using the same columns and data from the table within the state, it works properly — creating the local table artifact files and updating the dashboard as expected.


For completeness, I am using allennlp and have extended their WandBCallback. This is the full module of the callback I am using.

  • I’m creating the wandb.Table instance on start
  • I update the table at the end of each epoch
  • I create a new instance of the table and log the new instance, just as in the the above workaround, and then discard the new instance to save on memory.
import logging
from typing import Any

import wandb
from allennlp.training import GradientDescentTrainer
from allennlp.training.callbacks import TrainerCallback, WandBCallback


logger = logging.getLogger(__name__)


def get_training_stage_from_key(key: str) -> str:
    if key.startswith("training"):
        return "training"
    if key.startswith("validation"):
        return "validation"
    return "unknown"


@TrainerCallback.register("alt_wandb")
class AltWandBCallback(WandBCallback):
    def on_start(
        self,
        trainer: GradientDescentTrainer,
        is_primary: bool = True,
        **kwargs: Any,
    ) -> None:
        if not is_primary:
            return None

        super().on_start(trainer, is_primary, **kwargs)

        # Create prediction table
        columns = ["epoch", "stage", "task", "prediction", "target"]
        self.wandb_table = wandb.Table(columns=columns)

    def on_epoch(
        self,
        trainer: GradientDescentTrainer,
        metrics: dict[str, Any],
        epoch: int,
        is_primary: bool = True,
        **kwargs: Any,
    ) -> None:
        if not is_primary:
            return None

        self._log_predictions_table(metrics, epoch)

        filtered_metrics = {
            name: metric for name, metric in metrics.items() if "__" not in name
        }

        super().on_epoch(trainer, filtered_metrics, epoch, is_primary, **kwargs)

    def _log_predictions_table(self, metrics: dict[str, Any], epoch: int) -> None:
        prediction_metric_keys = {"training__predictions", "validation__predictions"}

        for key in prediction_metric_keys:
            if key not in metrics.keys():
                continue

            stage = get_training_stage_from_key(key)

            for task_prediction in metrics[key].values():
                logger.info(f"Adding data for {task_prediction}")
                self.wandb_table.add_data(epoch, stage, *task_prediction)

        new_table = wandb.Table(
            columns=self.wandb_table.columns, data=self.wandb_table.data
        )

        self.wandb.log({"predictions": new_table}, commit=False)