vscode-jupyter: IPython.display.Audio not working for WAV audio

Bug: Notebook Editor, Interactive Window, Editor cells

Issue has previously been discussed, but I have new information that is relevant. I’m forced to create a new issue because all previous issues have been closed and locked. Please do not close it without considering this new information and responding to it. I’m only looking for support for uncompressed wav audio format. No need for mp3, mpeg, ogg, or anything else. New information:

  1. Previous discussion with @rchiodo was limited to supporting ffmpeg codec. Support for wav audio files is not dependent in any way on ffmpeg codec. Wav audio can be played back without any codec. No dependency on ffmpeg exists for playing wav audio, which is the most common type used for machine learning because it can be sampled and reconstructed easily (ie wavenet/wavernn).
  2. Microsofter and VSCode creater, @bpasero, claims that sound may be played back without DOM support via node.js https://github.com/Microsoft/vscode/issues/421 This would eliminate previous reasons for closing this issue in the past.
  3. @mjbvz says that ffmpeg codec is unsupported by the core VSCode, but: a) does not state that ffmpeg cant be supported by an extension like vscode-python b) does not state that any codec is required in order to play wav audio files https://github.com/microsoft/vscode/issues/66050
  4. @mjbvz again states here that only certain types of media are unsupported and mentions a dependency on ffmpeg https://github.com/microsoft/vscode/issues/32540 He does not state that all audio playback is unsupported.

Steps to cause the bug to occur

Type the following code in VSCode, and run the cell (e.g. using SHIFT+ENTER):

import numpy as np
import IPython
T = 2.0    # seconds
sr = 22050 # sample rate
t = np.linspace(0, T, int(T*sr), endpoint=False) # time variable
x = 0.5*np.sin(2*np.pi*440*t)                # pure sine wave at 440 Hz
IPython.display.Audio(x, rate=sr)

You will get a disabled player. The same code in the browser will render a usable player.

Actual behavior

IPython.display.Audio renders a disabled player: image

Expected behavior

A valid, playable audio control should be rendered, same as in the browser: image

Your Jupyter and/or Python environment

Please provide as much info as you readily know

  • Jupyter server running: Local
  • Extension version: 2020.5.80290
  • VS Code version: 1.45.1
  • Setting python.jediEnabled: true
  • Setting python.languageServer: Microsoft
  • Python and/or Anaconda version: 3.7.6
  • OS: Linux (Ubuntu 18.04 LTS)
  • Virtual environment: conda

Developer Tools Console Output

Far too big to fit here. No relevant information. Just deprecation warnings and info output.

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 191
  • Comments: 38 (11 by maintainers)

Most upvoted comments

Hi all, I’ve found a workaround using the Web Audio API. Save the following into a file called vscode_audio.py:

import IPython.display
import numpy as np
import json

def Audio(audio: np.ndarray, sr: int):
    """
    Use instead of IPython.display.Audio as a workaround for VS Code.
    `audio` is an array with shape (channels, samples) or just (samples,) for mono.
    """

    if np.ndim(audio) == 1:
        channels = [audio.tolist()]
    else:
        channels = audio.tolist()

    return IPython.display.HTML("""
        <script>
            if (!window.audioContext) {
                window.audioContext = new AudioContext();
                window.playAudio = function(audioChannels, sr) {
                    const buffer = audioContext.createBuffer(audioChannels.length, audioChannels[0].length, sr);
                    for (let [channel, data] of audioChannels.entries()) {
                        buffer.copyToChannel(Float32Array.from(data), channel);
                    }
            
                    const source = audioContext.createBufferSource();
                    source.buffer = buffer;
                    source.connect(audioContext.destination);
                    source.start();
                }
            }
        </script>
        <button onclick="playAudio(%s, %s)">Play</button>
    """ % (json.dumps(channels), sr))

In your notebook, use:

from vscode_audio import Audio
Audio(audio, sr)

where audio in a numpy array with shape (channels, samples) or just (samples,) for mono.

This is quite hacky and slows the whole notebook for longer signals, can be improved. Anyway I suggest to reopen this issue.

It would be nice if it can play audio. image

I too, support the inclusion of audio playing capabilities in the python interactive mode. Not having it makes it difficult for people working with audio processing, inc for machine learning.

Here’s the active issue on VS code: https://github.com/microsoft/vscode/issues/118275

Please upvote that issue.

Hi all, I made a pull request that fixes this issue https://github.com/microsoft/vscode-notebook-renderers/pull/33. Could you open this issue for the time while this pull request is reviewed?

Hello all, I started working on workaround - extracting audio tracks (wav only) from HTML DOM and playing them using Javascript. I’ll keep updates here, planning to make a pull request. зображення

Hi, I will apreciate too to have this issue resolved. It will be nice to play the sounds directly in Vscode instead of going out to the system every time.

Maybe we should first leave this open as it doesn’t get fixed.

Looks like its fixed now in 1.71 (August 2022)

image

Ticket https://github.com/microsoft/vscode/issues/151149 has been selected as candidate for backlog. If we upvote it, it will go into the backlog. It requests the ability to play raw waveforms in VSCode.

Not sure if hosting your own VS code would work. It might.

We closed this bug because there’s not a lot of up votes on audio support and it would take us rewiring the ipython.display.audio javascript/html into some other control that plays the audio.

@austinmw - video also depends the same core issue - https://github.com/microsoft/vscode/issues/118275 As the latest comment states, please comment over there if your specific scenario doesn’t work after the codecs are available.

Thanks @lukepighetti I’ve just upvoted the issue. 😉

Bruh it’s still not fixed… come on

^ I’ve created a feature-request for a barebones functionality to play a mono/stereo np.array from a cell output. I think this should be addressed separately from the much larger task of a fully-fledged media player. If the team would care to provide this, it would give us audio engineers enough to work with. Forever grateful for the tremendous work of the VSCode team, and in particular the .ipynb unit. It’s a splendid technology.

(I’ve posted the same message in https://github.com/microsoft/vscode/issues/118275 which appears to be the nexus for this subject. It seems to me that the smart / practical first move is to first provide a minimal implementation that plays a raw audio-buffer from cell-output. Whatever obstacles are blocking a full implementation of the browser-based Jupyter Notebook cell-output-play-audio functionality, we can assume they are significant, as no plan to fix has been announced despite massive upvoting. So I propose this as a stop-gap.

I, too, would really appreciate this being implemented. This is blocking me from using the jupyter extension