code-server: Cannot run jupyter kernel using python extension

Here’s a matrix table with links for those who want it working. The bottomline being: you can’t use marketplace from code-server itself. You either need to manually download, drag-and-drop and install; or do something equivalent.

@jsjoeio What’s the best way to help here? Build a working version myself and send it over? Try and figure out how to edit the current command so the process starts working again? Can we make your scraper use the build artifacts from their CI/CD releases? Try to engage the maintainers to put the link directly to the VSIX in their releases? Can we re-upload to coder’s marketplace the VSIX from Open VSX?

So should Jupyter now work after we update?

Once 3.12.1 is out, yes!

https://open-vsx.org/extension/ms-toolsai/jupyter/2021.8.12 plus https://open-vsx.org/extension/ms-python/python/2021.9.1218897484 are working fine with v3.12.0 on Debian 11 (Bullseye).

Thanks @simeneide for working with me (Python/Jupyter noob here). That was very easy to follow and I received the same error as you.

I can’t promise when we will get to this but I will add it to the next cycle and label it as high-priority.

Video

(Apologies in advance if I pronounced your name wrong)

https://user-images.githubusercontent.com/3806031/127691663-7c037ef6-dad0-4342-a271-0470eaf3685a.mov

Assuming older versions of the extension work my guess is the extension was updated and our marketplace had a snafu building the new extension (as it seems to every time). Since we haven’t switched to Open VSX yet and we don’t have a way to force the marketplace to stop updating extensions automatically we may need to build it manually and overwrite it (again…).

@simeneide hooray! Well I’m happy it was easy as that 😃 Thanks for confirming so quickly.

yes sorry @jsjoeio , in my update I note that when I deleted my old config files it ran smoothing. dont know why, but it runs well now 😃

Yes! Jupyter kernel should be working in the next release (it was 3.12.1 but it will be 4.0.0) as @im-coder-lg mentioned!

I have a (kind of) idea. Why not install Anaconda3? That has Jupyter Notebook built-in, so you can create a notebook from that, right?

Okay we figured it out by adding bash as a dependency to the homebrew formula and it works locally.

We should be running the postinstall with sh, but we made it run with bash because of Windows 🤔

Powershell doesn’t have sh or bash by default 🤔 I think we need to revert this line until we can add proper Windows support, specifically revert this PR: https://github.com/cdr/code-server/pull/4116

Will discuss more tomorrow

Thanks @edvincent for posting the matrix and offering to help! To be honest, the hardest part is finding time. I’m the only maintainer working full-time on code-server and I’m trying to juggle all the issues we have. My current priorities are improving the release process (see milestone).

I’ve added this issue there as well but haven’t been able to dive in yet.

What’s the best way to help here?

Thank you for asking that!

Well…we completely refactored how VS Code exists in the code-server codebase and it’s in a funky place right now so it wouldn’t be easy to get you into the codebase trying to edit the current command (we’re lacking documentation cc @TeffenEllis).

Can we make your scraper use the build artifacts from their CI/CD releases?

Maybe? I’m not super familiar with this part of our codebase yet and haven’t had time to dive in.

Try to engage the maintainers to put the link directly to the VSIX in their releases?

You could try this! I don’t think we’ve had much success in the past though 😢

Can we re-upload to coder’s marketplace the VSIX from Open VSX?

If you have a working vsix file that works with the latest version of code-server and send that over here, that might be easy then for me to upload it to our marketplace 🤔 (at least I’m hopefully it’s a simple CLI command to upload, I think it should be).

Otherwise, hoping to dive in on this issue this week! Just gotta wrap up another thing

I’m think the problem is with the scraped extension?

[main 2021-09-17T01:28:50.508Z] RequestService#request https://extensions.coder.com/api/extensionquery
[main 2021-09-17T01:28:50.508Z] resolveShellEnv(): running (macOS/Linux)
[main 2021-09-17T01:28:50.693Z] RequestService#request https://storage.googleapis.com/vscode-extension-assets/vscode-extensions/vscode/ms-toolsai/jupyter/2021.8.12/Microsoft.VisualStudio.Code.Manifest
[main 2021-09-17T01:28:50.693Z] resolveShellEnv(): running (macOS/Linux)
[main 2021-09-17T01:28:50.762Z] Started scanning system extensions
[main 2021-09-17T01:28:50.762Z] Started scanning user extensions
[main 2021-09-17T01:28:50.763Z] Scanned user extensions: 0
[main 2021-09-17T01:28:50.794Z] Scanned system extensions: 83
[main 2021-09-17T01:28:50.794Z] Installing extension: ms-toolsai.jupyter
[main 2021-09-17T01:28:50.795Z] Started scanning system extensions
[main 2021-09-17T01:28:50.795Z] Started scanning user extensions
[main 2021-09-17T01:28:50.796Z] Scanned user extensions: 0
[main 2021-09-17T01:28:50.825Z] Scanned system extensions: 83
[main 2021-09-17T01:28:50.825Z] Started downloading extension: ms-toolsai.jupyter
[main 2021-09-17T01:28:50.826Z] ExtensionGalleryService#download ms-toolsai.jupyter
[main 2021-09-17T01:28:50.826Z] RequestService#request https://storage.googleapis.com/vscode-extension-assets/vscode-extensions/vscode/ms-toolsai/jupyter/2021.8.12/Microsoft.VisualStudio.Services.VSIXPackage?redirect=true&install=true

That specific VSIX that gets returned from that coder marketplace is not complete. The out directly doesn’t even have the .out/client/extension which is the main of the package.

I’m gonna guess that because vscode-jupyter does not upload a VSIX in their releases, so it’s not only a scrape, but there are some extra build steps that are actually done on coder’s side - but the build is failing and an incomplete extension is being generated and sent out?

Can someone check the builds logs of what’s being scraped from that vscode-jupyter repo? I’m gonna assume that the npm run package is failing somewhere?

The OpenVSX VSIX does have a full extension, and why it works when it gets installed from there / from a manually-downloaded version.

PS: The extension is also missing couple of the required python libraries actually. You’re likely missing running pip and dumping those two libraries from requirements.txt in the pythonFiles folder, along the lines of

pip install -t ./pythonFiles/lib/python -r requirements.txt

I do not think we have anything like that unfortunately. We have some comments about working versions but that is about it. Here are a few:

https://github.com/cdr/code-server/issues/3782#issuecomment-882715196 https://github.com/cdr/code-server/issues/2929#issuecomment-808774061 https://github.com/cdr/code-server/issues/2341#issuecomment-739562822 https://github.com/cdr/code-server/issues/2341#issuecomment-801450552

Do you know what is missing, @jsjoeio? Otherwise I am pretty sure I can cook up a simple docker image to reproduce this so that you can have a look, @balajib-b.

@simeneide the installed jupyter extension facilitates VS Code to interface with jupyter package installed in your env, it doesnt install jupyter notebook to your current environment. I would be happy if you could confirm that the python environment you are using has got jupyter installed. Open terminal and type “python -m pip list” and check jupyter notebook and its dependent packages are installed. i have been using mixed stack of code server, jupyter, miniconda, pytorch, tensorflow with no issues. Also requesting you to use latest version of code server.

Is it possible for you to update the repro steps? What do I need to install or how can I create a new jupyter notebook? Then I can try to reproduce locally and see how we can help you.

Updated, thanks for coming back to me 😃 I thought “everyone” had this workflow, because it is such a good one 😉

https://github.com/cdr/code-server/discussions/4490#discussioncomment-1633165

This was the reply, so there’s no 3.12.1 or 3.12.2.

Wait, no, Jupyter is open-source? https://github.com/microsoft/vscode-jupyter/releases/tag/2021.8.12

@jsjoeio Yes. This issue has only been resolved recently by https://github.com/microsoft/vscode-jupyter/pull/5812, which allowed updating both Python and Jupyter extensions on OpenVSX to a current version.
ℹ️ Don Jayamanne’s Python extension had been stuck at version 2020.10.332292344 on OpenVSX for months due to its dependency on the Jupyter extension and the runtime license of the latter.

@edvincent Why not use the OpenVSX Registry by default? Sooner or later this will be the default anyway.

According to FAQ.md > How do I use my own extensions marketplace?:

export SERVICE_URL=https://open-vsx.org/vscode/gallery
export ITEM_URL=https://open-vsx.org/vscode/item

ℹ️ This is working fine for my builds.

Again, thanks for looking into this. Although many similar comments, I can neither get this to work for 3.12.

i just installed 3.12 clean (deleted all cache and configs) and still have the issue on ubuntu 20.04 lts.

Updated to 3.12.0 and using jupyter-2021.8.12 with python-2021.9.1218897484, still got the same issue.

Thank you, @benz0li for mentioning this here, I can confirm! I use docker to set up cdr code server and that’s the first time in a long time this was working again:

## Code server
RUN mkdir -p ~/.local/lib ~/.local/bin
RUN curl -sfL https://github.com/cdr/code-server/releases/download/v3.12.0/code-server-3.12.0-linux-amd64.tar.gz | tar -C ~/.local/lib -xz
RUN mv ~/.local/lib/code-server-3.12.0-linux-amd64 ~/.local/lib/code-server-3.12.0
RUN ln -s ~/.local/lib/code-server-3.12.0/bin/code-server ~/.local/bin/code-server
RUN PATH="~/.local/bin:$PATH"

RUN curl -sfLO https://open-vsx.org/api/ms-toolsai/jupyter/2021.8.12/file/ms-toolsai.jupyter-2021.8.12.vsix \
 && curl -sfLO https://open-vsx.org/api/ms-python/python/2021.9.1218897484/file/ms-python.python-2021.9.1218897484.vsix \
 && ~/.local/bin/code-server --install-extension ./ms-toolsai.jupyter-2021.8.12.vsix || true \
 && ~/.local/bin/code-server --install-extension ./ms-python.python-2021.9.1218897484.vsix || true \
 && rm ms-toolsai.jupyter-2021.8.12.vsix \
 && rm ms-python.python-2021.9.1218897484.vsix

Hi,

I am having the same issue with

• code-server v3.11.1
• VS Code v1.57.1
• ms-python.python v2021.8.1159798656
• Jupyter ms-toolsai v2021.6.99, v2021.5.0 (also 2021.8.2031173646 is not compatible with VS Code v1.57.1) (Note : I am also using an anaconda python environment, wher Jupyter is installed and is working fine)

=> Is there’s somewhere a compatibility matrix of the code-server / VS Code / ms-toolsai versions to be used together and that are known to be functional ?

Thx !

Do you know what is missing, @jsjoeio? Otherwise I am pretty sure I can cook up a simple docker image to reproduce this so that you can have a look, @balajib-b.

that would be sufficient @simeneide , send me the yaml.

Thanks for the notes @gogobd!

It would be great if Jupyter Notebooks could be fixed in current versions.

It’s in the next milestone (3.12) so it will get worked on soon and should come in the next version. We have a couple competing priorities and an off-site this week so I can’t promise when but keep an eye out!

I have the very same problem in 3.11.1 and all previous versions. My last working configuration is code-server-3.9.3-linux-amd64. It would be great if Jupyter Notebooks could be fixed in current versions.

https://github.com/cdr/code-server/issues/3776 https://github.com/cdr/code-server/issues/3782 https://github.com/cdr/code-server/issues/2741

@simeneide i doubt whether your jupyter installation is in path. there could be few more possibilities pertaining to this error. ping me at ji.balajib[at]gmail.com lets connect remote and debug. Im happy to help

i think this way its going to be quick and then post back solution here. this could help @jsjoeio close few more issues.

Yes, all has and I am also able to start jupyter notebook. Currently using 3.11.0 as that is the latest found in brew (which is used when installing with the script). However, it fails much before it starts connecting to the jupyter kernel (as seen in video), so unsure if this is the problem. Are you not able to reproduce my steps? Alternatively, can you provide a step-by-step of how you get your setup to work?

Also: My steps above are working on 3.10, so something has happened.

Thanks for opening the issue! I have never used jupyter notebooks.

Is it possible for you to update the repro steps? What do I need to install or how can I create a new jupyter notebook? Then I can try to reproduce locally and see how we can help you.

Thanks!