WalletWasabi: 1.1.10 consistently crashes on startup on NixOS

I’m happy to report that the v1.1.10.2-200122 build does not crash on startup on NixOS, and as far as I can tell in some basic testing seems to have no major issues, so the Avalonia fixes seem to have helped.

Awesome! Closing. Will try to release silently tomorrow.

aaand fixed on master 😃

Confirmed on Lubuntu 14.04 VM with the current release (1.1.10.2).

Here we go again. Can you try the master branch from code? #2988

Thanks @nopara73. Could you also creating a release? (on your own fork of the repo is fine)

(The current Nix package for Wasabi just unpacks the released .tar.gz and patches the wassabee binary and .so files. https://github.com/NixOS/nixpkgs/pull/73262/ did land, so in theory we could switch the Nix package to build using dotnet-sdk 3.1, but I don’t know how much time I can spend on looking into that, so testing from a released .tar.gz would speed things up.)

@jmacato sure:

1. Download the VirtualBox image at https://nixos.org/nixos/download.html
2. Once booted, add the channel that has the change:

nix-channel --add https://github.com/mmahut/nixpkgs/archive/wasabi.tar.gz wasabi
nix-channel --update

1. Install 1.1.10.1:

nix-env -iA wasabi.wasabiwallet

1. Run wasabiwallet.

Thanks for the fix, but v1.1.10.1 seems to still have the issue reported here on NixOS (the hang on --version does appear to be fixed though):

$ wasabiwallet --version
Wasabi Client Version: 1.1.10.1
Compatible Coordinator Version: 3
Compatible Bitcoin Core Version: 0.18.1
Compatible Hardware Wallet Interface Version: 1.0.3

$ wasabiwallet
2019-12-23 03:40:14 INFO        Program (44)    Wasabi GUI started (ac132b16-1f34-4c8b-a709-43b9e2237d14).
2019-12-23 03:40:17 INFO        Global (163)    Config is successfully initialized.
2019-12-23 03:40:17 WARNING     IndexStore (111)        Old Index file detected. Deleting it.
2019-12-23 03:40:17 WARNING     IndexStore (114)        Successfully deleted old Index file.
2019-12-23 03:40:17 INFO        TransactionStore (28)   MempoolStore.InitializeAsync finished in 22 milliseconds.
2019-12-23 03:40:18 INFO        TorProcessManager (249) Starting Tor monitor...
2019-12-23 03:40:18 INFO        Global (225)    TorProcessManager is initialized.
2019-12-23 03:40:18 INFO        IndexStore (43) InitializeAsync finished in 160 milliseconds.
2019-12-23 03:40:18 INFO        TorProcessManager (66)  Tor is already running.
2019-12-23 03:40:18 INFO        TransactionStore (28)   ConfirmedStore.InitializeAsync finished in 502 milliseconds.
2019-12-23 03:40:18 INFO        Global (399)    Loaded AddressManager from `/home/user/.walletwasabi/client/AddressManager/AddressManagerMain.dat`.
2019-12-23 03:40:18 INFO        MainWindow.xaml (74)    UiConfig is successfully initialized.
2019-12-23 03:40:18 INFO        AllTransactionStore (27)        InitializeAsync finished in 848 milliseconds.
2019-12-23 03:40:18 INFO        BitcoinStore (39)       InitializeAsync finished in 893 milliseconds.
2019-12-23 03:40:18 INFO        HostedServices (49)     Started Software Update Checker.
2019-12-23 03:40:19 INFO        Global (334)    Start connecting to nodes...
2019-12-23 03:40:19 INFO        Global (356)    Start synchronizing filters...
2019-12-23 03:40:19 WARNING     Program (95)    System.NullReferenceException: Object reference not set to an instance of an object.
   at Avalonia.Shared.PlatformSupport.StandardRuntimePlatform.UnmanagedBlob.DoDispose()
   at Avalonia.Shared.PlatformSupport.StandardRuntimePlatform.UnmanagedBlob.Finalize()
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Avalonia.Shared.PlatformSupport.StandardRuntimePlatform.UnmanagedBlob.DoDispose()
   at Avalonia.Shared.PlatformSupport.StandardRuntimePlatform.UnmanagedBlob.Finalize()
Aborted (core dumped)

$ wasabiwallet
2019-12-23 03:40:38 INFO        Program (44)    Wasabi GUI started (d57f7806-6b7a-4cc8-8078-2afe277ede5f).
2019-12-23 03:40:41 INFO        Global (163)    Config is successfully initialized.
2019-12-23 03:40:42 INFO        IndexStore (43) InitializeAsync finished in 54 milliseconds.
2019-12-23 03:40:42 INFO        TransactionStore (28)   MempoolStore.InitializeAsync finished in 27 milliseconds.
2019-12-23 03:40:42 INFO        TorProcessManager (249) Starting Tor monitor...
2019-12-23 03:40:42 INFO        Global (225)    TorProcessManager is initialized.
2019-12-23 03:40:42 INFO        TorProcessManager (66)  Tor is already running.
2019-12-23 03:40:42 INFO        TransactionStore (28)   ConfirmedStore.InitializeAsync finished in 484 milliseconds.
2019-12-23 03:40:42 INFO        AllTransactionStore (27)        InitializeAsync finished in 517 milliseconds.
2019-12-23 03:40:42 INFO        BitcoinStore (39)       InitializeAsync finished in 563 milliseconds.
2019-12-23 03:40:42 INFO        Global (399)    Loaded AddressManager from `/home/user/.walletwasabi/client/AddressManager/AddressManagerMain.dat`.
2019-12-23 03:40:42 INFO        HostedServices (49)     Started Software Update Checker.
2019-12-23 03:40:42 INFO        MainWindow.xaml (74)    UiConfig is successfully initialized.
2019-12-23 03:40:42 INFO        Global (334)    Start connecting to nodes...
2019-12-23 03:40:42 INFO        Global (356)    Start synchronizing filters...
2019-12-23 03:40:47 WARNING     Program (95)    System.NullReferenceException: Object reference not set to an instance of an object.
   at Avalonia.Shared.PlatformSupport.StandardRuntimePlatform.UnmanagedBlob.DoDispose()
   at Avalonia.Shared.PlatformSupport.StandardRuntimePlatform.UnmanagedBlob.Finalize()
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Avalonia.Shared.PlatformSupport.StandardRuntimePlatform.UnmanagedBlob.DoDispose()
   at Avalonia.Shared.PlatformSupport.StandardRuntimePlatform.UnmanagedBlob.Finalize()
Aborted (core dumped)

oh okay, seems someone also encountered this on Avalonia 0.9 (via Arch Linux). This PR should hopefully fix this: https://github.com/AvaloniaUI/Avalonia/pull/3363

and Chromium seems to have stumbled upon this bug 10 months ago: https://r882-11cdd11fc1da-maruel-dot-chromiumcodereview.appspot.com/1707283002/ (credits to: @PJB3005)

Also maybe @MaxHillebrand with your new monster, can you fire up a lubuntu vm?

It’s actually Lubuntu (a light version of Ubuntu). Not sure any differences

Here’s it: https://github.com/zkSNACKs/WalletWasabi/issues/2939#issuecomment-577087483

@molnard as we also need a packaged binary for testing on tails, it would really be great if you can make both .deb and .tar.gz in your own fork and provide a download link. 🙏

Here we go again. Can you try the master branch from code? https://github.com/zkSNACKs/WalletWasabi/pull/2988

NixOS cannot find tor even when it is there. I found that this same problem is affecting other projects too and this is a relevant answer in StackOverflow:

This is usually from the binary not able to find a library that it’s linked against. If you do ldd .walletwasabi/client/tor/Tor/tor you’ll see that it couldn’t a few things. Alternatively, a binary might do dlopen in which you wouldn’t see it ldd, even though a particular library is needed. If you’re setting a runpath with patchelf, then you’ll want to keep $ORIGIN in the runpath so that it’s able to pick up other binaries in the same directory (E.g. --set-rpath ‘$ORIGIN:${rpath}’)

Apparently NixOS is a bit different that other linux distros and we have to decide if we want to support it officially because it will require further investigation, changes and testing specially focused on this distro.

NixOS is very different as it doesn’t conform to FHS:

[11:47:03] jon@jon-workstation ~/projects/nixpkgs (master)
$ tree /bin/
/bin/
└── sh -> /nix/store/m7v6p7yxqyhnc4sxp49zp857cjamp996-bash-interactive-4.4-p23/bin/sh

0 directories, 1 file
[11:47:08] jon@jon-workstation ~/projects/nixpkgs (master)
$ tree /usr
/usr
└── bin
    └── env -> /nix/store/gnw6yrqy249n62r4q8vy12ispviv3dav-coreutils-8.31/bin/env

All packages are installed in /nix/store/ at hashed paths (hash is based on all dependencies and configuration settings), so the only way they can run is if they reference their dependencies exactly, this allows for greater reproducibility of builds, as well allows for a way to gracefully add or remove packages from an environment. Example:

[11:52:44] jon@jon-workstation ~/projects/nixpkgs (master)
$ ruby
The program ‘ruby’ is currently not installed. It is provided by
several packages. You can install it by typing one of the following:
  nix-env -iA nixos.jruby
  nix-env -iA nixos.logstash6-oss
  nix-env -iA nixos.logstash7-oss
  nix-env -iA nixos.ruby
  nix-env -iA nixos.rubyMinimal
  nix-env -iA nixos.ruby_2_4
  nix-env -iA nixos.ruby_2_5
  nix-env -iA nixos.ruby_2_7
[11:53:15] jon@jon-workstation ~/projects/nixpkgs (master)
$ nix-shell -p ruby
these paths will be fetched (3.80 MiB download, 17.93 MiB unpacked):
  /nix/store/ngc0m5s0qrfv821h74nwscvia64iwmq2-ruby-2.6.5
copying path '/nix/store/ngc0m5s0qrfv821h74nwscvia64iwmq2-ruby-2.6.5' from 'https://cache.nixos.org'...

[nix-shell:~/projects/nixpkgs]$ ruby --help
Usage: ruby [switches] [--] [programfile] [arguments]
...
[nix-shell:~/projects/nixpkgs]$ exit
[11:54:30] jon@jon-workstation ~/projects/nixpkgs (master)
$ ruby --help
The program ‘ruby’ is currently not installed. It is provided by
several packages. You can install it by typing one of the following:
  nix-env -iA nixos.jruby
  nix-env -iA nixos.logstash6-oss
  nix-env -iA nixos.logstash7-oss
  nix-env -iA nixos.ruby
  nix-env -iA nixos.rubyMinimal
  nix-env -iA nixos.ruby_2_4
  nix-env -iA nixos.ruby_2_5
  nix-env -iA nixos.ruby_2_7

you can add or remove packages from a given environment without affecting your global system.

that there exist another important issue with Tor.

Fwiw, I’m not seeing this issue with a missing bundled tor binary on my system:

$ file ~/.walletwasabi/client/tor/Tor/tor
/home/user/.walletwasabi/client/tor/Tor/tor: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.26, stripped

This is with wasabi 1.1.9.2, which is the latest version that’s functional for me at time of writing. Did you notice your issue around a missing bundled tor binary with that version of wasabi, or on another version?

There is also this issue about wasabi not finding the tor executable in NixOS.

Right, I filed that issue as well. Unless I’m mistaken, the “no such file or directory” errors seen there are about not finding hwi-related files though, not Tor: https://github.com/zkSNACKs/WalletWasabi/issues/2289#issuecomment-532969715

(to complicate matters slightly, there were indeed some Tor-related issues around connectivity for that issue, but that ended up being irrelevant to the “no such file or directory” errors, see https://github.com/zkSNACKs/WalletWasabi/issues/2289#issuecomment-534207288)

Apparently NixOS is a bit different that other linux distros and we have to decide if we want to support it officially because it will require further investigation, changes and testing specially focused on this distro.

Right, as a reference for others, the historical issue that led to the current state of the Wasabi project not officially supporting the community-provided Nix packages is here: https://github.com/zkSNACKs/WalletWasabi/issues/2130

I have checked the wasabi package for NixOS and looks okay, it is maintained by the NixOS community and we could, alternatively, let them to deal with this if they really want to support Wasabi.

Right, it is in my (unofficial, since I’m not in maintainers for this specific package yet it seems) role as NixOS package maintainer that I filed this issue; I’m trying to build a version of Wasabi that I can use: https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/blockchains/wasabiwallet/default.nix. 😃

I mean, I think they would need to patch Tor with patchElf and invoke wasabi with the appropriate LD_LIBRARY_PATH paths (this is what i suspect is making Avalonia to fail, but i am not sure).

Via the info above, I think that this shouldn’t be necessary, since the Wasabi Tor integration seems to work well in my usage on NixOS to date with version 1.1.9.2 and earlier. We can look into the “missing bundled tor binary” issue you reported above independently, if it can be reproduced.

I installed a NixOS 19.09.1686 in a VM and followed the instructions provided by @mmahut . It failed immediately with the exact same error:

Clearly there is a problem with Avalonia here but it is also important to see that there exist another important issue with Tor.

[demo@nixos:~]$ ls .walletwasabi/client/tor/Tor
libcrypto.so.1.0.0  libevent-2.1.so.6  libgmp.so.10  libssl.so.1.0.0  libstdc++  tor

[demo@nixos:~]$ ls -l .walletwasabi/client/tor/Tor                
total 6716
-rwxr-x--- 1 demo users 2282608 Jan  1  2000 libcrypto.so.1.0.0
-rwxr-x--- 1 demo users  371064 Jan  1  2000 libevent-2.1.so.6
-rwxr-x--- 1 demo users  622427 Jan  1  2000 libgmp.so.10
-rwxr-x--- 1 demo users  486600 Jan  1  2000 libssl.so.1.0.0
drwxr-x--- 2 demo users    4096 Dec 29 04:07 libstdc++
-rwxr-x--- 1 demo users 3102224 Jan  1  2000 tor

[demo@nixos:~]$ ./.walletwasabi/client/tor/Tor/tor
bash: ./.walletwasabi/client/tor/Tor/tor: No such file or directory

NixOS cannot find tor even when it is there. I found that this same problem is affecting other projects too and this is a relevant answer in StackOverflow:

You can’t usually run binary files in NixOS, they will either need some environment variables set or to be patched with patchElf.

Source: https://unix.stackexchange.com/questions/91577/cannot-execute-binary-in-nixos-no-such-file-or-directory

There is also this issue about wasabi not finding the tor executable in NixOS.

Apparently NixOS is a bit different that other linux distros and we have to decide if we want to support it officially because it will require further investigation, changes and testing specially focused on this distro.

I have checked the wasabi package for NixOS and looks okay, it is maintained by the NixOS community and we could, alternatively, let them to deal with this if they really want to support Wasabi. I mean, I think they would need to patch Tor with patchElf and invoke wasabi with the appropriate LD_LIBRARY_PATH paths (this is what i suspect is making Avalonia to fail, but i am not sure).

When looking at something else, I came across https://github.com/NixOS/nixpkgs/pull/73262/, which helped a bit in demystifying how dotnet builds works on NixOS for me (I’m not a dotnet dev and also fairly new to NixOS). That PR, which is unmerged at time of writing, makes it possible to use different dotnet-sdk versions for different Nix packages, and shows that the only supported dotnet-sdk version on Nix before that is 2.2.

Since Wasabi uses dotnet-sdk 3.1 now, I was thinking that this could be related to the current issue.

However, trying out the PR and switching Wasabi to 3.1 still leads to the same Avalonia-related NullReferenceException as before. Also, dotnet-sdk isn’t actually used to build the wasabiwallet package from source, but rather it is (afaict) required to patch .so files and the wassabee binary that the Nix package fetches from the released tarball, as seen in the wasabiwallet package.

In short, I’m not sure if the dotnet-sdk 2.2 vs 3.1 thing is relevant here. Perhaps @mmahut can shed some light on this as the original author and maintainer of the wasabiwallet Nix package? At any rate following this issue might be useful, so we can figure out how to proceed with Wasabi on Nix.

I’d want to avoid this becoming a drain on Wasabi developers, since you made it clear back in https://github.com/zkSNACKs/WalletWasabi/issues/2130 that the project is not interested in officially supporting Nix packaging at this time.

It seems the this and https://github.com/zkSNACKs/WalletWasabi/issues/2836 are not the same, the other one is resolved.

Version check hangs for me too, so that’s unrelated. Opened an issue for that: https://github.com/zkSNACKs/WalletWasabi/issues/2857