wasmi: Cannot compile a Wasm module from a called host function using the same `Engine`

The issue is quite similar to #567 But instead of calling a WASM function we try to

  • create another WASM module using the same Engine or
  • wrap Func::wrap host function for while preinstantiating already created WASM module
         lock                                 deadlock
host ------------> WASM #1 ------------> host ------------> create WASM module or wrap `Func::wrap` host function for already created WASM module

Would it be possible to fix this?

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 34 (21 by maintainers)

Commits related to this issue

Most upvoted comments

@Robbepop fair enough. All your arguments sound reasonable indeed. Thanks for your kind support. Waiting for the proper fix 😃

I just merged https://github.com/wasmi-labs/wasmi/pull/1116 which allows to instantiate a Wasm module inside a called host function which kinda reminded me of this long-standing issue and gave it some thoughts.

This is mostly about making the CodeMap type internally synchronized to be efficiently usable as &self in all cases necessary without the needs of Mutex or RwLock.

The 4 most important operations on a CodeMap are the following:

  1. Access the Wasmi bytecode instructions of a pre-compiled Wasm function.
    • This is the hot path and needs to be as fast as possible.
    • The current implementation uses exactly one atomic operation per pre-compiled access.
  2. Compile an uncompiled Wasm function.
    • This is only relevant for lazy compilation and usually cold-path because this runs at most once per Wasm function.
    • The current implementation already uses several atomic ops and is still fast enough.
  3. Populating the CodeMap with a bunch of uninitialized Wasm functions.
    • The main trick is to populate multiple Wasm functions at once in order to minimize locking.
    • The main challenge of this issue is to implement this in a way that allows for very efficient 1)
    • The problem with this is that it might invalidate pointers due to resizing of the underlying buffer that stores all the compiled and uncompiled Wasm functions.
  4. Initializing an uninitialized Wasm function as either compiled or uncompiled.
    • Since this can act in-place per Wasm function we should be able to use today’s atomics mechanisms for it.

No, unfortunately #694 only provided the foundation for this feature to (maybe) be effective. Implementation needed to confirm.

Sorry for the confusion. Indeed my messages were a bit misleading.

Am I right to understand that all “static” states of instance will be in the Store?

Yes, as per the Wasm specification the Store is the owner of all instantiated Wasm related state. An instance only stores references into its associated Store. The Store even owns all Wasm instances associated to it.

Thank you @Robbepop for your swift reply.

Per your comment, I’d like to add more context for our use case.

A little bit background on us, we’re from RadixDLT and we’ve integrated wasmi v11.0 into RadixEngine/Scrypto about one year ago, and we’ve got this problem when attempting to upgrade to v23.0.

In RadixEngine, we execute user-provided WASM code dynamically. In order to efficiently execute them, we cache compiled WASM modules. With the new wasmi architecture, where wasmi module is coupled with engine for performance reason, the idea is broken, because we’re not allowed to either

  • A) Add more modules to the engine (in host function during callback)
  • or B) create new instance of existing module (with separate Store for state isolation).

I believe this is applicable for many other blockchain projects as well.

Sincerely hope you can help prioritise this.