2019-04-23 16:00:00
Running WebAssembly and WASI with .NET
For the moment, and for lack of a better name, I'm calling this project "wasm2cil". In a nutshell, it's a .NET compiler for WebAssembly/WASI modules.
I might be misusing the term "compiler". Maybe this should be called a "transpiler" or a "translator". Whatever you call it, what this project does is take a WebAssembly module and convert it into a .NET assembly on disk. The resulting DLL is not interpreted -- it contains the same functions as the WebAssembly module, translated from Wasm instructions to CIL instructions.
For example, I can start with the C code for SQLite and its shell application, compile it to Wasm with Clang, "transpile" it to a .NET assembly, and then run it with .NET Core. The result is the SQLite shell, entirely as managed code, with no pinvokes involved.
All of this is very much a work in progress.
But the current status is that some basic things do work. For example, the ".NET-ified SQLite shell" can read and write SQLite database files that seem to be fully interoperable with "regular" SQLite builds.
Pardon our dust
Like I said, this isn't ready for anybody to try using for anything serious. Just off the top of my head, the core stuff has the following problems:
Some of the WebAssembly instructions aren't implemented yet.
A few WebAssembly instructions are implemented with slow/dreadful placeholders.
Quite a few WASI methods are not implemented yet.
The WASI spec itself is still in flux and will probably change before it is finalized.
My WASI implementation currently only works on .NET Core (because it uses things like Stream.Read(Span<byte>) for performance).
The main thing I use to drive this project forward is attempts to get real C code (compiled by Clang 8) working. So, if a certain WebAssembly instruction is still throwing NotImplementedException() in my code, that's probably because I haven't seen it come out the back end of Clang yet.
And the SQLite stuff has problems of its own. Most notably, the VFS (platform layer) I'm using is based on the "demo" VFS, which is not full-featured.
So this blog post is NOT me announcing the release of a polished project that is ready for use. Rather, this is like I started remodeling my living room. And then I waited until everything was torn apart and covered with dust. And then I invited people to watch me continue working. This is messy, and it will continue to be so for a while.
Under the hood
(Clang/LLVM 8 with wasi-sysroot) = Awesome
Initially, I wasn't thinking about WASI at all. I just wanted to get Wasm instructions converted over to CIL instructions. My early test modules didn't have any external dependencies. I started with things like:
int foo(int x)
{
return x + 42;
}
As things progressed, I got more ambitious and started to work on things that needed a few library functions. The first real C program that worked was miniray.c, a little ray tracer. Its only deps are floating point math plus putchar() and puts(). (And since the puts() call was a string literal, I replaced it with 15 putchar() calls so I would only have one IO function to implement.) Anyway, I remember the moment when I first got my CIL version to output the same image as the regular native build. We compiler writers often find joy in two files with differing provenance and identical contents.
Eventually I decided to dive in and try to get SQLite to work. The key here is wasi-sysroot, an implementation of libc that only needs WASI for its "system calls".
Using Clang 8, instead of building like this:
clang --target=wasm32 ...
Build with this:
clang --sysroot=/path/to/wasi-sysroot --target=wasm32-unknown-wasi ...
The result is that a whole bunch of functionality suddenly started working. For example, I had been figuring I would have to write malloc() and free(), but this WASI-based-libc includes that. And strcmp() and memset() and printf() and so on.
Viewing my "clang-8/wasi-sysroot/wasm2cil" pipeline as a C compiler gives me lots of ways to test it. For example, on GitHub I found a C compiler testsuite with 220 test cases. My "compiler" passes 219 of them, and the one failing test does not appear to be related to a problem in my code.
(BTW, I was excited to get these results, but I also understand that the serious commercial test suites (like Plum Hall) are far more comprehensive.)
An alternative to pinvoke
Why am I doing this? Several reasons.
For one, I just wanted the experience. I haven't done anything so compiler-ish since the 68000 C compiler I wrote back around 1991. (Anybody else remember the Mac SE/30?) I've been feeling the need to get a better understanding of .NET at the CLR level, and getting my hands really dirty is the best way to do that.
I am also generally interested in the possibilities of compiling other languages for the CLR. And using WebAssembly as an intermediate step is one way to do that. I maintain SQLitePCL.raw (a low-level wrapper around SQLite), so when it comes to the issues of mixing unmanaged code with .NET, I have suffered more than most. Wouldn't life be grand if we could compile C code into a .NET assembly?
And Rust too. I want to be able to write stuff in Rust and (more easily) consume it from .NET.
Performance
I haven't done any careful benchmarking, so I'm not yet ready to say anything quantitative. However, I will say informally that the performance of C code converted to CIL has NOT been disappointing.
Links and Acknowledgments
I plan to write more about this project as I continue moving it forward. For now, I just want to give mention to the various resources I have found helpful:
The WebAssembly spec is not exactly easy to read, but it's a very high quality piece of work:
https://webassembly.github.io/spec/
WASI means "WebAssembly System Interface". It's all about the use of WebAssembly outside the browser, and IMHO, it's going to be huge. Here's the blog entry from last month where this was announced:
https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/
wasi-sysroot is here:
https://github.com/CraneStation/wasi-sysroot
I have drawn inspiration and information from other projects in the Wasm-to-dotnet arena, especially this one:
GitHub: RyanLamansky/dotnet-webassembly
The back end I'm using (to write the actual .NET assembly files) is Mono.Cecil:
https://github.com/jbevain/cecil
The ray tracer I mentioned:
https://github.com/mzucker/miniray
The C testsuite I mentioned:
https://github.com/c-testsuite/c-testsuite
The WebAssembly binary toolkit has been very handy:
https://github.com/WebAssembly/wabt
Open source
This project is on GitHub, open source, Apache License v2: