2021-01-19 12:00:00

Preview: The SQLite Llibrary as a .NET assembly

The latest pre-release of SQLitePCLRaw contains a new bundle package called SQLitePCLRaw.bundle_cil. This package is like SQLitePCLRaw's other "bundle" packages except that it involves no P/Invoke and no platform-specific shared libraries. Instead, bundle_cil provides the SQLite library as a pure .NET assembly which was compiled by Llama.

What is Llama?

• A toolchain for compiling "other languages" for .NET

• My side project

Llama includes a compiler that translates LLVM bitcode into CIL (Common Intermediate Language), aka MSIL, aka the instruction set of .NET assemblies.

For a bit more info on Llama, see my other blog posts.

How was this CIL build of SQLite made?

I compiled sqlite3.c with clang, just as one normally would, except I added the -emit-llvm flag to make it generate bitcode instead of the usual .o/.obj file.

Then I run Llama's compiler on sqlite3.bc, which results in sqlite3.dll, a .NET assembly. I also needed to provide various dependencies (handwave, handwave).

So will this new bundle be part of SQLitePCLRaw 2.0.5?

Sorry, no, Llama isn't "ready" yet. I've included bundle_cil in this 2.0.5 pre-release so folks can see my progress, but it will be excluded when I release 2.0.5 final. Hopefully bundle_cil will graduate later.

What is the status of this then?

Llama is still at "proof of concept" stage. Lots of things are broken or incomplete.

And how is that "proof of concept" going?

Quite well. The reason for this pre-release and blog entry is that Llama just passed a big milestone:

The CIL build of SQLite now passes all of the following:

• the test cases for SQLitePCLRaw

• the 600+ test cases for Microsoft.Data.Sqlite

• the 22,000+ SQLite test cases for Entity Framework Core (in EFCore.Sqlite.FunctionalTests)

... on Windows. On Linux, things are close but not quite there yet. I haven't tried on Mac yet.

What versions of .NET does Llama support?

Currently, Llama only works with .NET 5.0.

How is the performance of Llama-compiled code?

I haven't done any actual benchmarks. I assume that things are at least a little bit slower. I see lots of places where I know I could improve performance. I believe that Llama-compiled code will (eventually) be competitive on performance, but I currently have no evidence or measurements to support that belief.

How was the CIL build of SQLite integrated with SQLitePCLRaw?

SQLitePCLRaw uses an architecture wherein the details of the integration with the native SQLite library are encapsulated inside a "provider".

For the CIL build of SQLite, I added a new provider. It simply calls stuff in the sqlite3.dll assembly instead of using P/Invoke.

Nothing in SQLitePCLRaw above the provider API boundary had to be changed, so bundle_cil can be a drop-in replacement for bundle_e_sqlite3.

Will Llama work for other libraries in C besides SQLite?

That is certainly the goal.

In some ways, SQLite is much easier than other libraries. It is an incredibly high quality project and it is very cross-platform.

Also, SQLite is provided as an "amalgamation", a single C source file that contains the entire library. In this case, the easiest build system is no build system at all. As I've explored building other C-language projects with Llama, getting their build system to create an LLVM bitcode file is one of the trickiest parts.

OTOH, SQLite offers its own challenges. It has minimal dependencies, but those do include some very tricky things such as shared memory, thread synchronization primitives, and file locking.

So is Llama just for C code?

No, I hope to get Llama working for several other languages which have LLVM-based compilers. Llama's support for Rust is actually a bit further along than C. I am eager to try and get Llama working with Swift.

Why is it called bundle_cil?

I don't like the name, but I haven't thought of anything better yet.

How do I try it?

If you are using SQLitePCLRaw or some other wrapped that is built on same (such as sqlite-net or Microsoft.Data.Sqlite or Entity Framework Core), you just need to replace your bundle_e_sqlite3 package reference with bundle_cil, and set the version to the pre-release:

    <PackageReference Include="SQLitePCLRaw.bundle_cil" Version="2.0.5-pre20210119130047" />

If you do give this a try, please let me know how it goes.