2019-06-24 09:00:00

Dynamic loading of native code with .NET

I maintain SQLitePCLRaw, a .NET wrapper for SQLite. In one sense, this project is not terribly unusual. Plenty of other packages exist to provide a .NET-friendly way to access a native library. For example, SkiaSharp is a .NET wrapper around the Skia graphics library.

But even in this category, SQLitePCLRaw faces challenges that are uncommon, if not unique.

• There is more than one notion of "the SQLite library". We have plain SQLite. And then there is SQLCipher (derived from SQLite, developed by Zetetic). And Windows 10 has a SQLite library built-in (winsqlite3.dll). Android has one too, and then in version N they decided that people can't use it anymore. Apple provides a SQLite library with iOS and MacOS. People have custom builds of SQLite so they can configure it with compile options. All these SQLite libraries can be different versions, and can be compiled with different features. And SQLitePCLRaw tries to work with all of them.

• SQLite (in all its various forms) is extremely ubiquitous. It runs on everything from servers to embedded devices, and people use it for all kinds of things. .NET has become very diverse as well. We have .NET Framework, .NET Core, Mono, Xamarin, Unity, Windows, Mac, Linux, Windows Phone, UWP, and so on. Basically, SQLitePCLRaw tries to support .NET in all its forms.

So the number of scenarios in play here is like a matrix with N flavors of SQLite as the rows and M platforms as the columns. As an offhand count, I can name enough items to say that N is 9 and M is 11, so there are roughly 100 configurations to worry about. Not every matrix cell is special, so that might be overstating things a little, but it's the right ballpark.

DllImport

The most common way of calling native code with .NET is P/Invoke. This is done with an extern method definition that has a DllImport attribute which specifies the name of the unmanaged library in which it should be found. For example:

[DllImport("libc.dll")]
public static extern int abs(int x);

This says "In the shared library 'libc.dll', find a function called 'abs' and treat it like it returns an int and has one parameter of type int." Note that the actual signature of the unmanaged function must match the method definition, and nothing verifies this for you. So for example, if the parameter for abs() were actually a pointer, you could expect Bad Things to happen when you call it with the P/Invoke definition above.

It is also worth observing that the trivial example above does not show much of the power of P/Invoke. Here's something a bit more complicated:

[DllImport("libc.dll")]
public static extern int strlen(byte[] a);

This says "In the shared library 'libc.dll', find a function named 'strlen' which returns an int, and takes one parameter, and the actual type of the parameter is a pointer, but from the .NET side, let's pretend that the function takes an array of byte, and just do the right thing, mkay?" So, when you call this method from C#, P/Invoke will take the byte[] and pin it, to prevent the garbage collector from moving it around while the unmanaged function has it. And then when the native function returns, P/Invoke will automatically un-pin it.

P/Invoke can do all kinds of stuff to bridge the gap between managed and native code. We call that stuff "Marshaling".

As powerful as P/Invoke is, I have one major complaint with it, and it goes back to the DllImport attribute. Because DllImport is, well, an attribute, its parameters have to be constant. So, the name of the unmanaged library has to be hard-coded at compile time. But in my world, SQLite is running around using more names than Jason Bourne.

Like DllImport, but more dynamic

For several years my solution to this problem has been to take the P/Invoke part of my code and compile it multiple times, once for each name I need to give to DllImport. I refer to the instances of this module as "providers", and they are somewhat like plugins. The core of SQLitePCLRaw needs one of these providers to be registered.

Anyway, having all these providers is inelegant. It makes the build system awkward. It increases the number of assemblies and packages. For a long time I've been wanting to rework this code to use dynamic loading instead of DllImport. I envisioned writing a Do-It-Yourself equivalent for DllImport, but without using attributes, so the library name wouldn't have to be hard-coded.

The starting point for me was to think about how DllImport is implemented. Conceptually, I pictured it like this:

MyDelegateType DoThatDllImportThing(string libraryName, string functionName)
{
    string path = FindTheLibrary(libraryName);
    IntPtr hLibrary = LoadTheLibrary(path);
    IntPtr hFunction = LoadTheFunction(hLibrary, functionName);
    return CreateTheDelegate(hFunction);
}

I figured FindTheLibrary() would be easy, because it's just path stuff.

And LoadTheLibrary() is just a LoadLibraryEx() on Windows and dlopen() everywhere else. Similarly, LoadTheFunction() is either GetProcAddress() or dlsym(). So those are not a problem.

But CreateTheDelegate() would be a lot of work, because I would have to implement all that magic that P/Invoke does. Well anyway. It would be worth it.

Skipping ahead to today, I do now have dynamic loading implemented for SQLitePCLRaw 2.0 (a prerelease of which is now available on nuget.org), but things didn't go at all like I thought they would.

It turns out that my expectations about difficulty were almost completely backwards.

CreateTheDelegate() is already provided by the platform. Once you have the function pointer, all the stuff I thought would be hard is actually done by Marshal.GetDelegateForFunctionPointer().

But FindTheLibrary() is not "easy". In fact, implementing this in a cross-platform manner appears to be impossible. As far as I can tell, there are some .NET platforms where it just can't be done. FindTheLibrary() (or something like it) must exist, because DllImport uses it, but it is not public.

Side Rant: People think the hard part of software is because of tricky algorithms or difficult concepts. But very often, software is hard because you're trying to do something the platform won't let you do, because the platform only provides a higher-level interface, which, ironically, was intended to make things easier. For me, I almost always want the lower level APIs. You want to give me an HTTP library? Fine, but I still need sockets.

Success and failure

All I want here is for the tooling to support a way for me to include a dynamic library (dll | so | dylib) and put it somewhere such that I can find its path at runtime. But every .NET implementation does this a little differently.

Xamarin iOS and Android are both kinda weird (in different ways). In part, this is because of constraints of the underlying platforms (such as static linking on iOS), but it is also apparent that both of them were just designed to support DllImport (which is understandable). If either of these Xamarin platforms have a documented/supported way of doing what I want, I have thus far been unable to find it.

A bit surprising is that .NET Core 2.x is one of the most difficult situations. It stores unmanaged libraries in a folder called runtimes and then in a subfolder named for the RID, but (1) at debug time, this is in the packages directory, which is weird and not discoverable relative to the executing assembly, and (2) there is no public API to get the RID. I could try to figure out the RID myself, but that approach looks bug-prone.

Strangely, the .NET Framework is one of the easier cases, because it doesn't try to do anything in this area. It just lets me do. So I can provide a nuget targets file that copies the dynamic libraries to the output directory at build time. And then I use GetExecutingAssembly() and get its location and from there I can find the files I need.

Despair

The frustration of trying to solve this problem has driven me to some weird places.

"It's a %*($^#@ file and I just want it stored next to my executable. Why can't I do that? The tooling does it for images. Why can't I do this for an arbitrary file? Hey, maybe this would work if I just change the suffix from .so to .png?"

And I am apparently not alone in this. In my search for answers I found a repo provided by the Quicken Loans team (see https://github.com/RockLib/RockLib.EmbeddedNativeLibrary). In a nutshell, they package the native library as an embedded resource inside an assembly and then at runtime they load the resource and write it to disk and then load it.

Along those same lines, I've come really close to writing code to take a (dll | so | dylib) and encode it as base64 and dump it out as a string literal in C# source file.

All this silliness is also part of my motivation for exploring various ways of compiling C (and things like it) for the CLR. I've had early success compiling WASM to MSIL, and I've made significant progress doing the same for LLVM bitcode. If I could have SQLite compiled as a .NET assembly, all this pain would go away. Granted, it would be replaced by new and different pain, but I'm ready to welcome that.

But alas, .NET Core 3.0 is arriving with a breath of fresh air.

System.Runtime.InteropServices.NativeLibrary

I first learned about this new API from Brice Lambson of the Entity Framework Core team. Basically, it provides FindTheLibrary(), LoadTheLibrary(), and LoadTheFunction() in a cross-platform way. It is pretty much exactly what I need, a way to load unmanaged libraries without the limitations of the DllImport attribute.

As I write this, .NET Core 3.0 is in preview, so the docs on this API are a bit sparse. But after some experimentation and a bit of "Use The Source Luke", I've been able to get things working nicely.

One tricky part was that NativeLibrary.Load() has two overloads.

public static IntPtr Load(
    string libraryPath
    );

public static IntPtr Load(
    string libraryName, 
    System.Reflection.Assembly assembly, 
    Nullable searchPath
    );

The second call (the one that takes a "libraryName") is the one that has knowledge about where to look for unmanaged libraries. The simpler first overload wants a "libraryPath", and seems to be just a trivial wrapper around LoadLibraryEx/dlopen, the thing to use when you already know the path of the (dll | so | dylib) you want.

Looking forward

So why am I excited about this API when I can only use it on .NET Core 3.0?

Mostly, because of the .NET 5 announcement. I'm assuming that when .NET 5 ships (in late 2020), the Xamarin platforms will support the NativeLibrary API. Eventually, I will be able to use NativeLibrary everywhere and get rid of DllImport completely.

In the meantime, I'm using the NativeLibrary API as the pattern for implementing similar functionality for other platforms whereever possible. For example, for .NET Framework support, I have implemented my own clone of (a subset of) NativeLibrary. In the end, I still have to ship several of the old Dllimport providers with hard-coded names, but still, dynamic loading killed off some of the complexity, resulting in fewer packages and a simpler build.