Home About Eric Topics SourceGear

2019-08-14 15:00:00

Nullable references in C# 8.0

Let's start with some background. Recall that the CLR supports value types and reference types.

Value types include primitive types like int and double, but also struct types:

struct Point
{
    public double x;
    public double y;
}

Point p1 = new Point { x = 3, y = 5 };
Point p2 = p1; // copy the whole struct
Point p3 = null; // error -- not allowed

Like all value types, structs have "copy" semantics (shown in the p2 = p1 assignment above) and they do not allow null as a value.

But if we simply change the definition keyword from struct to class, then variables of type Point become references, and the semantics are different:

class Point
{
    public double x;
    public double y;
}

Point p1 = new Point { x = 3, y = 5 };
Point p2 = p1; // multiple refs to same object
Point p3 = null; // reference to nothing

Now when we assign p2 = p1, the object is not copied, but rather, we end up with two references to the same object.

Also a reference can refer to nothing, represented with the value null.

And this can cause problems.

Problems with nulls

Consider the following (pre C# 8.0) method, a member of some sort of class which has fields _x and _y that represent a location. The method is used to determine the distance from its coordinates to a given point, represented by the class form of Point shown above.

double DistanceTo(Point p)
{
    var dx = _x - p.x;
    var dy = _y - p.y;
    return Math.Sqrt(dx * dx + dy * dy);
}

What should we do if parameter p is null? In this implementation, a NullReferenceException will be thrown at runtime. We want to do better. But what options do we have?

The first thing that comes to mind is to check for null before we use the reference. But then what?

This function is supposed to return a value representing the distance between two points. It MUST either return a value or throw an exception. Inserting a null check merely offers us a chance to return a different value or throw a different exception.

To throw a different exception, I suppose we could do this:

double DistanceTo(Point p)
{
    if (p != null)
    {
        var dx = _x - p.x;
        var dy = _y - p.y;
        return Math.Sqrt(dx * dx + dy * dy);
    }
    else
    {
        throw new InvalidArgumentException();
    }
}

But that's really not much better than just letting the NullReferenceException happen.

And the only option I see for returning a different value is something like this:

double DistanceTo(Point p)
{
    if (p != null)
    {
        var dx = _x - p.x;
        var dy = _y - p.y;
        return Math.Sqrt(dx * dx + dy * dy);
    }
    else
    {
        return 42 * new System.Random().NextDouble();
    }
}

Which would be absurd.

So the truth is that once we have arrived in this method with a null reference, it's too late to do anything useful. If we want to "do better", we need to prevent that from happening.

An ounce of prevention...

Ideally, any attempt to call DistanceTo() with a null value should be caught at compile time:

double DistanceTo(Point p)
{
    // p should never be null here
    var dx = _x - p.x;
    var dy = _y - p.y;
    return Math.Sqrt(dx * dx + dy * dy);
}

// ... somewhere else ...

   // fail HERE, at compile time, not at run time
   var dist = p2.DistanceTo(null);

We could accomplish that by checking every call to the method and making sure that the argument cannot be null. But that's the sort of tedious work that we should expect the compiler to help with. And C# (pre-8.0) doesn't have a way to express that p should never be null, so what do we do?.

This kind of problem is what motivated the design of the nullable references feature for C# 8.0. The designers of the feature probably continued by next asking themselves: "What kind of special syntax could we use to specify that a reference type should not be null?"

And before going very far down that line of thinking, they remembered that we already have a syntax for nullable value types. The syntax for that is to just append a ? to the type name:

int? x = null;

That's the inverse of what we need for reference types, which are already nullable, and we need a way to make them not so. But it's a syntactic pattern that is already in the language, and consistency is important.

Bottom line, C# 8.0 uses the same syntax for nullability of reference types that we have been using for value types:

string  x; // not nullable
string? y; //     nullable

And yes, that means that the meaning of a type declaration like string (without the ?) has changed.

Whoa, isn't that a massive break in compatibility? Actually no. In fact, although this feature looks like a huge breaking change, the entire thing was carefully designed to preserve backward compatibility.

First of all, this whole feature is turned off by default, and has to be explicitly turned on. Second, all it really does is generate warnings.

Beneficial warnings

For example, if you explicitly mark a reference as nullable and then dereference it, you will get a warning:

double DistanceTo(Point? p)
{
    // p can be null, so
    // the compiler will complain about this:
    var dx = _x - p.x;
    var dy = _y - p.y;
    return Math.Sqrt(dx * dx + dy * dy);
}

Note that the compiler uses flow analysis to figure out a lot of stuff. For example, it won't fuss at you about dereferencing that nullable reference if you check it for null first:

double DistanceTo(Point? p)
{
    if (p != null)
    {
        // no warning here
        var dx = _x - p.x;
        var dy = _y - p.y;
        return Math.Sqrt(dx * dx + dy * dy);
    }
    else
    {
        // TODO whatever
    }
}

In addition (and returning to the problem we initially set out to solve here), the compiler will give a warning if you do anything which might store a null value in a non-nullable reference:

double DistanceTo(Point p)
{
    // p is not nullable, so no
    // warnings here
    var dx = _x - p.x;
    var dy = _y - p.y;
    return Math.Sqrt(dx * dx + dy * dy);
}

// ... somewhere else ...

   // but HERE, the compiler will complain:
   var dist = p2.DistanceTo(null);

So there is cause for optimism here. This feature can help us catch null problems earlier, before much damage is done.

Cars and clocks

Before I go on, please permit me a brief digression to ramble about how cars are different from clocks.

A clock doesn't really expect me to know anything about how it works. Clocks can have lots of cool technology inside, but generally speaking, knowing how a clock works does not help me tell time any better.

In contrast, people who know how cars work "under the hood" have a generally better experience driving and owning a vehicle. I know this because I don't know much about cars, but I know folks who do, and their lives seem to have less misery than mine.

(And it's not just cars. I also have two lawn mowers, a string trimmer, a power washer, a garden tiller, an ATV, two generators, and a snow thrower. All of them have an internal combustion engine, and because I don't know much about how such things work, I hate them all.)

But to be fair, cars mostly do function for people who don't understand them. When I get in my car, I nearly always end up where I wanted to go.

But some things are even worse than cars.

I don't have a 3D printer, but I've heard they are basically unusable without gaining a fair amount of knowledge about how they work.

Returning to the world of software, another example is C# async/await, which really doesn't work well until you understand its internals.

Finally, and getting back to the subject at hand, I'm going to claim that the C# 8.0 nullable references feature is in this bucket as well. I reserve the right to change my mind later, but right now it looks to me like C# nullable references cannot be used effectively without knowing how they work "under the hood".

So how does the nullable references feature work?

It's all fake.

(kinda)

Annotations

Consider the following two methods:

void Foo(string s)
{
    // whatever
}

void Bar(string? s)
{
    // whatever
}

The first thing to understand is that at the CIL level, looking at the output of the C# compiler, these two methods have the same type signature. Both of them accept a parameter of type string. The code generated for these two methods will be the same. The only thing different about them is their annotations.

When Roslyn (the compiler) compiles these two methods, it will generate a C# attribute called Nullable, and it will use that attribute to annotate things.

And these attributes will make no difference in the behavior of the methods or in the code generated for them.

Consistency?

So now let us pause and observe that (as I mentioned earlier) the syntax of this feature was designed to be consistent with the way nullability is expressed for value types, but that consistency is very shallow.

In both cases, the presence or absence of the question mark is the difference between nullable or not:

// not nullable
int x;
string a;

// nullable
int? y;
string? b;

So the syntax is the same, but under the hood, things are very, very different.

For value types, nullability is implemented using the CLR type system. A type declaration like int? is just shorthand for System.Nullable<int>, which is an actual CLR type.

But for a reference type, a declaration like string? is just an annotation to help the compiler complain. There is no such thing as System.Nullable<string>.

More on the annotations

Earlier I said that the nullable references feature is turned off by default and has to be explicitly activated. This makes it sound like there is a switch that needs to be flipped, but actually it's more like two switches.

Relevant to the nullable references feature, the Roslyn compiler has two different "contexts" which can be activated. It may help to think of these contexts as "modes" which can be turned on or off.

These two things can be turned on or off independently, at the project level, at the file level, or even line-by-line.

The result of these annotations is a "nullability" for each relevant expression in the code, and there are four possible nullabilities:

Revisiting DistanceTo(null)

An important part of understanding how this feature works is understanding how to "defeat" it.

#nullable enable // turn on annotations
double DistanceTo(Point p)
{
    var dx = _x - p.x;
    var dy = _y - p.y;
    return Math.Sqrt(dx * dx + dy * dy);
}
#nullable restore // reset annotations back to project default

// ... somewhere else ...

#pragma warning disable nullable
   var dist = p2.DistanceTo(null); // let's tell a lie
#pragma warning restore nullable

In the code snippet above, the DistanceTo() method is being compiled with annotations turned on (see the #nullable directives), so the parameter will get attributes saying that it is not-nullable.

However, the call to DistanceTo() is compiled with the corresponding warnings turned off (see the #pragma warning directives), so the nullability feature isn't giving any benefit for that call site.

At runtime, DistanceTo() will end up throwing a NullReferenceException in this case.

Another (more convenient) way to bend the rules is to use the so-called "null-forgiving" operator:

#nullable enable // turn on annotations
double DistanceTo(Point p)
{
    var dx = _x - p.x;
    var dy = _y - p.y;
    return Math.Sqrt(dx * dx + dy * dy);
}
#nullable restore // reset annotations back to project default

// ... somewhere else ...

   Point q = null;
   var dist = p2.DistanceTo(q!);  // another way to lie

The intent of this operator is to provide convenient syntax for cases where Roslyn believes an expression might be null but we believe it cannot be, so we want Roslyn to "trust us and be quiet about it". This operator is unofficially known as the "damn it" operator.

Finally, it is worth noting that because all these nullability annotations are implemented in the compiler and not in the CLR's type system, another way to store null in a not-nullable thing is to do so from another language like F#. (Yes, given that F# has its own aversion to nulls, that's a poor choice of example, but you get the idea. There are dozens of niche languages that can compile for the Common Language Runtime, and any of them not based on Roslyn will probably offer no objection if you call DistanceTo() with a null argument.)

Big picture

So, the C# 8 nullable reference feature can't actually make any strong assurances about things being null or not. This raises questions.

At the time of this writing, it is best to think of this feature as taking a few steps down a possibly long road which is headed vaguely in the direction of "better null handling for C#". I don't think anybody knows exactly what things will look like at the place where the road ends. Experience will tell.

Even the folks at Microsoft don't seem to have a clear vision of this yet. Apparently the Microsoft libraries are only about 20% annotated, and that effort has produced a lot of questions, plus a number of enhancements.

For example, annotation of System.Collections.Generic highlighted that Dictionary<TKey,TValue> should have the ability to specify that TKey should not be a nullable type. So they added a notnull generic constraint (which looks like it'll be pretty handy).

I recommend this recent blog entry by Phillip Carter which gives mention to several other new-ish aspects of the feature that I am not going to discuss in detail here:

Bottom line

Personally, I look forward to using this feature more, and despite my whimsical use of words like "fake" and "lie", I am not trying to criticize the work of those who designed it. Given a requirement of preserving backward compatibility, the design looks solid to me.

I am merely saying that (1) it is critical to know what the feature will or will not do for us, and (2) this feature involves a lot of complexity, and there are costs associated with that.

But overall, I am a big fan of strict compilers and static analysis, so I see this feature as a great step.

Copyright 2001-2023 Eric Sink. All Rights Reserved

If you are in need of contract software development services, I would appreciate your consideration of SourceGear.