An abstraction is a boundary with two sides. On the top side, the
abstraction presents a simplified view. Below, there is
something more complex and more real. The purpose of the abstraction is to
obscure what is really going on.
The world hidden underneath an abstraction is quite likely to be yet another
abstraction. In fact, it is typical to have many abstractions stacked
together, each one attempting to present an illusion which is even further from
the truth. If you stack them up vertically, the ones at the bottom are
more real than the ones at the top.
This is what programmers do. We build piles of abstractions. We
design our own abstractions and then pile them up on top of layers we got from
somebody else. Abstractions can be great. We use them because they
save us a lot of time. But abstractions can also cause lots of
problems. They're never perfect, as Joel Spolsky explains in his excellent
article on "The Law of
But you can't have the benefits of an abstraction without its
risks. We need to make wise decisions about our piles of
abstractions. I'll start by offering three rules to keep in
Using somebody else's code can save a lot of time. For example,
a special GUI component allows you to program at a higher level of
abstraction. Why write your own?
You have to remember that you are accepting a tradeoff. By using
somebody else's code you are inheriting somebody else's bugs. Often you
are accepting risks that are not under your control.
Writing in Java is faster than
writing in C. But C code runs much faster than Java code. It's a
tradeoff that you cannot avoid. All you can do is make the right
Abstractions increase overall
The goal of each abstraction is to
decrease complexity by presenting a simplified view of something else.
However, by the time you pile them all up, you've got a lot of complexity
which you may have to deal with. In fact, the more layers of abstraction
you have, the more complexity you've got involved.
A Really Tall
Let's work through an example. Suppose that I am working from my home
and I am ready to checkin some really important code changes to a source code
file. I'm using the SourceGear Vault client to connect to our server back
at SourceGear's main office. Below is a list of [almost] all the
abstractions which are in play.
The version control system itself is an abstraction. It presents our
users with concepts like Check Out, Check In, Label, Branch, Pin and
Share. This layer is obviously very important, since it's the only one
we can charge money for.
Vault is written entirely in C#, which is a very nice abstraction
indeed. From C# we get classes, objects, methods, strings,
looping constructs, logical operators, and the ability to attach names to
C# runs on the Common Language Runtime, which is a huge
abstraction. In fact, if we all weren't so worried about comparing .NET
to Java we would be calling the CLR a "virtual machine", which it
The CLR is written in lower level languages like C++ and C.
C++ is implemented by compiling it to x86 assembler code. We've taken a
big, big jump here. Compared to C++, assembly language doesn't feel very
abstract at all.
Did you think Assembly was the lowest level of programming? Certainly
not. Each x86 assembler instruction is a little program written in an
even lower level language called microcode.
gates. Microcode is implemented by circuits which provide
logical gates, including NOT, AND, OR, and NAND.
- Transistors. Logical
gates are implemented by transistors, an electronic component with three wires
sticking out of it.
The .NET framework gives us "collections" we can use to manage memory in
From the realm of OOP we get "objects", self-contained pieces of data which
are bound to the operations which can be performed on that data. Very
This is a big one. Because the .NET Common Language Runtime has a
garbage collector, we can create objects and know that they will automatically
be destroyed later when we are done with them.
In reality, memory has to be explicitly requested and released from
the operating system. Each chunk of memory is identified by a
Memory. This layer gives us another important illusion:
There is more memory available than we actually have.
Random Access Memory is itself an abstraction. Transistors don't
really remember anything. Furthermore, the notion of a bit doesn't
really exist. We simply assign conventions. When a
wire is at 5V, we call it a one. When it has no voltage on it, we call
it a zero. Collect a few hundred million of these in one place and
you've got a DIMM. (Actually, it's 3.3V nowadays,
The Check In
The Check In dialog has a button on it. When the user presses this
button, the Check In operation will commence. But the button itself is
an abstraction. It is designed to simulate the concept of a physical
button like you might find on your microwave or TV. No such button
really exists. Windows Forms provides this abstraction.
Windows Forms is a layer of abstraction which is built on the Win32 API
underneath. The button is actually a window with its own WndProc.
.NET tries to hide this world, but it's definitely still there. One of
the glaring "leaks" in the Windows Forms abstraction is the absence of the
Win32 ScrollWindow() call.
The button is actually drawn using graphics primitives from GDI. It
doesn't just magically appear. It needs to be drawn using things like
DrawRect, fonts and colors.
GDI contains primitives like DrawLine, but these are implemented in terms of
pixels. Graphics primitives are actually not quite so primitive.
If you think line drawing is easy, look up Bresenham.
Card. The pixels are actually an abstraction presented by
a video card.
The monitor presents the illusion that all those pixels are organized into
pictures and images.
I stop whenever I get to Physics or Chemistry. For my purposes, light is
real, not an abstraction.
the Vault Client:
We wrote this layer as part of Vault. It contains all the windows and
dialog boxes necessary to create the Vault GUI client.
This layer is a big part of Vault. It contains basic non-GUI primitives
which are necessary to write a Vault client. Create an instance of the
ClientInstance class. The methods on this class will communicate with
the Vault server and simultaneously keep your local working folder updated.
The previous layer actually calls VaultClientNetLib to communicate with the
Vault server. This layer is fairly thin. It is mostly a wrapper
around the Proxy Class.
Class. This important layer is generated by Visual Studio
.NET. It presents the illusion that the XML Web Service on the
Vault server is actually a C# class.
When a call is made through the proxy class, the parameters for that call are
bundled up in SOAP
format. This format presents the concept of a method invocation message.
SOAP is built on XML, a syntax framework for representing data.
The SOAP message is transported to the server over HTTP, the
networking protocol on which the Web is built.
The Vault user types the name of the server, but that name isn't
really useful. It has to be converted to an IP address before real
network communication can take place. The Domain Name System is used to
look behind the abstract name and get the actual machine address.
The Secure Sockets Layer offers the illusion that communication over the
Internet can be private. This layer tries to look just like a regular
socket, except all of the data is encrypted as it passes through to/from the
This layer is a great abstraction. Sockets present us with the illusion
of connections and the ability to send and receive data between endpoints.
The basic illusion of TCP is the idea that packets of data will arrive and in
fact, will arrive in the order they were sent.
TCP is built on IP, which is even lower level network protocol. At this
layer, packets may or may not actually arrive, and they may arrive in a
different order than how they were sent.
The IP packets are carried on a cat5 wire sticking out the back of my
The Internet connection at my home is a wireless antenna pointed at the top of
a grain elevator eight miles away. So right now, the important code
change I am trying to checkin is a bunch of radio signals which represent
packets that may or may not arrive, but they are flying through the air, 25
feet above a corn field.
the Vault Server:
The Vault server is an XML Web Service. This allows us to think of our
server as a collection of methods which will invoked in an "RPC-like"
- ASP.NET. The
illusion of XML Web Services is actually provided by ASP.NET.
This library provides a wrapper which insulates the rest of the server from
having to know anything about SQL.
Procs. This layer is a collection of stored procedures running
inside SQL Server.
The SQL language is an enormous abstraction. It presents concepts like
tables, rows and indices, as well as atomic transactions.
calls. Somewhere deep inside SQL Server 2000 is the place where
data is actually written to the disk file. They probably call the native
Win32 IO calls.
The filesystem is a very important abstraction. It presents the concept
of files and folders, as well as permissions and attributes.
The filesystem exists on a "partition", which is a portion of the space on a
array. The RAID controller presents the illusion of one hard
disk when it is actually several.
disk. In practical terms, this was the goal of the checkin all
along. My bits are finally stored in my hard disk. But the disk
itself is actually an abstraction...
The term "hard disk" sounds singular, but hard disks today usually have
several platters inside. These platters are the magnetic media where the
data actually resides.
So there you have it -- 46 layers of abstraction which are all involved when
I try to checkin my code. That means there are 46 layers in which
something might go wrong.
Actually the truth is that several of these abstractions are almost
perfect. For example, I've actually never had to worry about the layer
between Assembly and Microcode. As far
as I am concerned, Assembly is an abstraction that always Just
But it would be terribly wrong to ignore all those layers. Yes,
SourceGear's implementation of Vault required us to only write the code for a
few of the layers above. However, when it's time for QA and Tech Support,
all 46 layers are fully in play. Stuff Happens. When a customer has
a problem with Vault, the actual problem could be almost anywhere. We have
to figure out what's gone wrong, even if it's in one of the layers we didn't
create. Ask our tech support team how often layer 29 causes trouble.
How to Kill Your
When you build software, you're going to end up making a lot of decisions
- Which abstractions do you want to build on?
- Where will you get the implementations of those abstractions (platforms,
- How trustworthy are those implementations?
You have lots of alternatives. For example, you can often make a
tradeoff by choosing to work at a lower level of abstraction. By doing so,
your development process will move more slowly, but more of the risks will be
under your control. For example, if I had a really small magnet and really
fine motor control skills, I could skip layers 1 through 45, drive to my office
and modify those platters myself.
The stakes are higher than you might think. You can kill your project
by making the wrong decisions about abstractions. Do you remember the word
processor called WriteNow? This product was my favorite word processor
back when I was a Macintosh fanatic. WriteNow was really fast and had
just the right mix of features.
Today, WriteNow is dead because
somebody got burned by the decisions they made regarding abstractions. You
see, WriteNow was really fast because it was written in 68000 assembly
language. When Apple moved the Macintosh product line to the PowerPC,
WriteNow had nowhere to go.
These choices are hard, and learning from your mistakes is an excellent (but
painful) way to learn. But over the years, I've gathered the following
guidelines which help me make abstraction-related decisions:
Consider your context.
Developing a server
operating system is different from developing an web-based HR application so
employees can check their vacation days. There is no formula which works
well for all kinds of projects. You need to understand what kinds of
risks are appropriate for the kind of software you are trying to
As a general rule, developers of
internal corporate applications tend to use more third party components than
ISVs. If you're writing code for the IT department of a company whose
primary business is not software, then your salary is an expense, not an
investment. Your employer wants you to get the app done FAST, because it
costs less to get it done that way. Corporate IT developers want every
decent abstraction they can get.
ISVs like SourceGear face a different
set of problems. If a third-party component brings even a
minor loss of quality to the app, it can severely affect our sales as
prospective customers look at our competitors. But that same competition
is tugging you in the other direction, reminding you that time-to-market can
be critical. Using third-party apps may be the only way to get your
product to market within the window of opportunity, but the risks need to be
The size of your company should affect
your choices as well. Nothing is more frustrating than being unable to
ship a product because of a bug that you can't fix because it's in a
third-party library. For a very small company, the financial damage of a
situation like this can be severe. These are the times when you wish you
had chosen to put more of your risks inside your own circle of
Place your trust carefully.
I recommend approaching third
party code with a great deal of suspicion. Never assume that an unknown
component or platform will Just Work. A little paranoia will probably
pay off later.
When picking the pieces of your platform, as a general rule, "older is
better". You can walk with less worry on a path which has been well
trodden by many people for years. As an extremely
obvious example, C is old and mature enough to be a platform which will
yield very few surprises.
Evaluating newer technologies is harder. Try to figure out who else
is using the abstraction successfully. Grab the technology and take it
for a test drive. In the end, you may not get enough evidence to lead to
a completely confident decision. If you really need the convenience
offered by the abstraction, you may have to jump out with a little
Learn to see through the
The most important
point in this whole article is this: You need to understand what's going
on inside all your abstraction layers. Each abstraction presents an
illusion, but the best decisions happen when you can see through the
If you have a deep understanding of all
the technology abstractions that are involved with your software, then you
have two big advantages:
can quickly isolate problems.
can develop an intuition which will help you avoid those problems in the
The first point is fairly
obvious. Troubleshooting goes much better when you know what's going
on. Have you ever watched someone try to solve a problem in the presence
of several abstractions they didn't understand? They feel
helpless. Usually, they start making wild guesses about where the
problem could be. I call this "stab in the dark debugging".
More importantly, if you can see
through most abstractions then you can develop an intuition to make much
better technology decisions. Choosing the right libraries and components
in your platform can prevent lots of problems before they ever
Don't assume that this kind
of deep technical knowledge becomes less valuable as you climb the management
ladder. Understanding this stuff can be a huge advantage in many kinds
of decisions, right up to the most executive levels. I believe the
technical prowess of Bill Gates was a major reason why Microsoft beat every
competitor in the eighties and nineties, even though Bill probably wrote no
code, no specs, and no design documents.
Failures and Successes
It wouldn't be fair to only mention the mistakes of others when I've made so
many excellent and instructive mistakes of my own.
My most recent blunders in this area happened when we built SourceOffSite Collab on a pile of
abstractions which was way too short.
- We built our own implementation of the "server pages" concept because ASP
didn't meet our requirements perfectly. Collab includes its own
web server which processes pages we call "giglets". In between the <%
been modified with special Collab-specific hooks. In retrospect, we
should have found a way to work around the limitations of Microsoft's standard
dynamic page generation technology.
- We also implemented a complete system for XML-based procedure calls.
We rationalized this one because XML-RPC and SOAP just weren't quite
perfect for our needs. Hindsight now brings us to the same
conclusion as above -- changing our requirements to fit the established
platforms would have been the wiser choice.
Better decisions would have gotten SOS Collab to market sooner and we would
have fewer code maintenance problems now.
Not all of our decisions went badly. We made a great choice when we
decided to build Vault using .NET.
From the beginning, I hoped that .NET was "Java done right". I've used
Java extensively, and I loved the productivity gains we got during the beginning
and middle of the development cycle. But things got ugly at the end.
All those layers of abstraction started contributing to our bug list. I've
been involved in a couple of projects which completely failed because Java was
chosen. (Yes, this is merely my opinion, and yes, there were other
factors in the failure of those projects.)
Given our bad experiences with Java, our decision to use .NET took a fair
amount of courage. Early experiments looked promising, but we knew that we
would have to wait for the endgame to really know if .NET could really be
trusted all the way through.
As I write this, SourceGear Vault
1.0 has been shipping for over two months. We have no regrets.
For an abstraction pile as large as the one described above, it's remarkable
that this product works at all.
But the fact that it works well is nothing short of amazing.
We have test applications which continuously try to abuse Vault in ways that are
abusive and profane. If something goes wrong in any layer of
abstraction, the whole test will come to a halt. But we can let these
tests run for days at a time without any problems whatsoever.
This success stands as a testimony to how incredible .NET really is. We
built a reasonably full-featured version control system in 14 months, and it
works. Sure, we had some trouble. Layers 25, 37 and 40 didn't always
behave like they should. But layer 11 was problem-free, quite unlike its
Java counterpart. Considering the productivity gains we received, I never
expected things to go so smoothly.
Note: In response to the controversy generated by this article, I
posted some followup