Home About Twitter GitHub
The Business of Software The 22 Immutable Laws of Marketing Version Control by Example Software Development

Friday, 15 March 2019

Blog Reboot

Beware the Ides of March!

Tuesday, 24 January 2017

Don't use ReactiveUI

TL;DR

This blog post says the opposite of its lazy and deliberately provocative title. I have become a huge fan of ReactiveUI. I just want to ramble about the path I took to get here.

Listening to Paul Betts

I first heard about ReactiveUI at a conference presentation by Paul Betts. I think it was at Xamarin Evolve. Mostly I remember feeling dumb. He said a lot of things that I didn't understand.

I went to that session without much real experience in Model-View-ViewModel (MVVM) development. Conceptually, I understood the idea of a ViewModel. But Paul mostly talked about how ReactiveUI avoids certain problems. And since I had not experienced those problems, his words didn't sink in.

Talking to teenagers about risk

Each time one of my kids was approaching adolescence, I sat down and explained the risks associated with certain choices. Laws and moral judgements aside, the simple fact is that many choices involve risks, and I thought it would be helpful to pass along that bit of information.

And in each case, my child said, "Thanks Dad", and proceeded to always make wise and low-risk choices from that point on.

Well, actually, no.

Teenagers simply do not learn that way. They process risk very differently from people who are more mature. Tell a 16-year-old that "if you drive too fast you might get a ticket". The adolescent will immediately begin driving too fast, and, in all likelihood, will not get ticket. This is how teenagers realize they are smarter than their parents.

Tangent #1: It is almost certainly a good thing that young people are more brave. It would be Very Bad if everybody started out with the same level of risk aversion as the average 65-year-old. Go watch the "Tapestry" episode of Star Trek TNG.

Tangent #2: I really should claim no expertise in parenting, but if somebody forced me to write a book on parenting a teenager, I would say this: Let your kid suffer from their own choices. That said, it is worth the effort to try and help them avoid the really bad mistakes, the ones with consequences that last for decades. But they do have to learn to make their own choices. Realize this as early as you can. The path to frustration starts with making everything all about you.

How we learn new technologies

My metaphor has many problems. For starters, Paul Betts is not my Dad.

Also, the element of adolescent rebellion was not present. I didn't hear Paul's wisdom and run in the opposite direction because of my deep need to separate my identity from his. In fact, I started devouring everything I could find on MVVM and IObservable. I really wanted to understand what he was saying.

But the metaphor works in one significant way: Like a teenager, I had to learn by doing. Nobody's words made much of a difference. None of that reading helped me become a a user of ReactiveUI. I went down another path.

Actually, I went down several other paths.

Maybe it's just me

I observe that most developers want content that explains how to get something done. "If your objective is to do X, then do the following steps." The most popular books and articles tend to follow this pattern. Questions of this form are the ones that do well on StackOverflow.

But this is almost never what I want.

I much prefer content that explains how things work. Once I understand that, I can figure out the steps myself.

When I am developing software, I always, ALWAYS do better when I understand what is going on "under hood", when I can see through the leaky abstractions.

And as I mentioned, I am apparently in the very small minority on this. If 90% of the world disagrees with me, does that put me in the top 10% ? Or does it mean my approach is somehow defective? Modesty aside, my history contains enough successes to allow me some confidence in believing that my approach is better.

I also observe that my approach is just a different spelling for the old adage, "Give a man a fish and you feed him for a day. Teach a man to fish and he eats for a lifetime."

If you tell a software developer what to type and where to click, you can help them complete today's task. But if you instead teach them how things work, they will be able to apply that understanding on other days too.

Hmmm. I'm talking myself into this. I don't know why most people prefer shallow recipes, but I really do think deep understanding is better.

Still, I like to stay open-minded about things. I've got a lot of failures too.

The truth is that my approach has tradeoffs. The need to understand everything tends to slow me down during the early stages. I usually gain some of that back in the fourth quarter of the game, where deeper understanding is often helpful in diagnosing tricky problems.

But again, in the decision making around software development, absolutes are rare. I'll admit that sometimes a simple set of steps without depth are exactly what is needed.

Maybe the ReactiveUI docs are just bad?

I don't know. Maybe. I've read the docs plenty. They don't seem bad to me. I also see nothing there that makes me want to defend them as the best docs ever.

Suppose that I regret not choosing ReactiveUI sooner. Further suppose that I wanted to blame somebody else for my choices. I guess I could find something to complain about. But I also don't tend to find that criticizing somebody else's work is helpful.

And remember, I started this journey sitting in front of an audience, listening to Paul Betts, and feeling dumb. To be clear, in that kind of context, I *like* feeling dumb. It's an opporunity to learn.

So why did I not choose ReactiveUI sooner?

I guess I don't really know. But I'm pretty sure that nothing has made me appreciate ReactiveUI more than the suffering that comes from not using it.

And that remark isn't very helpful, is it? I'd like to try and do better. Let's see...

"Son, it's just basic statistics. If you're going to always drive 15 MPH over the speed limit, you will eventually get caught. Suppose you roll the dice 20 times in a row without getting a 12. You still might get a 12 on the next roll, right?"

Oh, wait, wrong topic. Let me try again.

Why is ReactiveUI awesome?

In some software development situations, like mobile apps, if you take a step back and look at the forest instead of the trees, you will see that most of your code is reacting to something that changed.

There are lots of tools you can use to approach this kind of app. You can use C# events and callbacks and switch statements and delegates and lambdas and observables and notifications and bindings and more.

For simple apps, none of these approaches are much better than any other. But as your app gets more complicated, some approaches cope more gracefully than others.

Most cars drive pretty smooth at 30 MPH. But at 75 MPH, some vehicles are still giving a smooth ride, while others are shaking.

Let's try a conceptual example or two. Suppose you have a button, and you want something to happen when the user presses that button. This is pretty simple. All reasonable solutions to this problem are about the same.

On the other hand, let's say you have a list of items. The items in that list come from a SQL query. That query has 4 inputs, each of which comes from a UI control. Every time one of those controls changes its value, the query needs to be re-run and the list needs to be updated. A couple of those controls need to be disabled under certain circumstances.

These UI elements have a complicated relationship. We still have plenty of choices in how to express that relationship in code, but this situation is complicated enough that we start to see differences between those approaches. Some of the ones that worked out really well in the simple case seem kinda tedious for this case.

If my driveway has half an inch of snow, all methods of clearing it are about the same. But if my driveway has 15 inches of snow, a shovel is decidedly inferior to a tractor.

Why do I like ReactiveUI? Because I have found that it copes gracefully as the situation gets more complicated.

Why is this? Much of the credit goes to the "reactive" foundation on which ReactiveUI is built. Reactive Extensions. Rx. IObservable. These building blocks are particularly adept at expressing the relationship between a group of things that are changing. ReactiveUI adds another layer (or two) on top of these things to make that expressiveness more convenient when implementing user interfaces.

To be honest, I fudged a little bit when I said that all solutions are roughly equivalent when the problem is simple. That's not quite true. For simple situations, I'd have to admit that ReactiveUI might be a little worse. There is a learning curve.

If I am writing a grocery list, I could use a word processor, but a pencil and paper is actually simpler. But if I am writing a novel, the word processor is the clear winner.

I'm claiming that the effort to learn Rx and ReactiveUI is worth the trouble. My claim is based on this notion that ReactiveUI shines as complexity increases, but also on my belief that most people underestimate the complexity of their app.

If you disagreed with me above when I said that "most of your code is reacting to something that changed", you might be underestimating the complexity of your app. It is in fact very common to start implementing under the assumption that something will not change and then later realize that you need notifications or events. Or an observable.

Hmmmm.

Would the paragraphs above have changed my course earlier?

I don't know. Probably not.

I didn't start this believing that I could write the best ReactiveUI advocacy ever. Looking at it now, I can't believe I wrote it with no code in it. The canonical ReactiveUI evangelism pamphlet has gotta have WhenAnyValue() in it somewhere.

I just think it's interesting that despite my best efforts, I was unable to really understand the benefits of ReactiveUI until I tried using its alternatives. My current project is late. If I had chosen ReactiveUI earlier, maybe it would be, er, less late? There are questions here worth asking.

But am I 100% certain that it is always better to spare yourself the learning experience of using less effective approaches? No.

Can I credibly claim that everyone should choose ReactiveUI in every situation? Certainly not.

Maybe all I can say is that I am currently having a great experience with ReactiveUI.

Maybe that means the rest of this blog post is useless.

But you should have known that when you saw the cheesy title.

 

Wednesday, 15 June 2016

SQLite and Android N

TLDR

The upcoming release of Android N is going to cause problems for many apps that use SQLite. In some cases, these problems include an increased risk of data corruption.

History

SQLite is an awesome and massively popular database library. It is used every day by billions of people. If you are keeping a list of the Top Ten Coolest Software Projects Ever, SQLite should be on the list.

Many mobile apps use SQLite in one fashion or another. Maybe the developers of the app used the SQLite library directly. Or maybe they used another component or library that builds on SQLite.

SQLite is a library, so the traditional way to use it is to just link it into your application. For example, on a platform like Windows Phone 8.1, the app developer simply bundles the SQLite library as part of their app.

But iOS and Android have a SQLite library built-in to the platform. This is convenient, because developers do not need to bundle a SQLite library with their software.

However

The SQLite library that comes with Android is actually not intended to be used except through the android.database.sqlite Java classes. If you are accessing this library directly, you are actually breaking the rules.

And the problem is

Beginning with Android N, these rules are going to be enforced.

If your app is using the system SQLite library without using the Java wrapper, it will not be compatible with Android N.

Does your app have this problem?

If your app is breaking the rules, you *probably* know it. But you might not.

I suppose most Android developers use Java. Any app which is only using android.database.sqlite should be fine.

But if you are using Xamarin, it is rather more likely that your app is breaking the rules. Many folks in the Xamarin community tend to assume that "SQLite is part of the platform, so you can just call it".

Xamarin.Android 6.1 includes a fix for this problem for Mono.Data.Sqlite (see their release notes).

However, that is not the only way of accessing SQLite in the .NET/Xamarin world. In fact, I daresay it is one of the less common ways.

Perhaps the most popular SQLite wrapper is sqlite-net (GitHub). If you are using this library on Android and not taking the extra steps to bundle a SQLite library, your app will break on Android N.

Are you using Akavache? Or Couchbase Lite? Both of these libraries use SQLite under the hood (by way of SQLitePCL.raw, which I maintain), so your app will need to be updated to work on Android N.

There are probably dozens of other examples. GitHub says the sqlite-net library has 857 forks. Are you using one of those? Do you use the MvvmCross SQLite plugin? Do any of the components or libraries in your app make use of SQLite without you being aware of it?

And the Xamarin community is obviously not the whole story. There are dozens of other ways to build mobile apps. I can think of PhoneGap/Cordova, Alpha Anywhere, Telerik NativeScript, and Corona, just off the top of my head. How many of these environments (or their surrounding ecosystems) provide (perhaps accidentally) a rule-breaking way to access the Android system SQLite? I don't know.

What I *do* know is that even Java developers might have a problem.

It's even worse than that

Above, I said: "Any app which is only using android.database.sqlite should be fine." The key word here is "only". If you are using the Java classes but also have other code (perhaps some other library) that accesses the system SQLite, then you have the problems described above. But you also have another problem.

To fix this, you are going to have to modify that "other code" to stop accessing the system SQLite library directly. One way to do this is to change the other code to call through android.database.sqlite. But that might be a lot of work. Or that other code might be a 3rd party library that you do not maintain. So you are probably interested in an easier solution.

Why not just bundle another instance of the SQLite library into your app? This is what people who use sqlite-net on Xamarin will need to do, so it should make sense in this case too, right? Unfortunately, no.

What will happen here is that your android.database.sqlite code will continue using the system SQLite library, and your "other code" will use the second instance of the SQLite library that you bundled with your app. So your app will have two instances of the SQLite library. And this is Very Bad.

The Multiple SQLite Problem

Basically, having multiple copies of SQLite linked into the same appliication can cause data corruption. For more info, see this page on sqlite.org. And also the related blog entry I wrote back in 2014.

You really, really do not want to have two instances of the SQLite library in your app.

Zumero

One example of a library which is going to have this problem is our own Zumero Client SDK. The early versions of our sync library bundled a copy of the SQLite library, to follow the rules. But later, to avoid possible data corruption from The Multiple SQLite Problem, we changed it to call the system SQLite directly. So, although I might like to claim we did it for a decent reason, our library breaks the rules, and we did it knowingly. All Android apps using Zumero will need to be updated for Android N. A new release of the Zumero Client SDK, containing a solution to this problem, is under development and will be released soon-ish.

Informed consent?

I really cannot recommend that you have two instances of the SQLite library in your app. The possibility of corruption is quite real. One of our developers created an example project to demonstrate this.

But for the sake of completeness, I will mention that it might be possible to prevent the corruption by ensuring that only one instance of the SQLite library is accessing a SQLite file at any given time. In other words, you could build your own layer of locking on top of any code that uses SQLite.

Only you can decide if this risk is worth it. I cannot feel good about sending anyone down that path.

Stop using android.database.sqlite?

It also makes this blog entry somewhat more complete for me to mention that changing your "other code" to go through android.database.sqlite is not your only option. You might prefer to leave your "other code" unchanged and rewrite the stuff that uses android.database.sqlite, ending up with both sets of code using one single instance of SQLite that is bundled with your app.

A Lament

Life was better when there were two kinds of platforms, those that include SQLite, and those that do not. Instead, we now have this third category of platforms that "previously included SQLite, but now they don't, but they kinda still do, but not really".

An open letter to somebody at Google

It is so tempting to blame you for this, but that that would be unfair. I fully admit that those of us who broke the rules have no moral high ground at all.

But it also true that because of the multiple SQLite problem, and the sheer quantity of apps that use the Android system SQLite directly, enforcing the rules now is the best way to maximize the possibility of Android apps that break or experience data corruption.

Would it really be so bad to include libsqlite in the NDK?

 

Monday, 20 July 2015

MongoDB and WTFs and Anger

Recently, Sven Slootweg (joepie91) published a blog entry entitled Why you should never, ever, ever use MongoDB. It starts out with the words "MongoDB is evil" and proceeds to give a list of negative statements about same.

I am not here to respond to each of his statements. He labels them as "facts", and some (or perhaps all) of them surely are. In fact, for now, let's assume that everything he wrote is correct. My point here is not to say that the author is wrong.

Rather, my point here is that this kind of blog entry tells me very little about MongoDB while it tells me a great deal about the emotions of the person who wrote it.

Like I said, it may be true that every WTF the author listed is correct. It is also true that some software has more WTFs than others.

I'm not a MongoDB expert, but I've been digging into it quite a bit, and I could certainly make my own list of its WTFs. And I would also admit that my own exploration of Couchbase has yielded fewer of those moments. Therefore, every single person on the planet who chooses MongoDB instead of Couchbase is making a terrible mistake, right?

Let me briefly shift to a similar situation where I personally have a lot more knowledge: Microsoft SQL Server vs PostgreSQL. For me, it is hard to study SQL Server without several WTF moments. And while PostgreSQL is not perfect, I have found that a careful study there tends to produce more admiration than WTFs.

So, after I discovered that (for example) SQL Server has no support for deferred foreign keys, why didn't I write a blog entry entitled "Why you should never, ever, ever use SQL Server"?

Because I calmed down and looked at the bigger picture.

I think I could make an entirely correct list of negative things about SQL Server that is several pages long. And I suppose if I wanted to do that, and if I were really angry while I was writing it, I would include only the facts that support my feelings, omitting anything positive. For example, my rant blog entry would have no reason to acknowledge that SQL Server is the mostly widely used relational database server in the world. These kinds of facts merely distract people from my point.

But what would happen if I stopped writing my rant and spent some time thinking about the fact I just omitted?

I just convinced myself that this piece of software is truly horrible, and yet, millions of people are using it every day. How do I explain this?

If I tried to make a complete list of theories that might fit the facts, today's blog entry would get too long. Suffice it to say this: Some of those theories might support an anti-Microsoft rant (for example, maybe Microsot's field sales team is really good at swindling people), but I'm NOT going to be able to prove that every single person who chose SQL Server has made a horrible mistake. There is no way I can credibly claim that PostgreSQL is the better choice for every single company simply because I admire it. Even though I think (for example) that SQL Server handles NULL and UNIQUE in a broken way, there is some very large group of people for whom SQL Server is a valid and smart choice.

So why would I write a blog entry that essentially claims that all SQL Server users are stupid when that simply cannot be true? I wouldn't. Unless I was really angry.

MongoDB is undisputably the top NoSQL vendor. It is used by thousands of companies who serve millions of users every day. Like all young software serving a large user base, it has bugs and flaws, some of which are WTF-worthy. But it is steadily getting better. Any discussion of its technical deficiences which does not address these things is mostly just somebody venting emotion.

 

Monday, 8 June 2015

My initial experience with Rust

First, a digression about superhero movies

I am apparently incapable of hating any movie about a comic book superhero.

I can usually distinguish the extremes. Yes, I can tell that "The Dark Knight" was much better than "Elektra". My problem is that I tend to think that the worst movies in this genre are still pretty good.

And I have the same sort of unreasonable affection toward programming languages. I have always been fascinated by languages, compilers, and interpreters. My opinions about such things skew toward the positive simply because I find them so interesting.

I do still have preferences. For example, I tend to like strongly typed languages more. In fact, I think it is roughly true that the stricter a compiler is, the more I like it. But I can easily find things to like in languages that I mostly dislike.

I've spent more of my career writing C than any other language. But in use cases where I need something like C, I am increasingly eager for something more modern.

I started learning Rust with two questions:

The context

My exploration of Rust has taken place in one of my side projects: https://github.com/ericsink/LSM

LSM is a key-value database with a log-structured merge tree design. It is conceptually similar to Google LevelDB. I first wrote it in C#. Then I rewrote/ported it to F#. Now I have ported it to Rust. (The Rust port is not yet mentioned in the README for that repo, but it's in the top-level directory called 'rs'.)

For the purpose of learning F# and Rust, my initial experience was the same. The first thing I did in each of these languages was to port LSM. In other words, the F# and Rust ports of LSM are on equal footing. Both of them were written by someone who was a newbie in the language.

Anyway, although Rust and F# are very different languages, I have used F# as a reference point for my learning of Rust, so this blog entry walks that path as well.

This is not to say that I think Rust and F# would typically be used for the same kinds of things. I can give you directions from Denver to Chicago without asserting they are similar. Nonetheless, given that Rust is mostly intended to be a modern replacement for C, it has a surprising number of things in common with F#.

The big comparison table

  F# Rust
Machine model Managed, .NET CLR Native, LLVM
Runtime CLR None
Style Multi-paradigm, functional-first Multi-paradigm, imperative-first
Syntax family ML-ish C-ish
Blocks Significant whitespace Curly braces
Exception handling Yes No
Strings .NET (UTF-16) UTF-8
Free allocated memory Automatic, garbage collector Automatic, static analysis
Type inference Yes, but not from method calls Yes, but only within functions
Functional immutable collections Yes No
Currying Yes No
Partial application Yes No
Compiler strictness Extremely strict Even stricter
Tuples Yes Yes
Discriminated unions 
type Blob =
    | Stream of Stream
    | Array of byte[]
    | Tombstone

enum Blob {
    Stream(Box),
    Array(Box<[u8]>),
    Tombstone,
}
Mutability To be avoided Safe to use
Lambda expressions 
let f = 
  (fun acc item -> acc + item)

let f = 
  |acc, &item| acc + item;
Higher-order functions List.fold f 0 a a.iter().fold(0, f)
Integer overflow checking No open Checked Yes
Let bindings 
let x = 1
let mutable y = 2

let x = 1;
let mut y = 2;
if statements are expressions Yes Yes
Unit type () ()
Pattern matching 
match cur with
| Some csr -> csr.IsValid()
| None -> false

match cur {
    Some(csr) => csr.IsValid(),
    None => false
}
Primary collection type Linked list Vector
Naming types CamelCase CamelCase
Naming functions, etc camelCase snake_case
Warnings about naming conventions No Yes
Type for integer literals Suffix (0uy) Inference (0) or suffix (0u8)
Project file foo.fsproj (msbuild) Cargo.toml
Testing framework xUnit, NUnit, etc. Built into Cargo
Debug prints printf "%A" foo println!("{:?}", foo);

Memory safety

I have written a lot of C code over the years. More than once while in the middle of a project, I have stopped to explore ways of getting the compiler to catch my memory leaks. I tried the Clang static analyzer and Frama-C and Splint and others. It just seemed like there should be a way, even if I had to annotate function signatures with information about who owns a pointer.

So perhaps you can imagine my joy when I first read about Rust.

Even more cool, Rust has taken this set of ideas so much further than the simple feature I tried to envision. Rust doesn't just detect leaks, it also:

That last bullet is worth repeating: With Rust, you never stare at your code trying to figure out if it's thread safe or not. If it compiles, then it's thread safe.

Safety is Rust's killer feature, and it is very compelling.

Mutability

If you come to Rust hoping to find a great functional language, you will be disappointed. Rust does have a bunch of functional elements, but it is not really a functional language. It's not even a functional-first hybrid. Nonetheless, Rust has enough cool functional stuff available that it has been described as "ML in C++ clothing".

I did my Rust port of LSM as a line-by-line translation from the F# version. This was not a particularly good approach.

So if you're porting code from a more functional language, you can end up with code that isn't very Rusty.

If you are a functional programming fan, you might be skeptical of Rust and its claims. Try to think of it like this: Rust agrees that mutability is a problem -- it is simply offering a different solution to that problem.

Learning curve

I don't know if Rust is the most difficult-to-learn programming language I have seen, but it is running in that race.

Anybody remember back when Joel Spolsky used to talk about how difficult it is for some programmers to understand pointers? Rust is a whole new level above that. Compared to Rust, regular pointers are simplistic.

With Rust, we don't just have pointers. We also have ownership, borrows, and lifetimes.

As you learn Rust, you will reach a point where you think you are starting to understand things. And then you try to return a reference from a function, or store a reference in a struct. Suddenly you have lifetime<'a> annotations<'a> all<'a> over<'a> the<'a> place<'a>.

And why did you put them there? Because you understood something? Heck no. You started sprinkling explicit lifetimes throughout your code because the compiler error messages told you to.

I'm not saying that Rust isn't worth the pain. I personally think Rust is rather brilliant.

But a little expectation setting is appropriate here. Some programming languages are built for the purpose of making programming easier. (It is a valid goal to want to make software development accessible to a wider group of people.) Rust is not one of those languages.

That said, the Rust team has invested significant effort in excellent documentation (see The Book). And those compiler error messages really are good.

Finally, let me observe that while some things are hard to learn because they are poorly designed, Rust is not one of those things. The deeper I get into this, the more impressed I am. And so far, every single time I thought the compiler was wrong, I was mistaken.

I have found it helpful to try to make every battle with the borrow checker into a learning experience. I do not merely want to end up with the compiler accepting my code. I want to understand more than I did when I started.

Error handling

Rust does not have exceptions for error handling. Instead, error handling is done through the return values of functions.

But Rust actually makes this far less tedious than it might sound. By convention (and throughout the Rust standard library), error handling is done by returning a generic enum type called Result<T,E>. This type can encapsulate either the successful result of the function or an error condition.

On top of this, Rust has a clever macro called try!. Because of this macro, if you read some Rust code, you might think it has exception handling.

// This code was ported from F# which assumes that any Stream
// that supports Seek also can give you its Length.  That method
// isn't part of the Seek trait, but this implementation should
// suffice.
fn seek_len(fs: &mut R) -> io::Result where R : Seek {
    // remember where we started (like Tell)
    let pos = try!(fs.seek(SeekFrom::Current(0)));

    // seek to the end
    let len = try!(fs.seek(SeekFrom::End(0)));

    // restore to where we were
    let _ = try!(fs.seek(SeekFrom::Start(pos)));

    Ok(len)
}

This function returns std::io::Result<u64>. When it calls the seek() method of the trait object it is given, it uses the try! macro, which will cause an early return of the function if it fails.

In practice, I like Rust's Result type very much.

Nonetheless, when doing a line-by-line port of F# to Rust, this was probably the most tedious issue. Lots of functions that returned () in F# changed to return Result in Rust.

Type inference

Rust does type inference within functions, but it cannot or will not infer the types of function arguments or function return values.

Very often I miss having the more complete form of type inference one gets in F#. But I do remind myself of certain things:

Iterators

Rust iterators are basically like F# seq (which is an alias for .NET IEnumerable). They are really powerful and provide support for functional idioms like List.map. For example:

fn to_hex_string(ba: &[u8]) -> String {
    let strs: Vec = ba.iter()
        .map(|b| format!("{:02X}", b))
        .collect();
    strs.connect("")
}

However, there are a few caveats.

In Rust, you have a lot more flexibility about whether you are dealing with "a Foo" or "a reference to a Foo", and most of the time, it's the latter. Overall, this is just more work than it is in F#, and using iterators feels like it magnifies that effect.

Performance

I haven't done the sort of careful benchmarking that is necessary to say a lot about performance, so I will say only a little.

Integer overflow

Integer overflow checking is one of my favorite features of Rust.

In languages or environments without overflow checking, unsigned types are very difficult to use safely, so people generally use signed integers everywhere, even in cases where a negative value makes no sense. Rust doesn't suffer from this silliness.

For example, the following code will panic:

let x: u8 = 255;
let y = x + 2;
println!("{}", y);

That said, I haven't quite figured out how to get overflow checking to happen on casts. I want the following code (or something very much like it) to panic:

let x: u64 = 257;
let y = x as u8;
println!("{}", y);

Note that, by default, Rust turns off integer overflow checking in release builds, for performance reasons.

Miscellany

Bottom line

I am seriously impressed with Rust. Then again, I thought that Eric Bana's Hulk movie was pretty good, so you might want to just ignore everything I say.

In terms of maturity and ubiquity, C has no equal. Still, I believe Rust has the potential to become a compelling replacement for C in many situations.

I look forward to using Rust more.

 

Friday, 8 May 2015

Announcing Zumero for SQL Server, Release 2.0

Zumero for SQL Server (ZSS) is a solution for replication and sync between SQL Server and mobile devices. ZSS can be used to create offline-friendly mobile apps for iOS, Android, Windows Phone, PhoneGap, and Xamarin.

Our 2.0 release is a major step forward in the maturity of the product.

Highlights:

Lots more info at zumero.com.

 

Thursday, 30 April 2015

Microsoft is becoming cool again

Isn't "New Microsoft" awesome?

.NET is going open source? And cross-platform? On Github?!?

The news out of Redmond often seems like a mis-timed April fools joke.

But it's real. This is happening. Microsoft is apparently willing to do "whatever it takes" to get developers to love them again.

How did this company change so much, so quickly?

A lot of folks are giving credit to CEO Satya Nadella. And there could be some truth to that. Maaaaaaybe.

Another popular view: Two of the most visible people in this story are:

I understand why people think maybe these two guys caused this revolution. They both seem to do a decent job I suppose.

But the truth is that New Microsoft started when Microsoft hired Martin Woodward.

What?!? Who the heck is Martin Woodward?

Martin's current position is Executive Director of the .NET Foundation. Prior to that, he worked as a Program Manager on various developer tools.

Nobody knows who Martin is. Either of the Scotts have 2 orders of magnitude more Twitter followers.

But I think if you look closely at Martin's five+ year career at Microsoft, you will see a pattern. Every time a piece of Old Microsoft got destroyed in favor of New Microsoft, Martin was nearby.

Don't believe me? Ask anybody in DevDiv how TFS ended up getting support for Git.

It's all about Martin.

So all the credit should go to Martin Woodward then?

Certainly not.

You see, Martin joined Microsoft in late 2009 as part of their acquisition of Teamprise.

And Teamprise was a division of SourceGear.

I hired Martin Woodward (single-handedly, with no help or input from anybody else) in 2005. Four years later, when Microsoft acquired our Teamprise division (which I made happen, all by myself), Martin became a Microsoft employee.

Bottom line: Are you excited about all of the fantastic news coming out of Build 2015 this week? That stuff would never have happened if it were not for ME.

Eric, how can we ever thank you?

So, now that you know that I am the one behind all the terrific things Microsoft is doing, I'm sure you want to express your appreciation. But that won't be necessary. While I understand the sentiment, in lieu of expensive gifts and extravagant favors, I am asking my adoring fans to do two things:

First, try not to forget all the little people at Microsoft who are directly involved in the implementation of change. People like Martin, or the Scotts, or Satya. Even though these folks are making a relatively minor contribution compared to my own, I would like them to be supported in their efforts. Be positive. Don't second-guess their motives. Lay aside any prejudices you might have from Old Microsoft. Believe.

Second, get involved. Interact with New Microsoft as part of the community. Go to Github and grab the code. Report an issue. Send a pull request.

Embellishments and revisionist history aside...

Enjoy this! It's a GREAT time to be a developer.

 

Monday, 27 April 2015

What Mongo-ish API would mobile developers want?

A couple weeks ago I blogged about mobile sync for MongoDB.

Updated Status of Elmo

Embeddable Lite Mongo continues to move forward nicely:

I also refactored to get better separation between the CRUD logic and the storage of bson blobs and indexes (making it easier to plug-in different storage layers).

Questions about client-side APIs

So, let's assume you are building a mobile app which communicates with your Mongo server in the cloud using a "replicate and sync" approach. In other words, your app is not doing its CRUD operations by making networking/REST calls back to the server. Instead, your app is working directly with a partial clone of the Mongo database that is right there on the mobile device. (And periodically, that partial clone is magically synchronized with the main database on the server.)

What should the API for that "embedded lite mongo" look like?

Obviously, for each development environment, the form of the API should be designed to feel natural or native in that environment. This is the approach taken by Mongo's own client drivers. In fact, as far as I can tell, these drivers don't even share much (or any?) code. For example, the drivers for C# and Java and Ruby are all different, and (unless I'm mistaken) none of them are mere wrappers around something lower level like the C driver. Each one is built and maintained to provide the most pleasant experience to developers in a specific ecosystem.

My knee-jerk reaction here is to say that mobile developers might want the exact same API as presented by their nearest driver. For example, if I am building a mobile app in C# (using the Xamarin tools), there is a good chance my previous Mongo experience is also in C#, so I am familiar with the C# driver, so that's the API I want.

Intuitive as this sounds, it may not be true. Continuing with the C# example, that driver is quite large. Is its size appropriate for use on a mobile device? Is it even compatible with iOS, which requires AOT compilation? (FWIW, I tried compiling this driver as a PCL (Portable Class Library), and it didn't Just Work.)

For Android, the same kinds of questions would need to be asked about the Mongo Java driver.

And then there are Objective-C and Swift (the primary developer platform for iOS), for which there is no official Mongo driver. But there are a couple of them listed on the Community Supported Drivers page: http://docs.mongodb.org/ecosystem/drivers/community-supported-drivers/.

And we need to consider Phonegap/Cordova as well. Is the Node.js driver a starting point?

And in all of these cases, if we assume that the mobile API should be the same as the driver's API, how should that be achieved? Fork the driver code and rip out all the networking and replace it with calls to the embedded library?

Or should each mobile platform get a newly-designed API which is specifically for mobile use cases?

Believe it or not, some days I wonder: Suppose I got Elmo running as a server on an Android device, listening on localhost port 27017. Could an Android app talk to it with the Mongo Java driver unchanged? Even if this would work, it would be more like a proof-of-concept than a production solution. Still, when looking for solutions to a problem, the mind goes places...

So anyway, I've got more questions than answers here, and I would welcome thoughts or opinions.

 

Monday, 13 April 2015

Mobile Sync for Mongo

We here at Zumero have been exploring the possibility of a mobile sync solution for MongoDB.

We first released our Zumero for SQL Server product almost 18 months ago, and today there are bunches of people using mobile apps which sync using our solution.

But not everyone uses SQL Server, so we often wonder what other database backends we should consider supporting. In this blog entry, I want to talk about some progress we've made toward a "Zumero for Mongo" solution and "think out loud" about the possibilities.

Background: Mobile Sync

The basic idea of mobile sync is to keep a partial copy of the database on the mobile device so the app doesn't have to go back to the network for every single CRUD operation. The benefit is an app that is faster, more reliable, and works offline. The flip side of that coin is the need to keep the mobile copy of the database synchronized with the data on the server.

Sync is tricky, but as mobile continues its explosive growth, this approach is gaining momentum:

If the folks at Mongo are already working on something in this area, we haven't seen any sign of it. So we decided to investigate some ideas.

Pieces of the puzzle

In addition to the main database (like SQL Server or MongoDB or whatever), a mobile sync solution has three basic components:

Mobile database

  • Runs on the mobile device as part of the app

  • Probably an embedded database library

  • Keeps a partial replica of the main database

  • Wants to be as similar as possible to the main database
Sync server

  • Monitors changes made by others to the main database

  • Sends incremental changes back and forth between clients and the main database

  • Resolves conflicts, such as when two participants want to change the same data

  • Manages authentication and permissions for mobile clients

  • Filters data so that each client only gets what it needs
Sync client

  • Monitors changes made by the app to the mobile database

  • Talks over the network to the sync server

  • Pushes and pulls incremental changes to keep the mobile database synchronized

For this blog entry, I want to talk mostly about the mobile database. In our Zumero for SQL Server solution, this role is played by SQLite. There are certainly differences between SQL Server and SQLite, but on the whole, SQLite does a pretty good job pretending to be SQL Server.

What embedded database could play this role for Mongo?

This question has no clear answer, so we've been building a a lightweight Mongo-compatible database. Right now it's just a prototype, but its development serves the purpose of helping us explore mobile sync for Mongo.

Embeddable Lite Mongo

Or "Elmo", for short.

Elmo is a database that is designed to be as Mongo-compatible as it can be within the constraints of mobile devices.

In terms of the status of our efforts, let me begin with stuff that does NOT work:

Broadly speaking, except for the above, everything works. Mostly:

More caveats:

Notes on testing:

How Elmo should work:

Notes on the code:

Next steps:

 

Tuesday, 27 January 2015

Improvements in Xamarin.Forms 1.3

Back in November I wrote a blog entry about performance problems resulting from the design of the layout system in Xamarin.Forms. I am pleased to report that things took a big step forward with the recent release of version 1.3.

Reviewing the problem

In a nutshell, the Layout classes do too much. They contain functionality to make sure everything gets updated whenever something changes. In principle, this is good, since we obviously don't want stale stuff on the screen. But in practice, there are many cases where the built-in update code ends up being slower than necessary.

For example, suppose I'm going to add ten child views to a layout. With the built-in update code, a layout cycle will get triggered ten times, once for each child view I add. Worse, if I'm trying to do any kind of subview recycling, the odds are high that I want to add a child view while I am processing a layout cycle. This will trigger a recursive layout cycle, resulting in the end of civilization as we know it.

Instead, what I want is one layout cycle which happens after all ten child views have been added.

The solution I proposed

IMHO, the best design for this kind of problem is to have multiple layers:

Xamarin.Forms has the High-Level layer and the Middle layer, but it does not have the Low-Level layer. So I proposed creating it.

I didn't get exactly what I wanted, but...

The solution in Xamarin.Forms 1.3

In Xamarin.Forms 1.3, the Middle layer is still the lowest thing we've got. However, there are new capabilities which allow the Middle layer to pretend like it is a Low-Level layer. It still has a bunch of built-in update code, but now that code can be turned off. :-)

The important new capabilities are:

By returning false from my override of ShouldInvalidateOnChildAdded() and ShouldInvalidateOnChildRemoved(), I can have a Layout which doesn't do any automatic updates when I add or remove children.

And by overriding OnChildMeasureInvalidated(), I can have a Layout which refuses to do real estate negotiations with its child views.

This is good.

How I'm using this

Because of this new stuff, an upcoming release of our DataGrid component will be even faster. Our panel layout class will look something like this:

private class myLayout : Layout<View>
{
    Func<View,Rectangle> getBox;

    public myLayout(Func<View,Rectangle> f)
    {
        getBox = f;
    }

    public void LayoutOneChild(View v)
    {
        Rectangle r = getBox (v);
        v.Layout (r);
    }

    public void LayoutAllChildren()
    {
        foreach (View v in Children) {
            LayoutOneChild (v);
        }
    }

    protected override bool ShouldInvalidateOnChildAdded (View child)
    {
        return false; // stop pestering me
    }

    protected override bool ShouldInvalidateOnChildRemoved (View child)
    {
        return false; // go away and leave me alone
    }

    protected override void OnChildMeasureInvalidated ()
    {
        // I'm ignoring you.  You'll take whatever size I want to give
        // you.  And you'll like it.
    }

    protected override void LayoutChildren (double x, double y, double width, double height)
    {
        LayoutAllChildren ();
    }
}

This Layout class is obviously very simplistic, but it merely scratches the surface of what becomes possible now that Xamarin.Forms has [something that can imitate] a Low-Level subview layer.

Kudos and thanks to the Xamarin.Forms team!

 

Copyright 2001-2019 Eric Sink. All Rights Reserved