The correct answer is: "ALL apps".

Actually, that's an exaggeration. I see no reason for my alarm clock or calculator apps to sync data with a server. Let me rephrase:

Sync is the best architecture
for apps where a database and a server are involved.

CRUD over REST is an Anti-Pattern

Many apps today keep the data only on the server and access it using REST. This approach has some big negative consequences:

• Performance: Your app is less responsive.

  A round-trip from a mobile device to a server takes a LONG time. Under good network conditions, it could be a quarter of a second. That's enough time for the user to perceive a lag.

  Sometimes the round-trip will take longer, more like a full second, or more. That's an eternity, a one-star review in the App Store.

  Numerous studies show that when it comes to the responsiveness of a user interface, fractions of a second matter a LOT. And people don't necessarily complain about your app. They just stop using it.

  If you're going to make a round-trip to the server in response to a user action, your app is going to feel sluggish.

• Reliability: Your app doesn't work offline.

  WiFi is great, but not everything is a coffee shop.

  Cellular data networks have dead spots, sometimes in surprising places. Even in areas with solid 3G or LTE coverage, the latency varies wildly.

  If your app is going to stop working when the wireless network is unavailable or unreliable, then your app is going to stop working a lot.

• Complexity: Your app's code has to deal with networking issues everywhere.

  People wrap their REST calls in pretty wrappers to make them look like networking is not involved. But it is. A robust app needs to deal with all the things that can go wrong.

  If you're going to do all your basic database operations over a REST API, all your code is networking code.

The real question is "What kinds of apps would benefit from having user interaction dependent on a wireless network with inconsistent coverage and random latency spikes?"

Your app does lots of basic database operations which are often referred to as CRUD (Create, Read, Update, Delete). These operations do not want to happen over a network. They want to happen in a database which is local, on the mobile device.

And so you need sync. In the background (so the user doesn't have to wait on it). With automatic conflict resolution.

And that's hard.

Solutions that make sync easy

Do you like SQL? If so, I invite you to look at Zumero, our database sync platform based on SQLite, the relational database software preinstalled on over a billion mobile devices.

Do you prefer NoSQL? Then I'm sure our worthy competitors over at Couchbase would be happy for you to look at Couchbase Lite.

Either way, keep your CRUD off the Internet.

Zumero SQLite for Xamarin starts with the core "replicate and sync" functionality from the Zumero Client SDK and adds:

• ADO.NET -- Familiar C# API for SQL developers. (provided by System.Data.SQLite, which is maintained by the same folks who develop SQLite itself)

• Full-database encryption -- Critical piece of the "replicate and sync" story. Mobile devices get lost. (provided by the excellent SQLCipher library)

• Painless setup -- Install the component and you've got everything you need. (provided by the neato component store feature of Xamarin Studio)

Comfy

In some ways, the core of the Zumero Client SDK is not very easy to use. That is intentional. It is designed to be the sort of SDK that leaves many things as the responsibility of the app (or the next layer up).

• It doesn't actually include SQLite itself, since the app might care about which version of SQLite it wants, or how it needs to be compiled.

• It doesn't include any page-level encryption support, since it doesn't know whether the app wants to use SQLCipher or the SQLite Encryption Extension or something else.

• It plugs into SQLite using C, because that's the native language that SQLite uses for its API.

Power users and control freaks need this kind of SDK. And it comes first, since everything else is built on top of it. But most users want an experience that is more "comfy".

• They don't want to think about versions of SQLite or compile options for same.

• They don't want to choose an encryption library and figure out how to integrate it with their app.

• They don't want to write their mobile app in C.

The Zumero Client SDK actually does include some carpeting and furniture on top of that cold, hard C floor. For Android developers using Java, it presents the android.database.sqlite API. And for iOS developers using Objective-C, it offers a wrapper that lets you work with things that have NS prefixes.

(Which reminds me, check out our iOS wiki app sample on GitHub.)

Anyway, these efforts are just the beginning. Our direction is toward "comfy", and we are moving fast.

This component for Xamarin is a step forward, and I think it is an important one.

Samples

This new component includes three sample apps, all of which are modified versions of samples provided by Xamarin.

• Data (iOS, based on https://github.com/xamarin/monotouch-samples/tree/master/Data)

• Notepad (Android, based on https://github.com/xamarin/monodroid-samples/tree/master/NotePad-Mono.Data.Sqlite)

• Tasky (cross-platform, based on https://github.com/xamarin/mobile-samples/tree/master/Tasky)

The Data and Notepad samples have simply been ported to System.Data.SQLite (which involved very few changes).

The Tasky sample goes a bit further and adds a very simple ability to sync the task list.

Evolve

I am in Austin for Xamarin's Evolve conference, which starts today. Xamarin is doing some of the coolest stuff in mobile development today. We are thrilled to be a part of it.

"Enterprises have invested billions in certain databases and development frameworks and APIs. They also have invested in people who are trained for these systems."

This is exactly why we are so excited about the future here at Zumero.

In the enterprise arena, the words "data" and "SQL" are nearly synonymous. Sure, new stuff like "big data" and "NoSQL" are gaining traction, but they have a huge amount of ground to cover before they catch Oracle, IBM and Microsoft.

Some people believe the biggest competitor for SQL is SQL. The old, established SQL vendors I just mentioned are facing new challenges from the so-called "NewSQL" vendors like VoltDB, NuoDB, and Google Spanner.

The business world loves SQL. If you think the previous statement is BS, you're spending too much time with the "early adopter" part of the bell curve.

Broadly speaking, many enterprise companies have not really started developing for mobile devices. Some have done pilot projects. Few if any would say that they are at full steam on anything "mobile".

But this transition is happening. And it is accelerating. And as it does, we will see more and more developers asking how they can build a mobile solution which (1) integrates with their SQL database, and (2) leverages the SQL skills of their team. SQL is what they have. SQL is what they know.

We think that Zumero is going to have a great story for the customers in this wave:

• On the device, SQLite. Fast, robust, and familiar. Already pre-installed on a billion phones and tablets.

• On the backend, Big SQL. In the flavor of your choice.

• Zumero handles the synchronization between the two.

• Your developers don't have to worry about networking. Your app will work offline or online.

• Your app needs the network only for sync, so your end-users never have to wait.

• Familiar APIs on both sides

• Server on-premises or in the cloud, whichever you prefer

Our vision for this is large. The incarnation of Zumero available today is just the beginning.

Mobile offline RSS reader, Part 3

Lots of new stuff. I've now got three separate executables:

• z_rss_create.exe -- For each feed that has not yet been initialized, create its dbfile and set the permissions on it.

• z_rss_update.exe -- Retrieve XML and store items for some feeds. The word "some" is defined as "all feeds that have never been updated" plus "up to five feeds that have not been updated in at least an hour".

• z_rss_sync.exe -- Synchronize the feed dbfiles up to the Zumero server.

All of these are part of the "admin" side of things. In other words, I still haven't written any code that is intended to run on a mobile device. So far, I'm only doing stuff that is intended to run as a cron job on a server somewhere.

C# Rocks

The Zumero core code is written in C, mostly because SQLite is written in C.

But most people don't actually use SQLite from C. Rather, they call it through some binding for their favorite higher-level language. Or they are even further away, and they're not calling SQLite at all, because some sort of ORM is calling it for them.

I'm probably as much a fan of C as anybody. I've had love affairs with a lot of different programming languages, but C is the one I always come back to.

Still, there's something nice about writing this:

string sync_result = db.ExecuteScalar(
    @"SELECT zumero_sync(                            
        'main',                             
        ?, 
        ?, 
        zumero_internal_auth_scheme('zumero_users_admin'), 
        'admin', 
        ?
        );", 
    server_url, 
    dbfile_name_for_this_feed, 
    password
    );

Instead of this:

int rc;
char* sync_result = NULL;
sqlite3_stmt* pStmt = NULL;

rc = sqlite3_prepare_v2(
    db, 
    "SELECT zumero_sync("
    "'main',"
    "?,"
    "?,"
    "zumero_internal_auth_scheme('zumero_users_admin'),"
    "'admin',"
    "?"
    ");", 
    -1, 
    &pStmt, 
    NULL); 
if (rc) goto fail;
rc = sqlite3_bind_text(pStmt, 1, server_url, -1, NULL); 
if (rc) goto fail;
rc = sqlite3_bind_text(pStmt, 2, dbfile_name_for_this_feed, -1, NULL); 
if (rc) goto fail;
rc = sqlite3_bind_text(pStmt, 3, password, -1, NULL); 
if (rc) goto fail;
rc = sqlite3_step(pStmt);
if (rc != SQLITE_ROW) goto fail;
sync_result = strdup(sqlite3_column_text(pStmt, 0));
sqlite3_finalize(pStmt);
pStmt = NULL;

// use sync_result

free(sync_result);
sync_result = NULL;

I like C, but I like higher-level languages too.

In fact, my original plan was to use C# for the first part of these series and then do the subsequent parts in Python or whatever. But I forgot how much I like C#. I'm comfy in this particular chair, and I don't feel like getting up. So for now, Python will have to wait.

Why am I putting each feed in its own SQLite file?

If you stop and think about it, it's really amazing how much power is packed into today's smart phones.

As I briefly mentioned in the post where I announced Zumero, our company got its start by doing mobile stuff. Way back in 1997, we were one of the many contractors working on a Motorola project to build one of the first smart phones. That device had a CPU that would be considered glacially slow by today's standards. The idea of storing movies or songs wasn't even considered, since WiFi and 3G didn't exist yet, and the amount of flash storage on the phone was a number that currently rounds to zero.

In relative terms, the iPhone 5 is like a supercomputer.

But bandwith and on-device storage are still precious commodities, and mobile apps need to pay careful attention to how these resources are used.

So why am I putting each feed in a separate SQLite file? Because not everybody wants every feed, and I don't want to waste bandwidth and storage on things that the user is not going to read.

If you're building a mobile app on Zumero (or any other replicate-and-sync database platform), it is worth thinking about the best way to structure your data such that each device can get exactly the stuff it wants.

SQLite Rocks

Splitting things up sounds like it would cause a lot of hassle, but it turns out that this is one area where SQLite shines. By using the ATTACH statement, SQLite allows you to work with multiple database files from a single connection handle.

For example, in the "all_feeds" database, I have a table called "about", which is where I store the title of the feed. I am also using the presence of a title as a flag which indicates whether I have created the dbfile for the feed's content or not.

So, when somebody adds a new feed to the list, I need to:

• In the per-feed dbfile (which I need to create), create tables for storing content and permissions.

• In the "all_feeds" dbfile, insert a row for the title of the feed.

These two operations need to happen in two different SQLite databases. But like I said, SQLite makes this easy. Take a look at some snippets from z_rss_create.cs:

I already have SQLiteConnection db as the SQLite handle for my "all_feeds" database. To do the two steps above, I first ATTACH the new dbfile into this handle, giving it the name "cur".

db.Execute("ATTACH ? AS cur;", dbfile_name_for_this_feed);

Now I can perform SQL operations on either database by simply qualifying the table name with "cur." (for the new dbfile for this feed) ...

db.Execute(
        @"CREATE VIRTUAL TABLE 
        cur.items 
        USING zumero
        (
          id TEXT PRIMARY KEY NOT NULL, 
          title TEXT NOT NULL,
          summary TEXT NOT NULL,
          pubdate_unix_time INTEGER NOT NULL
        );"
        );

... or "main." (for the "all_feeds" db).

db.Execute("INSERT INTO main.about (feedid, title) VALUES (?,?)",
        q.feedid,
        f.Title.Text
        );

Furthermore, by placing both of these operations inside an explicit transaction, the two steps become atomic.

This same technique is also used in z_rss_update.cs where I need to (1) store the actual content of the feed in its dbfile, and (2) update "all_feeds"."last_update" to remember when we retrieved the the XML for that feed.

More to come in part 4...

eric$ ./sqlite3
SQLite version 3.7.15.2 2013-01-09 11:53:05
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> SELECT zumero_sync(
    'main',
    'https://zinst393e9343b87.s.zumero.net',
    'all_feeds'
    );
0;0;0;3584;0;476;213;217

sqlite> SELECT * FROM feeds;
2|http://ericsink.com/rss.xml
1|http://feeds.hanselman.com/ScottHanselman
3|http://rssnewsapps.ziffdavis.com/PCMAG_dvorak.xml

Yes, I left the permissions wide open, specifically so that you could add to the list.

But... Dvorak?

Really?

Mobile offline RSS reader, Part 2

In part 1, I got started with a single 'feeds' table that contained nothing but the URL. Now I want to add a place to store the title of the feed, and a place to remember the last time I retrieved the XML.

CREATE VIRTUAL TABLE IF NOT EXISTS about USING zumero(
    feedid INTEGER UNIQUE NOT NULL REFERENCES feeds (feedid),
    title TEXT NOT NULL
    );

CREATE VIRTUAL TABLE IF NOT EXISTS last_update USING zumero(
    feedid INTEGER UNIQUE NOT NULL REFERENCES feeds (feedid),
    when_unix_time INTEGER NOT NULL
    );

Note that unlike the feeds table, these two tables are protected from changes by unauthenticated users. Let's take a closer look at the SQL statements from part 1 where I configured the Access Control List (ACL).

First, I use a convenience function to create the ACL table itself, which, by convention, is named "z_acl".

SELECT zumero_define_acl_table('main');

Then it is time to insert several ACL entries, each of which is just a row in the z_acl table. Here is the first one:

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    '',
    '*',
    zumero_named_constant('acl_result_deny')
    );

This entry establishes a default. Basically it says, "don't let anybody do anything".

• The 'who' column specifies which users this will match at permission-checking time. An entry that specifies 'anyone' will match, well, anyone at all, whether they are authenticated or not.

• The meaning of the 'scheme' column depends on the value of the 'who' column. In this case, 'scheme' is not needed, so it is an empty string.

• The 'tbl' column is also not used in this entry, so it is an empty string as well.

• The 'op' column specifies which operations this will match at permission-checking time. The asterisk here is interpreted as a wildcard.

• Finally, the 'result' column is used to provide a result for the permission-check when this entry has matched, which it always will.

When checking for permissions, entries are checked in order from most-specific to least-specific. The ACL entry above will always match, but if something else matches first, it will not apply. The following entries are the exceptions, the cases where we want to actually allow something.

The following entry says, "people who are members of the admin group are allowed to do anything they want":

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    zumero_internal_auth_scheme('zumero_users_admin'),
    zumero_named_constant('acl_who_any_authenticated_user'),
    '',
    '*',
    zumero_named_constant('acl_result_allow')
    );

For many situations, I would just stop here and add no further entries to the ACL. Only the admins can do anything.

But in this situation, I specifically wanted to allow unauthenticated users to have a couple of permissions. This one allows 'read' access ('pull') to the entire dbfile:

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    '',
    zumero_named_constant('acl_op_pull'),
    zumero_named_constant('acl_result_allow')
    );

And the following ACL entry is the one that isn't looking so wise in the hindsight of the moment:

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    'feeds',
    zumero_named_constant('acl_op_tbl_add_row'),
    zumero_named_constant('acl_result_allow')
    );

Note the value for the 'tbl' column in this final ACL entry. I've granted 'add_row' permission to 'anyone', but only for the 'feeds' table. This is why the two new tables I just created will remain safe from changes by non-admin users.

These permissions are enforced at the server not on the client. When performing operations on the local copy of the SQLite database, you can do anything you want. But when you sync, if you are attempting to push changes which include operations beyond your permissions, your push will fail with "zumero:permission_denied".

Stay tuned for part 3.

I hate being excluded from what all the cool kids are doing, so I'm going to build an RSS reader which (1) is designed for mobile devices, and (2) has offline support.

An RSS Reader built on SQLite/Zumero, Part 1

The basic design is to use one SQLite file to keep a list of all the feeds, plus one additional dbfile for each feed.

Here's the SQLite script to create the 'all_feeds' database:

.load zumero.dylib

.echo ON

BEGIN TRANSACTION;

CREATE VIRTUAL TABLE feeds USING zumero(
    feedid INTEGER PRIMARY KEY,
    url TEXT NOT NULL UNIQUE
    );

-- configure the permissions on this dbfile to allow 'anyone' to 
-- (1) pull the dbfile, and 
-- (2) add rows to the feeds table  
-- and nothing else.

SELECT zumero_define_acl_table('main');

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    '',
    '*',
    zumero_named_constant('acl_result_deny')
    );

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    zumero_internal_auth_scheme('zumero_users_admin'),
    zumero_named_constant('acl_who_any_authenticated_user'),
    '',
    '*',
    zumero_named_constant('acl_result_allow')
    );

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    '',
    zumero_named_constant('acl_op_pull'),
    zumero_named_constant('acl_result_allow')
    );

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    'feeds',
    zumero_named_constant('acl_op_tbl_add_row'),
    zumero_named_constant('acl_result_allow')
    );

INSERT INTO feeds (url) VALUES ('http://feeds.hanselman.com/ScottHanselman');

COMMIT TRANSACTION;

SELECT zumero_sync(
    'main',
    'https://zinst393e9343b87.s.zumero.net',
    'all_feeds',
    zumero_internal_auth_scheme('zumero_users_admin'),
    'admin',
    'SECRETPASSWORD'
    );

Highlights:

• The ".load zumero.dylib" loads the Zumero SQLite extension so that I can use Zumero features in SQLite.

• The main thing going on here is the statement where I CREATE the feeds table. And it's a very simple table. All I need is a URL for the feed.

• Then you see a bunch of stuff to configure permissions. As it says in the comments, I want unauthenticated users to be able to see the feed list or add a feed, but all other modifications are prohibited unless you are, well, me.

• I seed the list with the feed from Hansleman's blog.

• Finally, I sync this SQLite database up to my server. Since I am creating a dbfile and defining its schema, I need to use the admin user/password that was created when I signed up for this Zumero server.

To execute this script, I save it to a file called setup_dbfile_all_feeds.sql and then pipe it into the sqlite3 shell.

eric$ ./sqlite3 ./all_feeds.db < setup_dbfile_all_feeds.sql

Like I said, anyone can add feeds to this list. The server URL is actually the one shown in these examples. I'm going to open my local copy of "all_feeds" and add my own RSS feed URL to the list:

eric$ ./sqlite3 ./all_feeds.db 
SQLite version 3.7.11 2012-03-20 11:35:50
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .load zumero.dylib
sqlite> INSERT INTO feeds (url) VALUES ('http://ericsink.com/rss.xml');
sqlite> SELECT * FROM feeds;
1|http://feeds.hanselman.com/ScottHanselman
2|http://ericsink.com/rss.xml
sqlite> SELECT zumero_sync(
    'main',
    'https://zinst393e9343b87.s.zumero.net',
    'all_feeds'
    );
0;0;3584;0;448;0;1249;1264

Note that I didn't need to pass any authentication credentials to the zumero_sync() function.

So far, my RSS reader doesn't do anything. It's just a list of feeds. Granted, it's a really cool list, since it supports incremental sync, but still. I'm probably going to need my RSS reader to do something with, er, RSS. And for that, I'm going to need more than just SQL statements piped into the sqlite3 shell.

Here's my starting point, in C#:

using System;
using System.Collections.Generic;
using System.ServiceModel.Syndication;
using System.Xml;

using SQLite; // https://github.com/praeclarum/sqlite-net

namespace z
{
    class Program
    {
        // define a little class to represent rows of the feeds table

        public class feed_row
        {
            public string feedid { get; set; }
            public string url { get; set; }
        };

        public static void Main (string[] args)
        {
            // open the local SQLite db
            SQLiteConnection conn = new SQLiteConnection("all_feeds.db");

            // tell SQLite to allow load_extension()
            conn.EnableLoadExtension(1);

            // load the Zumero extension
            // this is the equivalent of 
            // ".load zumero.dylib" in the sqlite3 shell
            conn.ExecuteScalar("SELECT load_extension('zumero.dylib');");

            // iterate over all the rows in the feeds table
            var rows = conn.Query ("SELECT feedid, url FROM feeds;");
            foreach (feed_row q in rows)
            {
                Console.WriteLine(q.url);
                try
                {
                    XmlReader xr = new XmlTextReader(q.url);
                    SyndicationFeed f = SyndicationFeed.Load(xr);

                    // TODO store the items from this feed
                }
                catch (Exception e)
                {
                    // TODO failed trying to retrieve or parse this feed.
                    // TODO log the failure?
                    // TODO delete the feed row?
                    // TODO launch nethack?
                }
            }

            conn.Close();
        }
    }
}

In order to call the SQLite library from C#, I'm using SQLite.cs, by Frank Krueger. That wrapper doesn't have a way to call sqlite3_enable_load_extension(), so I added one. Here's the diff, which I plan to submit as a pull request:

eric$ diff orig_SQLite.cs SQLite.cs 
183a184,192
>         public void EnableLoadExtension(int onoff)
>         {
>             SQLite3.Result r = SQLite3.EnableLoadExtension(Handle, onoff);
>           if (r != SQLite3.Result.OK) {
>               string msg = SQLite3.GetErrmsg (Handle);
>               throw SQLiteException.New (r, msg);
>           }
>         }
> 
2645a2655,2659
>       [DllImport("sqlite3", 
>                 EntryPoint = "sqlite3_enable_load_extension", 
>                 CallingConvention=CallingConvention.Cdecl)]
>       public static extern Result EnableLoadExtension (IntPtr db, int onoff);
> 

I'm using System.ServiceModel.Syndication.SyndicationFeed to do the RSS parsing for me. I don't have any previous experience with that library, but so far, it just works.

The statement I used to compile this program is:

eric$ gmcs -reference:System.ServiceModel.Web.dll \
      -out:z.exe AssemblyInfo.cs SQLite.cs Main.cs
...
Compilation succeeded - 3 warning(s)

Since I'm running on Mac OS X (Lion), I had to bring my own SQLite to the party. On Mac OS, the sqlite3 library and shell are preinstalled with the OS, but they were compiled without support for dynamic extension loading. If you're on a Mac, it's easy to grab and build your own copies:

eric$ curl --silent --remote-name \
      http://www.sqlite.org/sqlite-amalgamation-3071502.zip

eric$ unzip sqlite-amalgamation-3071502.zip 
Archive:  sqlite-amalgamation-3071502.zip
   creating: sqlite-amalgamation-3071502/
  inflating: sqlite-amalgamation-3071502/sqlite3.h  
  inflating: sqlite-amalgamation-3071502/shell.c  
  inflating: sqlite-amalgamation-3071502/sqlite3ext.h  
  inflating: sqlite-amalgamation-3071502/sqlite3.c  

eric$ mv sqlite-amalgamation-3071502/* .

eric$ rmdir sqlite-amalgamation-3071502

eric$ clang -dynamiclib -arch i386 -arch x86_64 \
      -o libsqlite03071502.dylib sqlite3.c

eric$ clang -o sqlite3 -arch i386 -arch x86_64 sqlite3.c shell.c 

eric$ ./sqlite3 --version
3.7.15.2 2013-01-09 11:53:05 c0e09560d26f0a6456be9dd3447f5311eb4f238f

I also created a .config file because SQLite.cs does DllImport("sqlite3", and need to map that to the SQLite library we just built.

<configuration>
    <dllmap dll="sqlite3" target="libsqlite03071502.dylib" />
</configuration>

Running the program yields the expected output:

eric$ mono ./z.exe
http://ericsink.com/rss.xml
http://feeds.hanselman.com/ScottHanselman

So, after all that, this code STILL doesn't really do anything useful. It demonstrates that I can successfully use SQLite and Zumero from C#. And it retrieves and parses each feed. But I have not yet implemented any of the things that are supposed to happen next. Hopefully I'll make more progress in part 2.

What is Zumero?

To describe Zumero, we first describe SQLite.

SQLite is a lightweight (but surprisingly powerful) implementation of a SQL database.

SQLite is the standard database software for iOS, Android, and Windows RT. It is installed on over a billion mobile devices.

But like any other computer, and perhaps more so, a mobile device is not isolated. It needs to share data with a server.

And SQLite has no synchronization capabilities.

Zumero solves that problem.

Zumero is "sync for SQLite".

No, really, what is Zumero?

It's a better database platform for mobile.

Mobile apps need to talk to the cloud about data.

A lot of people are building mobile apps that use the "remote procedure call" model. Every user operation requires a network request to the server.

We think "replicate and sync" is a better model. The app can interact with a local copy of the database instead of going through the network. Synchronization can be handled entirely in the background.

Putting network activity in the background brings two big wins:

• The developer wins because networking issues are isolated in a small part of the app. The real features of the app can be the focus of the developer's attention, and are easier to code.

• The user wins because the app is faster, more responsive, and works offline.

Consider a typical situation where a mobile app responds to a user action by filling a list box with the results of a database query. Neither the developer nor the user wants the network involved between the button tap and the refresh of the display. Mobile networks are just not consistent enough. Even a perfectly-working LTE connection usually has enough latency to make the app feel sluggish.

Oh good grief. Eric, I thought you were a geek? Tell us what Zumero is.

On the client, Zumero is a SQLite extension. The server is built on node.js.

The client tries very hard to be invisible to the developer. We didn't want to introduce a whole new API for everybody to learn, so developing with Zumero isn't much different from SQLite. Porting an existing SQLite app to use Zumero is usually straightforward.

The SQLite extension uses SQLite's "virtual tables". A Zumero table mostly works just like a regular table, except that it can be synced.

The Zumero server manages the central, authoritative copy of the database and handles all the details of synchronization, merging changes, conflict resolution, caching, logging, authentication, and permissions.

Why SQLite?

1. Because it's an incredible piece of software. If you're not familiar with SQLite, give it a serious look. It has come so far that keeping the word "Lite" in the name seems inappropriate. SQLite is fast, robust, and far more capable than most people know.

2. Because it's everywhere. Every smart phone and tablet has SQLite inside. Even Microsoft has given up the notion of shrinking SQL Server and decided to standardize on SQLite as its recommended database for Windows RT.

3. Because it needs a sync feature. The real question here is not "Why SQLite?" but rather, "Why NOT SQLite?" Why should people consider anything except SQLite for dealing with mobile data? The main answer is obvious: SQLite doesn't sync. Zumero was created to fill that need, not by introducing a new mobile database system, but rather, by adding painless synchronization features to the high quality database that is already there.

What platforms do you support?

Right now (March 2013), the Zumero Client SDK includes support for iOS and Android. WinRT is planned. Blackberry is not, but that might change if there is evidence of customer demand.

We will also be supporting two of the cross-platform mobile solutions: Apache Cordova (aka Phonegap) and Xamarin. We've made significant progress on both of these, but neither of them was ready for the initial release.

The client SDK also includes support for (regular) Windows and Mac OS. This may seem odd, since Zumero is all about mobile. But very few so-called mobile apps are accessed exclusively from mobile devices. You may (and probably will) want the ability to also access your data from web apps, desktop applications, or even other servers.

Even if you don't need any of these things, the SQLite command-line shell is a handy way to perform administrative tasks for your Zumero server, such as initial setup of your schema, configuring permissions, or analyzing logs.

How does Zumero store data on the server?

The Zumero server has a plugin architecture designed to support a variety of SQL databases. We currently have two implementations of the database backend: SQLite and PostgreSQL.

At the present time (still March 2013), the Zumero server is available as a cloud-hosted service with the SQLite backend, but it will be made available in other flavors later.

What about server platforms?

From a strictly technical perspective, the Zumero server runs just fine on Linux, Mac, and Windows.

The current cloud-hosted offering is Linux, running in the Amazon cloud.

In the future, we will make the server available in three ways: in the cloud, on-premises, or as a desktop server for development purposes. Not all combinations of platform and deployment are likely to be interesting. Things will probably end up something like this:

Is Zumero open source?

No.

Who are Zumero's competitors?

As always, that depends on how narrowly you define "competitor". :-)

• At one extreme, we are not aware of anybody doing EXACTLY what we are doing.

• At the other extreme, everybody doing anything with data and mobile is a competitor.

If I had to name just one other player, it would be Couchbase Lite. Couchbase founder J. Chris Anderson recently did a great blog entry called Why Mobile Sync?. His tune sounds a lot like the one we're singing about Zumero.

So it wouldn't be outrageous to describe Zumero by saying, "It's like Couchbase Lite. Except it's SQL".

SQL? I thought SQL was dead?

Hardly. The NoSQL offerings are gaining traction, but SQL still has far greater market share, especially in the enterprise.

How does Zumero compare to Parse/Kinvey/Stackmob and their ilk?

We don't really think of Zumero as a "Backend as a Service" (BaaS) company, since we're focused on the database part of the story.

But if you ignore all the other mobile backend stuff (like push notifications), it would be reasonable to describe Zumero by saying, "It's like Parse/Kinvey/Stackmob. Except it's a replicate-and-sync model. And it's SQL."

Is Zumero part of SourceGear?

No. Yes. Sort of.

In terms of market positioning, Zumero is separate. It does not share SourceGear branding. It has no particular connection with SourceGear's products.

But financially, Zumero is currently part of SourceGear. (This approach is similar to what we did with Teamprise, which we sold to Microsoft in 2009.)

Corey and I think of Zumero as our new startup, which is being incubated inside SourceGear.

Mobile data? Really? How is this related to version control?

It's not.

But the genesis of Zumero happened when we did the iPad app for Veracity. We liked the "replicate and sync" model for mobile so much that we decided to take the databasey part of Veracity and productize it as a platform for building all kinds of mobile apps. Along the way, the technology evolved a lot, to the point where it doesn't look much like the original Veracity code, but that's what we started with.

This is actually somewhat of a return to our roots for SourceGear. We spent a lot of our first three years doing mobile development as contract work for Motorola. Of course, today's mobile devices are just a little more powerful than that phone we worked on back in 1997. :-)

What does this mean for SourceGear's other products, Vault and Veracity?

They are continuing to move forward. Vault 7 is underway and aiming for a Q3 release. Veracity 2.5 is done and will be released sometime in the next two weeks.

How can I try Zumero or find out more about it?

http://zumero.com/get-started

Public discourse on this topic tends toward the extremes. I participated in a recent discussion which started with someone worrying that his company will have to support Microsoft Surface in addition to Android and iPad, and ended with someone saying, "Or you could use HTML5 and get all three platforms for free."

Marc Andreessen, an influential bishop of the web, has said: "The application model of the future is the web application model. The apps will live on the web. Mobile apps on platforms like iOS and Android are a temporary step along the way toward the full mobile web. Now, that temporary step may last for a very long time. Because the networks are still limited. But if you grant me the very big assumption that at some point we will have ubiquitous, high-speed wireless connectivity, then in time everything will end up back in the web model. Because the technology wants it to work that way."

It seems like most people look at this situation and see black or white. I tend to see shades of gray.

The question

The main question in play here is: How thick should clients be?

Let me define my terms:

• I define the Client as the thing which is used by exactly one user, which interacts directly with that user, and which is probably physically close to that person.

• I define the Server as the thing which is shared by multiple users, which interacts directly with the Client, and which could be physically located anywhere.

• I define the Pipe as the connection between the Client and the Server.

• I define the notion of a Thick client as a relative term. Thicker clients have more app-specific code and are less dependent on the Server. Thinner clients leave more of the app-specific work to be done on the Server.

There are two main variables in decisions about the thickness of clients:

• The quality of the Pipe: This includes bandwidth, latency, availability, reliability, and cost

• The Client side costs: This includes cost of hardware, software development, deployment, upgrades, and maintenance.

And, there are two laws which apply:

• As the quality of the pipe goes up, the client can get thinner.

• As the client side costs go down, the client can get thicker.

The Extremes

If the Pipe quality were perfect, the client could be "wafer thin".

A Pipe with perfect quality would have infinite bandwidth and no latency. It would be available everywhere. It would be 100% reliable. And it would cost nothing.

If the Pipe were perfect, the Client would be almost unnecessary, needing to provide nothing more than a physical presence for the Server.

On the other hand, if the Client side costs went to zero, the client can be as thick as we want.

A Client with zero costs would be constructed from physical components that cost nothing. It would contain software that cost nothing to build, which could be installed and upgraded and maintained with exactly zero cost or hassle or effort.

If the Client side costs were zero, there would be no need for the Server, except to facilitate communication between Clients.

And why does the thickness of Clients vary?

Because varying Client thickness creates opportunities to compete in the marketplace.

You can differentiate yourself from your competition by making your Client thicker (more expensive) and talking about performance and a better user experience and stuff like that.

Or, you can differentiate yourself by making your Client thinner (less expensive) and talking about lower Total Cost of Ownership and easier installation and stuff like that.

This issue is not new

Back in the 1960s and 1970s, when we only had mainframes and minicomputers, there was a distinction between smart terminals (thick clients) and dumb terminals (thin clients).

In the 1980s, we got workstations (really expensive thick clients, purchased by people who perceived them as cheap compared to the mainframes and minis) and microcomputers (far less expensive thick clients, purchased by people who previously didn't have a computer at all).

In the early 1990s, the high cost of workstations gave rise to X terminals, thin client devices which couldn't do much more than display the graphical user interface. My manager bought one of those fancy new 19.2k modems and actually tried doing Motif widget development from home.

In the mid 1990s, web browsers appeared. For a very brief time, this technology was regarded only as a way to collaborate on hypertext documents. This phase of the web lasted for most of an afternoon. Meanwhile, back in Champaign, Illinois, the Unsung Hero and His Eminence were busy building a web browser which had more "stuff" in it. What kind of stuff? The sort of stuff that made web browsers into a platform for delivery of apps. And the technologies of the web have been moving primarily in that direction ever since.

• Java applets (developed a fatal disease called Swing)

• ActiveX (declared dead seven years after it went missing)

• Flash (murdered by Steve Jobs)

• Silverlight (murdered by HTML5)

In the late 1990s, people (Oracle, I think?) tried to sell something called a Network Computer. It was a little PC with a video card, some RAM, an ethernet card, a web browser, and no hard disk. Thin.

Microsoft tried to murder the infant web. Then they tried to be its Mom. Then Bill Gates retired and I haven't been able to figure out Microsoft's strategy ever since.

While Microsoft was trying to cling to Windows as a thick client solution, Citrix was trying to make it thinner, building multi-user mainframe-style computing for Windows. The medical clinic here in Champaign deployed Citrix throughout its network, which explains the increased latency of my doctor, but his fees didn't go down.

HTML5 arrived. Actually, the spec is still a long way from being finalized, but nobody knows that. People needed a name, so they started saying "HTML5" before it was fully cooked. Common usage of the term "HTML5" is actually fairly accurate, at least compared to the way telecom companies use the term "4G".

And now, this war has moved to the battlefield of mobile. Smartphones and tablets.

Black and white

As I said above, people exhibit a black-and-white mentality about this issue.

In part, this is because people who make polarizing predictions tend to sound more visionary. In some situations, being inspiring is far more important than being correct. Venture capitalists in a bubble need to appear visionary so they can attract the deals with the most buzz, because those are the companies that can be sold quickly.

Another reason that people like to hear black-or-white predictions about the future is that it makes them feel better. Uncertainty is uncomfortable.

Broadly speaking, nobody likes articles like this one, essays which claim that the world is defined in shades of gray. This is why you stopped reading two scroll bars ago.

But on this issue, the black-and-white predictions are simply wrong.

I may not be inspiring, but I'm right

The quality of the pipe will continue to go up. In 1992, I was using a 14.4 modem. In 2012, the slowest network connection in my life is hundreds of times faster and far more reliable.

The client side costs will continue to go down. In 1992, I was using an HP Snake 730 workstation. In 2012, the hardware specs of my iPad are hundreds of times better in every category, and it cost 97% less.

Neither of these technologies is like batteries, which basically have not advanced since Kennedy was president. Both sides just keep getting better, steadily, year after year, with no end in sight. Individual battles are won and lost, but the battle rages on. Neither side has gained a permanent and decisive technological advantage over the other. And if you think that's going to happen in your lifetime, I think you're wrong.

Inspiring, perhaps. But still wrong.

Bets

Nonetheless, we must place our bets on one side or the other. There's a lot of money to be made in being on the winning side of these little battles.

Furthermore, my use of the "war" metaphor does not bring issues of national loyalty or citizenship into play. People who bet on thick clients in the last battle are free to bet on thin clients in this one.

Because of my earlier experience, folks tend to think I should always be a "web apps" guy. In the early days of the web browser, I was there. I wasn't nearly as successful as Mr. Andreessen. Marc was above the title. My name was after the fade to black, on the third page of the credits, and it took me three takes to get my line right. But web technologies are a big part of my career. I'm a believer, and I am an appreciative user of many of the web apps which have changed the world.

Nonetheless, for this round, I'm betting on native apps, for three reasons:

• Recent declines in Client side costs. For example, the App Store makes a huge difference in issues of installation and upgrades.

• Current problems with quality of the Pipe. Users of smartphones and tablets have high expectations regarding the quality of the user experience.

• My own preference. I'd rather spend my time creating products that delight users. Wal-Mart may be successful, but the goal of making everything cheaper just doesn't look like much fun.

So that's why I pushed SourceGear into shipping a native iPad app for onVeracity.com.

When I first started exploring this path, somebody on the team asked me, "Why can't the iPad people just use the web app?"

Like I said then, web apps and native apps can and will coexist.

But native apps are just better. They always have been. That's why they cost more.

I underestimated the iPad

I started out very skeptical of the iPad. It was almost as portable as a phone, and almost as capable as a laptop. I counted too many occurrences of the word "almost" and assumed it would be a tweener product that just didn't have a place in the world.

In practice, it turns out this device takes a whole bunch of stuff that I used to do with a computer and lets me do those things ANYWHERE (such as when I'm sitting outside the school waiting for my daughter's cheer practice to end). And that means I do them more often.

Compared to my pre-iPad life, I now read more books. I tweet [a little] more. I check email more. I play games more. I more actively plan my time.

No other technology purchase in my life has had this much impact.

Two kinds of apps

During the thousands of hours I have spent using my iPad, I have found myself partitioning apps into two categories:

• Apps I can only use with a network connection (Safari, Words With Friends, Zillow, Maps, TweetBot, etc.)

• Apps I can use truly ANYWHERE (iBooks, Note Taker HD, Pages, Civ Rev, etc.)

My current iPad has Verizon LTE. Prior to that, I had 3G models. My home, my office, and my coffee shop all have WiFi. If the result of all this wireless Internet were a perfectly reliable and consistent connection, available everywhere I go, then I would have no reason to group my apps into two categories. But that is not the reality of mobile Internet in 2012:

• There are dead spots in LTE and 3G coverage.

• When I have LTE/3G, it is usually awesome/decent, but sometimes I get latency of 2-3 seconds, for no apparent reason.

• I went very quickly from being amazed at how many places offer WiFi to being shocked at how many places do not. In my case, this includes the mall, the church foyer, and the auditorium at my daughter's school.

• I go to hotels and conferences where the WiFi routers are happy to give me a connection to a pipe which is hopelessly overloaded

So, I have a bunch of apps which require me to think about the availability of a wireless Internet connection. At best, this is a distraction. At worst, it is an obstacle.

At this point, I'll poke fun at myself for complaining about #firstworldproblems. I'm thankful for mobile Internet. Really, I am. It is one of the coolest inventions in the history of the world. But I have come to have a somewhat different kind of appreciation for apps I can use without giving a thought to connectivity issues.

I warned you about this part

As I write this, I'm at #agile2012 in Dallas. In the SourceGear booth, we are showing off our new iPad app for onVeracity.com.

onVeracity.com is our hosting service for Veracity. Free accounts can have unlimited private repositories and up to five users. Paid accounts are $20/user/month.

Veracity combines distributed version control (like git) with bug tracking and wiki, which are also built on the distributed model. It is open source (Apache License 2) and cross-platform (supported on Mac, Linux, and Windows).

Because Veracity's architecture means that everything is distributed (not just version control), our iPad app supports full functionality while offline.

Query bugs:

Add bugs

Modify bugs

View wiki pages:

Edit wiki pages:

Browse changes to the source code:

See the details of any version control changeset:

Look at code diffs:

View burndown charts:

View summaries of recent activity:

And all of those things work without wireless Internet. I can put my iPad in airplane mode and keep working. I only need a network connection when I want to push and pull changes to/from the server:

After seeing a demo of what our iPad app can do, one of the visitors to our booth reacted by saying, "I hate tools. But that is compelling."

More changes

Just like the iPad itself, this app has made my daily life a bit different:

• My bug list is more up to date

• I read and review more code

• My backlog is cleaner

• I update my estimates more often

• I pay more attention to the project documents in our wiki

Try it

The Veracity iPad app is free and available in the App Store now. It works in conjunction with onVeracity.com, which is free for up to five users.