Cloud vs local: where to put Space Invader mosaics?

👾 Where to put user data? There's the cloud way and the local way and what I really want is neither

My favorite unsolved software problem is this: where, oh where, do you put user data?

For example, I have this web app that allows you to track Space Invader mosaics you already “collected” on a map:

As you go around Paris and keep finding the mosaics 📸, your map fills in with those marked as already flashed:

It quickly becomes pretty important that the app doesn't lose the list of mosaics already marked as flashed.

So, where do we store the list of mosaics for each user?

The cloud way

Most apps store data in the cloud. That means on computers belonging to (or in practice, rented by) whoever runs the application.

This has a few advantages: the data is not lost when the user loses their device, and the users can access their data from multiple devices.

It also has a disadvantage for the user: the data is not, philosophically or practically, in their control. If the app shuts down tomorrow, the user may never recover the data.

What’s the alternative?

The local way

Back in the days before the cloud, software used to be local: apps (back then we were calling them programs) were reading and writing files on the user computer.

For the Space Invader map, I ended up using a modern equivalent of this paradigm. Thanks to the web storage API, my web app can store the list of space invader mosaics directly in the user browser. No need to create any accounts; the app just “remembers” your flashed mosaics.

But this has its own disadvantages. When the data is ours, it also means it’s ours to lose. If my phone breaks, or I reinstall my browser, the list of mosaics is gone. It also doesn’t synchronize between devices.

To work around this, I added a feature to import/export the mosaic list. This way each user can sync or backup the list manually, using their preferred method of storing files (Google Drive, Dropbox, whatnot).

This works and has the advantage of being simple.

But I keep thinking that there must be a better way.

The missing third way

Here’s how this should work: I’d love to have a personal data store that automagically keeps my local files in sync between devices.

This data store should be:

• local-first, so that all data modifications happens locally first, and are opportunistically replicated/synchronized later

• private, so that the cloud provider synchronizing my data doesn’t automatically see it (end-to-end encryption)

• integrated with all the computing platforms I use (my Android phone, my web browser, etc.)

This would allow all sorts of applications to just work locally, reading and writing local files. But the files would be synchronized between devices in a way that’s seamless and private at the same time.

Conclusion

Every artifact of our digital life lives somewhere.

• If it lives in the cloud, that’s nice because the data is backed up, and we can access it from multiple devices. But the same data is then at the mercy of whoever runs the service.

• If it lives locally, that’s nice because we’re in control of the data. But we then have to manually back it up and copy it around between devices.

A better way (automagic, end-to-end encrypted sync) is easy to sketch, but hard to build in practice.

In the meantime, I’m manually backing up my Invader list in Google Drive 🙃.

Postcard from Paris

Frank Turner played a great show last Friday 🥳. When the world outside gets worse, the music shows get better.

Take good care of yourself 🫶,
– Przemek