•

tech

Notes on local first

Some notes on the theory and practice of local-first apps. Everything that you should consider, with links to resources for each topic, including example implementation strategies.

Sandro Maglione

Sandro Maglione

Software development

Local-only

Only storing files on the device, not using the internet at all:

  • No collaboration
  • No way to sync changes between devices

Data is trapped on a single device.

Example of local-only apps are all software pre-internet. You would buy a CD and install the full software on your device. There was no way to collaborate on the same data, which was all stored locally on your device.

Many games are local-only as well. You download the game and play it on your own device with your own data.

Also, most mobile apps are local-only, with the full software and data stored on the device.

When the app is local-only you removed network latency completely from the UI, so you can read data from a local, single source of truth.

Mutations are synchronous and there is no complexity in handling conflicts from multiple sources.

Offline-first

Offline-first are apps that work offline without an internet connection, but the authority and source of truth is still on the server.

Acts like a "glorious" cache, allowing offline access only until the server is back online.

Not only browsing data (i.e. read operations) should work, but the user should also be able to create and update data (i.e. write operations) while in a disconnected state.

The data is still completely stored on the server. The app depends on the server and on the service provider. The user doesn't own the data since the client is not the primary data source.

Offline-first acts just like a cloud-based app when online (with optionally optimistic updates and local caching to improve the UX), and temporarily stores the data on the client when offline.

If the server updates some data in conflict with the client, the client is forced to rely on the state from the server.

If the service provider shuts down, the data is lost.

Local-first

Multi-device software with some sort of data syncing between devices (multiple users or single user with multiple devices).

Offline support such that the app can be used without an internet connection.

Independent of the software developer/provider, such that the app continues working even if the server shuts down.

Always available, collaborative (but private), and responsive.

Server?

Focus on the client, the server plays a mediator role to sync or backup data between clients.

A server can be a simple web socket that allows to connect and move data between clients.

The server can also apply some validation rules to make sure the shape of the data is valid (with optimistic updates on the client that can be reverted).

It can also handle side effects (e.g. payments, notifications, emails, etc.) for things where clients don't have direct access.

Database

Database inside the client: WASM (SQLite and Postgres).

Make the database reactive, such that operations are reflected on the client immediately.

Store each operation in a transaction queue that is sent to the syncing server and streamed to other clients.

"Two storage": Reactive local database (source of truth for views on the client) and transaction queue (stream updates to and from other clients).

Syncing must guarantee eventual consistency, such that all clients end up with the same state.

Another option is to store local operations and download the current state from the server. You then apply the local operations to the server state.

When you come back online, the client downloads a fresh copy of the document, reapplies any offline edits on top of this latest state, and then continues syncing updates.

Deletes are stored on each client that performed the operation.

If that client wants to undo the delete, then it’s also responsible for restoring all properties of the deleted objects, which are stored with the delete operation.

This system relies on clients being able to generate new object IDs that are guaranteed to be unique. This can be easily accomplished by assigning every client a unique client ID and including that client ID as part of newly-created object IDs (each client owns its operations).

Migrations

When the database is primarily on the clients, applying migrations is problematic. You need a way to sync the whole migration history just like the rest of the data.

You have N + 1 databases to keep in sync (with N = "number of clients"). Distributed database replication where clients can go offline at any time.

In an event sourcing architecture, is it possible to model migrations as events?

Ordering of events: Hybrid Logical Clocks

Goal: Deterministic ordering of events that doesn't rely on a centralized system such that it's possible to order events generated by separated systems.

A client keeps a log of its local events. During syncing with other clients, we need a way to merge logs from other clients in order. The order must be the same for all clients, such that the final state is deterministic and consistent across all clients.

What should you sync?

A major issue is deciding what to sync in a client from the server log:

  • Cannot sync all the log history every time
  • Avoid applying the same operations multiple times on the same client

Central question: How does a client know what messages to ask for?

Just syncing all the messages "since the last time you opened the app" is not enough. You have no guarantee that no messages "in the past" may be present in the log when you come back offline.

Say that another client (c-other) made a change at time t1 which is not yet uploaded to the server (offline).

Your client (c-you) goes offline at time t2 (t2 > t1), thinking that all the changes before t2 are already synced.

Then c-other comes back online and adds t1 to the log. Your client would ask for "everything uploaded after t2" and would not download the change made by c-other at t1, even if the change was relevant.

A Merkle Tree can help to check if a time-range of messages was synced without checking each message one by one.

Algorithms like Kafka log compaction can help to reduce the size of the log.

Authentication

For most local-first the fact that data is stored on the client can be enough to guarantee authentication.

That's the case when a user doesn't need to access information belonging to other users, or in cases when you work in a "circle of trust". In these cases, a simple one-time authentication code to access the data is enough.

When connecting to external services, the authentication is implicitly handled by the service provider (e.g. OAuth). As long as you have access to the service, the local-first app doesn't need other guarantees.

Service worker

Caches all the code needed to run completely on the client.

PWA to ensure offline support.

Web sockets

Open connections to sync server to stream data changes between clients.

You can store the log of changes on the client as well (cache), so that you don't need to connect to the server when the client reloads.

Implementation

The first requirement is a local storage. Some options are:

The API will interact and store data locally, no need of external asynchronous requests and all the problems with latency, network, error handling.

Local storage is enough to make a "local-only" app, where all the data is trapped on the device.

It also checks the requirements for offline-first and provider-independence, as long as the full source code is available without other requests.

The UI is independent of the data source, therefore there should be no difference on that layer from local-first or cloud-based apps.

Syncing

A key feature of local-first is collaboration. The app should connect with other clients to upload and download data changes.

This is where we encounter the main challenges. Each client made a set of changes: INSERT, UPDATE, DELETE. The objective is to share changes between clients, such that all clients (storage) end up in the same state.

Some options are:

  • CRDTs (Conflict-free Replicated Data Types)
  • Event sourcing (with CQRS (Command Query Response Segregation))
  • OT (Operational Transformation)

In this example let's focus on event sourcing with a local relational database (e.g. PGLite). The only operation to consider is INSERT, since we only add logs to the list of events.

You cannot perform destructive operations through event sourcing.

The app interacts with the database with usual SQL queries. Each query results in a new event being added to the log. We therefore have two data sources: the database and the log.

During syncing, the logs are sent out to the server to update other clients. The other client then downloads and applies the changes to its own database.

Key questions:

  1. How to apply the events such that the final step is deterministic and (eventually) consistent across all clients?
  2. Since the client cannot download the full log history, how we decide what to sync such that the client doesn't miss any event?

Applying the events can be done if we have a consistent ordering between separate clients. The ordering cannot rely on a centralized server and cannot trust the user local clock. A solution here are Hybrid Logical Clocks.

A solution for point 2 is to have a pointer on the client to the "last synced event". The pointer doesn't rely on the client clock, but instead it points to a log entry in the server history. The server history is ordered such that events are stored in order of upload (not in order of creation). The client can therefore request all the (relevant) events since the last pointer.

The "last pointer" can simply be the ID of the last synced event in the log (excluded events local to the client, which don't have the ID assigned by the server).

Other considerations are things such as authentication, data migrations, server storage.

Resources

👋・Interested in learning more, every week?

Timeless coding principles, practices, and tools that make a difference, regardless of your language or framework, delivered in your inbox every week.