In modern software development, the cloud model is the standard architecture for building apps. It provides a simple solution with centralized storage and synchronization across devices. But it comes at the cost of complexity, privacy, and security.
#diagram
A new model emerged recently to address these problems: local-first. It moves the focus back to the client, with all the data stored locally on the user's device. Sync engines are often a key component of local-first apps, allowing to synchronize local data between devices.
But a sync engine is not enough to make an app truly local-first. Local-first is based on 7 ideals that go beyond data synchronization, including aspects like offline support, collaboration, and privacy.
Sync engines and local-first systems address the limitations of cloud-driven models in complementary ways, with sync engines enabling seamless synchronization and local-first offering autonomy, privacy, and offline functionality.
So what's real local-first then? How is a sync engine different from a centralized model? How does a sync engine fit in the ideals of local-first? Let's find out!
Problems of the could-driven model
The most common model of app development is server-first. All the data and backend logic is stored on the server, and the client is just a consumer.
Add example diagram of cloud-based architecture
This model has some obvious advantages:
- Centralized data storage
- Server logic and secret keys hidden from the user (security)
- Less code sent and executed on the client (performance and bundle size)
- Easier data migrations
A centralized model avoids most of the complexity caused by distributed systems. All the data is stored and managed in a single place, and there is no need to synchronize multiple conflicting clients.
The server is the main authority, and all changes are finalized there and served on all the clients.
Add example diagram of distributed systems architecture
This model has also tremendous costs, especially considering the capabilities of modern clients and user devices.
Problem 1: Performance
No matter how powerful and fast your device is, loading a web page still requires a roundtrip to the server. While your browser is busy sending all kinds of requests, the only option left to the client is showing a loading spinner.
Add diagram of loading spinner + network request
This is a major bottleneck for all clients and frameworks. As long as the data lives outside your device, you must pay the price of a network request every time you want to load something.
const request = async () => {
// π Between this...
const response = await fetch("/user/1");
const json = await response.json();
// π ...and this it takes time, and the user is waiting π€¨
return json;
}Solutions like partial pre-rendering, server components, Suspense and server-side rendering can only mitigate a problem that it's intrinsic to the client-server model.
Problem 2: Code complexity
A normal day in the life of a frontend developer is filled with request errors, validations, handling loading states and client effects.
const request = async () => {
// π Between this...
const response = await fetch("/user/1");
const json = await response.json();
// π ...and this many unexpected things can happen!
// - Losing connection
// - Server error
// - Invalid data
// - Timeout
//
// All things that the client must handle, increasing time and complexity π«
return json;
}These tasks are another consequence of the client-server model:
- API requests are asynchronous, so you need a way to handle a result that may not be ready yet
- Tapping into an external system required all kinds of validations and serialization
- Race conditions are lurking behind any kind of asynchronous logic and hard to predict and debug
- Caching is hard (hardest problem of computer science), but it's necessary to avoid unnecessary work and gain in performance
Problem 3: No offline support
In a model where the server is the central authority, offline support is often an afterthought.
Add diagram showing broken app with no connection, and use of cache as fallback
The client expects the data to come from the server. Having to store changes while offline and then syncing them when the connection is back introduces a new level of complexity that most software are unwilling to pay.
Problem 4: Privacy
A centralized model is also a privacy nightmare. Once you move your data to the server, you lost all the guarantees of privacy and security.
You don't really own your data anymore. Anything can happen to the cloud provider, and your information can be lost or stolen. Furthermore, most apps don't offer much support for exporting or deleting your data. At best, you are left with an unusable JSON file that you can't import anywhere else.
Promises of local-first
Local-first moves the focus back to the client. With modern technologies is possible to have all the components of a backend in the browser, database included.
This allows to move all the data and responsibilities to the client, with some clear advantages:
- Privacy and security (all local in the user device)
- Offline support by default
- No more complexity of the network asynchronous model
- Instant (even synchronous) operations, both read and write
This also removes all the complexity of a backend architecture. No need to manage and scale servers and databases, all the data and logic lives on the client.
"Why did you need the cloud in the first place? [...] You did not have a way for two clients to speak to each other. [...] But if we get rid of that, then you stop needing to pay for most of your cloud bills."
Kyle Simpson (Local First Podcast)
It's a win-win for both the user and the developer: a better user experience, private, secure, and offline by default, and a lot less complexity for the developer.
What's the catch then? Multi-device support, collaboration, and long-term storage are missing. We need to move from offline-only to a fully distributed model. This introduces a new set of challenges.
The role of a sync engine
When every client operates on his own copy of the data, the problem becomes how to synchronize changes between clients. This is a distributed system problem.
A distributed system is a group of devices each keeping its own copy of data and working independently, but with the need to share updates with each other and stay in sync.
Add diagram of problem with multiple independent clients
Anyone can change anything at all times, and each client expects its changes to be synchronized with the others. The objective is to propagate changes in such a way that all clients end up in the same state.
This synchronization should be fast, reliable, deterministic, and above all transparent to the user. Sync engines offer a solution to this problem.
A sync engine is responsible to collect all the changes made by the client, resolve conflicts, and propagate them to all other clients.
It acts as a thin server layer only responsible for synchronization. The client doesn't need to care about syncing, it can simply read and write to its local database. The database is the single source of truth, and the sync engine is responsible for keeping it in sync with the others.
The client can now operate directly on local data with a synchronous API, and the sync engine takes care of the rest:
const Todo = () => {
const { store } = useStore();
return (
<div>
<Checkbox onCheck={() => store.mutate(mutations.completeTodo(todo.id))} />
</div>
);
};Sync engine and local-first
A sync engine is a key component of any local-first apps, but by itself it's not enough to make an app truly local-first.
Modern apps use sync engines mostly for UX and DX reasons, without aspiring to build a complete local-first solution.
In fact, the local-first model goes beyond the advantages of a sync engine, and instead it's based on 7 ideals:
- Fast
- Multi-device
- Offline
- Collaboration
- Longevity
- Privacy
- User control
On top of that, the most recent definition of local-first proposed during localfirstconf 2024 in Berlin adds a new requirement: Server independence.
In local-first software, the availability of another computer should never prevent you from working:
- Multi-device/Multiplayer (not local-only)
- Offline support
- It works even if the app developer goes out of business and shuts down the servers
A sync engine achieves the ideals of a fast app, multi-device, and enables collaboration between users and devices. However, by itself a sync engine doesn't guarantee the other ideals.
Sync engines still rely on centralized servers, not preventing potential risks for data breaches and privacy.
They also fall short in empowering users to work offline, since most sync engines are not designed to store data on a local database. Over time, their reliance on specific server infrastructures can threaten data longevity, not giving users real ownership of their data.
In contrast, the ideals of local-first start with the userβs device as the source of truth, addressing these shortcomings in ways sync engines cannot. Sync engines fall short of the following ideals:
- Offline support
- Longevity
- Privacy
- User control
- Server independence
Ideal 1: Offline
A sync engine doesn't make offline support a default feature. Most apps use a sync engine to make the UI reactive as the changes are synced with the server. This model however still relies on the server as a central authority.
Add diagram of sync engine based on the server
As long as the primary source of the data is the cloud, offline support becomes just a caching workaround. This causes problems with offline changes, leading to all sort of inconsistencies when the connection is restored.
So much so that making change while offline became a meme, and users don't trust to touch the app while offline π
Ideal 2: Longevity
When the data storage is all dependent on a server architecture there is no long-term guarantee on the safety of your data.
If the server goes down or closes forever, exporting your data and moving to another solution is nearly impossible.
Services like Google Takeout offer to export your data in various formats (json, csv, pdf, txt, xml, images, audio, documents) but the format won't work with another app. You are left with a collection of files that you can't import anywhere else.
A sync engine offers a bridge between client and server, but it's not a solution to the problem of long-term storage.
Ideal 3: Privacy
Privacy is still an unsolved problem. Complete privacy is only achievable when all the data lives on the user device. Multi-device support and collaboration open a whole new set of challenges.
Ideally a sync engine should offer a solution to encrypt the data and keep it private. However, this is not a trivial task, and often not the primary concern of most apps.
On top of that, questions like ownership and authentication add even more complexity to the problem of privacy, and all sort of edge cases arise in a distributed model.
Ideal 4: User control
Users should be allowed complete control over their data. In practice this often involves re-implementing all the basic file system operations (create, read, write, delete, move, copy, etc.).
This is not really a concern of a sync engine. It's up to the developer to implement the right APIs and logic to handle this.
Ideal 5: Server independence
Martin Kleppmann in his talk at localfirstconf 2024 offered a new definition of local-first:
"If it doesn't work if the app developer goes out of business and shuts down their servers, then it's not local-first."
This adds another stringent constraint to what it means to be local-first. Ideally it should be possible for a user to move to another provider and continue working without any interruption.
This is an ambitious goal that today is hard if not impossible to achieve. The proposed model would be a unified protocol for syncing data between devices.
Just a simple sync service in between that works with all your apps, just syncing data between local devices. Image from "The past, present, and future of local-first - Martin Kleppmann (Local-First Conf)"
If the same protocol is implemented and shared between multiple providers, it becomes possible to make the app independent of any single provider.
Sync engine or full local-first?
Not all apps need to be fully local-first. Achieving all the ideals of local-first requires more effort that may not be justified for all apps.
Apps based on a cloud model can still benefit from a sync engine:
- Better user experience: change are applied locally and instantly, and only synced automatically in the background
- Better developer experience: managing state becomes as easy as mutating some data, the sync engine takes care of the rest
On top of these benefits, local-first provides some more guarantees, at the cost of added complexity:
- Offline support: sync engines are not designed to store data on a local database, so they can't guarantee offline support
- Longevity: sync engines rely on centralized servers, so the users are not in control of their data
- Privacy: with sync engines the main source of the data is still the server, local-first instead relies on the user's device