β€’

newsletter

My struggle with data storage for web apps

I want data storage to be simple and fast. Local storage. But I also want collaboration and syncing. Remote storage. Here is a list of possible solutions.


Sandro Maglione

Sandro Maglione

Software development

The worst enemy of any app is storing data. Well, it's also its best friend. And that's precisely what makes storage such a big hassle 🀯

Most problems derive from organizing data storage, being it database, local storage, state management. Conversely, storage is what makes an app usable, remembering things πŸ™Œ

So, what would be the ideal storage solution for a web app?

That's how my struggle evolves πŸ‘‡


Remote database: complexity, complexity everywhere

Storing data in a remote database is a simple model. Single source of truth, all in one place, designed to be fast and easy to query.

It's everything that's around it that becomes bloated. Suddenly, you need a backend, and API, asynchronous request, possible failures, serialization.

The link between your device and the remote database is indirect and complex. Image from "Solving the distributed schema problem with @effect/schema by Jess Martin (Effect Days 2024)"The link between your device and the remote database is indirect and complex. Image from "Solving the distributed schema problem with @effect/schema by Jess Martin (Effect Days 2024)"

With 2024's technologies, this seems absurd to me. Having tasted the promise of local-first makes remote databases even more sour.

I really don't look forward for yet another complex app with remote storage πŸ˜’

Local storage is (mostly) just a cache

Don't like remote? Obvious solution: local!

Well, no.

Most local storage solutions are designed to be, well, local. Lightweight, fast, distributed. A glorious cache. That's indeed not such a robust solution for a production web app πŸ˜•

Local storage is also local, aka not shared. Want shared data? A whole new set of problems come with it 😈

Syncing is (mostly) not here yet

The advantages of local data are clear. However, we want the advantages of remote storage as well. Everything!

That's where we enter the realm of distributed systems. As hard of a problem as it gets it turns out πŸ₯²

A solution may be a sync engine. Indeed, the idea sounds awesome: a single lightweight backend only working as a "pipe" linking multiple local devices.

Just a simple sync service in between that works with all your apps, just syncing data between local devices. Image from "The past, present, and future of local-first - Martin Kleppmann (Local-First Conf)"Just a simple sync service in between that works with all your apps, just syncing data between local devices. Image from "The past, present, and future of local-first - Martin Kleppmann (Local-First Conf)"

Problem is we are not there yet. The web evolved as a mostly-stateless model: make a request, get response, bye-bye. Many teams working on a possible solution, but all looks still like a big work in progress.

I wrote at length about this and the whole situation with local first in a previous newsletter: The State of Local-First.

What about data import/export?

If a sync engine is not available, what about an async engine?

Instead of syncing all the data in "real time", you allow the user to export its local data, and import it back in another device.

You keep all the advantages of local-only, while preserving at least in part a resemblance to "shared data".

I have yet to explore this in practice. Writing this make it sound like not too-bad a solution, I'll see πŸ€”

Hybrid solution (dump the backend)

Import/export can be made easier using an in-between cloud storage.

Instead of exporting on-device and requiring the user to copy/paste/import manually, a cloud can sit in between to do this semi-automatically.

You click a button, a file is exported on your cloud provider, you open another device, click import, and select the file from the cloud.

Doesn't sound too bad, does it? I'll need to explore this as well πŸ€”

Performance and privacy

It's insane to me that to send a message to a friend in my neighborhood, my message must first hit a server on the other side of the world.

It's an issue of performance, but also privacy and ownership. I want my apps to store my ideas and information, why is this required to be tied to a server?

It's not, or at least that's the direction I am heading to.


New release coming: Effect with React 19: Project Template πŸš€

I packed all my most common practices while working with Effect in React πŸ‘‡

Tweet not found

The embedded tweet could not be found…

I am also planning an update to Typeonce's landing page (updated offer). This may interest you, stay tuned πŸ‘€

See you next πŸ‘‹

Start here.

Timeless coding principles, practices, and tools that make a difference, regardless of your language or framework, delivered in your inbox every week.