Explore data availability options

[Mirko-von-Leipzig]

I'm actually unsure of what we need here :)

Will edit this once I'm more certain.

---

We need some form of data availability solution.

• what data needs to go on here?
• do we need full chain history; can we prune e.g. epoch data?

---

Initially we need some backup for critical failure recovery. Further on we want a more decentralised solution.

[bobbinth]

I think there are two things we need on this:

1. Make sure we have set up DB backups (probably geo-replicated) to recover from server failures.
2. Integrate with decentralized DA networks like Avail or Celestia.

The first one is mostly a DevOps issue and should be relatively transparent for the functioning of the system. The second one may require some protocol-level changes (e.g., a block may need to include signatures from the validators of the DA layer).

[Mirko-von-Leipzig]

Only available option I'm aware of for SQLite is litestream which (ab)uses the SQLite WAL file. It's a small separate process which uploads the data to any cloud bucket -- which is (a) cheap, and (b) simple. Biggest pain point would be ito network deployment e.g. reset network requires deleting the cloud bucket etc.

But its pretty perfect otherwise. Literally just an additional process running on the instance, and a separate cloud bucket setup somewhere with the correct permissions. Completely transparent to the store outside of having the correct WAL configuration.

[bobbinth]

Biggest pain point would be ito network deployment e.g. reset network requires deleting the cloud bucket etc.

Could you give more detail what does this mean (or under what conditions could this happen)?

[Mirko-von-Leipzig]

Could you give more detail what does this mean (or under what conditions could this happen)?

It just means an additional thing to consider during a testnet/devnet deployment, while we are still resetting the network each time.

Essentially we just need to understand what litestream does if the database is deleted and recreated. I imagine it will be confused and break somehow. So probably we need to

1. Switch litestream off on the instance
2. Delete the node data on the instance
3. Delete the cloud backup - or ensure that the backups have a naming schema. Maybe there is value in keeping old ones for a short while at least?
4. Re-enable litestream
5. Start the node again

[bobbinth]

Makes sense!

Btw, one other potential way to do backups is to have some kind of replicated storage for the raw block data. Right now, this data is being saved to disk as flat files (1 file per block). But there is no reason why we couldn't save these to an external, geo-replicated container. Though, if we do just this, restarting a node would require re-executing the blocks from genesis - so, probably doing something like litestream is still probably a good idea.

Both of these would cover the first option (DB backup). We still need to think what we'd want to do for the decentralized DA option.

[Mirko-von-Leipzig]

Do we have a timeline in mind for when we need this by?

Database backup could be done anytime; just some devops work.

[bobbinth]

I'd say we should try to do DB backups with the next release (v0.8). So, let's create a more concrete issue for it.

The decentralized DA is probably something we should start working on in April/May.

[Mirko-von-Leipzig]

#633