Recover one or few node's db from corruption

[hasethuraman]

Hi,

I am thinking these scenarios.

(1)
Have 3 etcd instances and say one of the etcd (follower) crashed and started reporting database corruption. Etcd read and write will continue to work as long as other 2 etcd instances are undisturbed.
Can I take a snapshot from the leader and recover this instance?
If yes, I am wondering if there is a data loss. Because a snapshot is nothing but a copy of the database which wont have WAL entries. If my thought is correct, how does the follower instance will learn the entries from WAL in other nodes?

(2)
Same question as above when two followers are crashed - etcd cluster will become inoperable. I hope still I can take a snapshot and recover. Same concern as in first how WAL entries from other nodes are learned.

(3)
When one follower and leader crash due to corrupted database. I hope with kv loss, I can take a snapshot from the follower (which doesnt have db corruption) and recover with kv loss.

Please guide.

Thanks
Hari

[serathius]

https://etcd.io/docs/v3.5/op-guide/data_corruption/

[hasethuraman]

Thank you @serathius .

We need snapshot to restore the entire etcd cluster. But snapshot (db) may not have committed wal entries and yet to be written to db file. Am I missing to see the answer for my question from the docs?

[hasethuraman]

@serathius Please let me know if snapshot may not have the key-values those are committed in WAL but not in snapshot yet. If yes, is there a workaround?