Fix corrupted sst file

[ltagliamonte-dd]

Hello Friends!
I'm recording the following error for multiple keys:

{"time":"2024-09-28T18:01:44.329724216Z","level":"WARN","msg":"retrying redis OP","attempt":1,"error":"response returned from Conn: ERR Corruption: block checksum mismatch: stored = 875772211, computed = 2997829094, type = 4  in /data/kvrocks/db/003571.sst offset 763295792 size 16306"}

I've tried to run a compaction but it hasn't helped:

E20240930 21:30:38.654457    40 event_listener.cc:162] [event_listener/background_error] reason: compaction, bg_error: Corruption: block checksum mismatch: stored = 822083584, computed = 1414082609, type = 4  in /data/kvrocks/db/003571.sst offset 762364784 size 16313
E20240930 21:30:38.654842    67 server.cc:1398] [task runner] Failed to do compaction: Corruption: block checksum mismatch: stored = 822083584, computed = 1414082609, type = 4  in /data/kvrocks/db/003571.sst offset 762364784 size 16313

I see that rocksdb offers ldb --repair but I can't find a way to build it in my docker image using x.py

[PragmaTwice]

ldb is a tool inside rocksdb, not kvrocks.

So maybe you can clone rocksdb and follow its build instructions.

[ltagliamonte-dd]

Thanks for the replay, using ldb I wasn't able to recover my db.
ldb is having issues facebook/rocksdb#13053.

I'm having this problem happening over my cluster "frequently" (3 times in 2 weeks) and I'm looking for a long term solution I can't just drop a shard and rebuilt it every time.
Can anyone help me understanding what's is going wrong?

I'm using latest kvrock: v2.9.0
and this is my running config:

command=/usr/local/bin/kvrocks/kvrocks
    --dir %(ENV_KVROCKS_DATA_DIR)s
    --port 6379
    --rocksdb.compaction_readahead_size 10485760
    --rocksdb.target_file_size_base 1024
    --rocksdb.wal_ttl_seconds 0
    --rocksdb.wal_size_limit_mb 0
    --maxclients 20000
    --log-dir stdout
    --log-level error

i don't have replication that's why wal_ttl_seconds and wal_size_limit_mb are set to 0.
Cc @git-hulk

[git-hulk]

@ltagliamonte-dd That's weird. As far as I know, this issue is generally related to the hardware(disk). Did it happen on the same node(disk device)?

[ltagliamonte]

Different nodes. Everything is running in docker containers.
The env is pretty dynamic, when it kvrocks bootstrap..it downloads last successful backup from S3.
If there are any issue during the processing the backup is not created.

[git-hulk]

Different nodes. Everything is running in docker containers. The env is pretty dynamic, when it kvrocks bootstrap..it downloads last successful backup from S3.

Do you mean that got the SST mismatch CRC error after restoring the backup from the S3?

[ltagliamonte]

No I checked the logs the mismatch happens randomly in the middle of data processing (best) or after a backup. Haven't find yet a common denominator.