fsync discrepancy between v2 and v3 backends

[kjin97]

Hi! We are moving from an etcd v2 to an etcd v3 backend and noticed an interesting performance discrepancy between the two. Using strace and the etcd metrics, we like to track the fsync frequency of etcd nodes to ensure we don't wear out the SSD.

In the v2 backend, a get with --quorum resulted in an fsync and a raft commit.

In the v3 backend, a get with --consistency=l (linearizable get) does not cause an fsync or a raft commit.

While this is great news for us from the performance standpoint, I am concerned that this results in a less strict consistency than in the v2 backend. Was this an optimization the team made in the transition to the MVCC, or have some consistency guarantees been lost between the two backends? If anyone who is very familiar with how etcd interacts with raft could help me here, that would be greatly appreciated!

To reproduce:

start an etcd cluster

curl -s -XGET $ETCD/metrics | grep 'etcd_server_proposals_applied'
curl -s -XGET $ETCD/metrics | grep 'fsync_duration_seconds_count'

optionally: start a background strace

ETCDCTL_API=3 etcdctl put foo bar

observe that v3 puts increment the raft commit index

ETCDCTL_API=2 etcdctl set foo bar

observe that v2 sets increment the raft commit index

ETCDCTL_API=3 etcdctl get foo

observe that v3 gets do not increment the raft commit index

ETCDCTL_API=2 etcdctl get foo --quorum

observe that v2 quorum gets do increment the raft commit index.

[tjungblu]

This might've been answered already for you somehow after a month of posting this, but the magic is actually in here:
https://github.com/etcd-io/etcd/blob/main/server/etcdserver/v3_server.go#L123-L129

Where you're notified when a new index is processed, this plays together with the read loop to find the current index:

etcd/server/etcdserver/v3_server.go

Lines 725 to 776 in 5073af6

	func (s *EtcdServer) linearizableReadLoop() {
	for {
	requestId := s.reqIDGen.Next()
	leaderChangedNotifier := s.leaderChanged.Receive()
	select {
	case <-leaderChangedNotifier:
	continue
	case <-s.readwaitc:
	case <-s.stopping:
	return
	}
	// as a single loop is can unlock multiple reads, it is not very useful
	// to propagate the trace from Txn or Range.
	trace := traceutil.New("linearizableReadLoop", s.Logger())
	nextnr := newNotifier()
	s.readMu.Lock()
	nr := s.readNotifier
	s.readNotifier = nextnr
	s.readMu.Unlock()
	confirmedIndex, err := s.requestCurrentIndex(leaderChangedNotifier, requestId)
	if isStopped(err) {
	return
	}
	if err != nil {
	nr.notify(err)
	continue
	}
	trace.Step("read index received")
	trace.AddField(traceutil.Field{Key: "readStateIndex", Value: confirmedIndex})
	appliedIndex := s.getAppliedIndex()
	trace.AddField(traceutil.Field{Key: "appliedIndex", Value: strconv.FormatUint(appliedIndex, 10)})
	if appliedIndex < confirmedIndex {
	select {
	case <-s.applyWait.Wait(confirmedIndex):
	case <-s.stopping:
	return
	}
	}
	// unblock all l-reads requested at indices before confirmedIndex
	nr.notify(nil)
	trace.Step("applied index is now lower than readState.Index")
	trace.LogAllStepsIfLong(traceThreshold)
	}
	}

Which in turn queries the raft-node via, which explains this very well:

etcd/raft/node.go

Lines 182 to 188 in 31d9664

	// ReadIndex request a read state. The read state will be set in the ready.
	// Read state has a read index. Once the application advances further than the read
	// index, any linearizable read requests issued before the read request can be
	// processed safely. The read state will have the same rctx attached.
	// Note that request can be lost without notice, therefore it is user's job
	// to ensure read index retries.
	ReadIndex(ctx context.Context, rctx []byte) error

	// ReadIndex request a read state. The read state will be set in the ready.
	// Read state has a read index. Once the application advances further than the read
	// index, any linearizable read requests issued before the read request can be
	// processed safely. The read state will have the same rctx attached.
	// Note that request can be lost without notice, therefore it is user's job
	// to ensure read index retries.

I hope this answers your questions somewhat, the reads of the raft nodes don't get persisted on the WAL which is why you don't see any fsync.