Member promote request sent to follower get unexpected response when auth is enabled

[monsterxx03]

What happened?

Assume we have a etcd cluster of node 217 and 218(two or three nodes setup make no difference in this case).

And auth is enabled in this cluster(I'll omit --user option in example)

etcdctl member list

+------------------+---------+-------------+----------------------------+----------------------------+------------+
|        ID        | STATUS  |    NAME     |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------------+----------------------------+----------------------------+------------+
| 66a6baefa8d6cecb | started | etcd-node-1 | http://10.252.90.217:22380 | http://10.252.90.217:22379 |      false |
| c6b6f4cbd45937f5 | started | etcd-node-2 | http://10.252.90.218:22380 | http://10.252.90.218:22379 |      false |
+------------------+---------+-------------+----------------------------+----------------------------+------------+

etcdctl endpoint status --cluster

+----------------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | DB SIZE IN USE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+
| http://10.252.90.217:22379 | 66a6baefa8d6cecb | 3.6.0-alpha.0 |           3.6.0 |   20 kB |          16 kB |     false |      false |         6 |         93 |                 93 |        |
| http://10.252.90.218:22379 | c6b6f4cbd45937f5 | 3.6.0-alpha.0 |           3.6.0 |   20 kB |          16 kB |      true |      false |         6 |         94 |                 94 |        |
+----------------------------+------------------+---------------+-----------------+---------+----------------+-----------+------------+-----------+------------+--------------------+--------+

218 is the leader node.

etcdctl --endpoint=http://10.252.90.217:22379 member promote c6b6f4cbd45937f5

{"level":"warn","ts":"2022-05-25T14:56:30.869+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:64","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0003fc000/10.252.90.217:22379","method":"/etcdserverpb.Cluster/MemberPromote","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: request timed out"}
Error: etcdserver: request timed out

Check log in 218(the leader), we get bunch of logs like following:

{"level":"warn","ts":"2022-05-25T14:56:30.816+0800","caller":"etcdhttp/utils.go:72","msg":"unexpected v2 response error","remote-addr":"10.252.90.217:39136","internal-server-error":"auth: user name is empty"}                                                                                                              
{"level":"warn","ts":"2022-05-25T14:56:30.816+0800","caller":"etcdhttp/peer.go:151","msg":"failed to promote a member","member-id":"c6b6f4cbd45937f5","error":"auth: user name is empty"}                                                                                                                                     
{"level":"warn","ts":"2022-05-25T14:56:30.850+0800","caller":"etcdhttp/utils.go:72","msg":"unexpected v2 response error","remote-addr":"10.252.90.217:39136","internal-server-error":"auth: user name is empty"}                                                                                                              
{"level":"warn","ts":"2022-05-25T14:56:30.851+0800","caller":"etcdhttp/peer.go:151","msg":"failed to promote a member","member-id":"c6b6f4cbd45937f5","error":"auth: user name is empty"}                      

Check log in 217(the follower receive promote request):

{"level":"debug","ts":"2022-05-25T15:11:55.228+0800","caller":"v3rpc/interceptor.go:185","msg":"request stats","start time":"2022-05-25T15:11:50.336+0800","time spent":"4.89210609s","remote":"10.252.90.217:37732","response type":"/etcdserverpb.Cluster/MemberPromote","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}

What did you expect to happen?

217(the follower) should return

Error: etcdserver: can only promote a learner member

How can we reproduce it (as minimally and precisely as possible)?

Use my above steps

Anything else we need to know?

I checked code,

etcd/server/etcdserver/cluster_util.go

Line 292 in c3bc411

req, err := http.NewRequest("POST", requestUrl, nil)

seems when follower is forwarding promote member request to leader, didn't set token info in header.

When auth is disabled, follower will return normal error Error: etcdserver: can only promote a learner member

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.6.0-alpha.0
Git SHA: c3bc4116d
Go Version: go1.18
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.6.0-alpha.0
API version: 3.6

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.