PD may be panic when enabling tidb_enable_tso_follower_proxy and restart PD

[okJiang]

Bug Report

What did you do?

1. enable tidb_enable_tso_follower_proxy
2. many tidb instances(40+) and the write traffic is very high.
3. restart PD, if it is the PD leader, the trigger probability will also increase.

What did you expect to see?

PD restart successfully

What did you see instead?

PD panic

pd_log.log

What version of PD are you using (pd-server -V)?

[okJiang]

Analyze:

1. PD server down
2. Client delete the stale stream
3. PD server up
4. Client just called

   pd/client/clients/tso/client.go

   Line 246 in c2f72ac

   func (c *Cli) updateConnectionCtxs(ctx context.Context, connectionCtxs *sync.Map) bool {

5. Client discovery the new etcd server up

   pd/client/clients/tso/client.go

   Line 471 in c2f72ac

   func (c *Cli) getAllTSOStreamBuilders() map[string]tsoStreamBuilder {

6. Client try to create a new stream to PD server

   pd/client/clients/tso/client.go

   Lines 443 to 452 in c2f72ac

   	log.Info("[tso] try to create tso stream", zap.String("addr", addr))
   	cctx, cancel := context.WithCancel(ctx)
   	// Do not proxy the leader client.
   	if addr != leaderAddr {
   	log.Info("[tso] use follower to forward tso stream to do the proxy",
   	zap.String("addr", addr))
   	cctx = grpcutil.BuildForwardContext(cctx, forwardedHost)
   	}
   	// Create the TSO stream.
   	stream, err := tsoStreamBuilder.build(cctx, cancel, c.option.Timeout)

7. At this time, PD server's initialization does not completed. tsoDispatcher is nil.

   pd/server/grpc_service.go

   Line 553 in c2f72ac

   s.tsoDispatcher.DispatchRequest(ctx, tsoRequest, s.pdProtoFactory, doneCh, errCh, s.tsoPrimaryWatcher)

8. So, PD panic