job failed after shuffle pod restart

[ChenLingPeng]

How to reproduce

1. Submit a job like PageRank use external shuffle service
2. After executors running, stop some external-shuffle-service pod in executor's host
3. The external-shuffle-service pod will restart with some new pod IP
4. Driver exit with failed status

See the log in driver/executor, it shows pod always try to fetch block using old shuffle-pod-ip

[liyinan926]

Yes, we use the shuffle pod IP to identify the shuffle pod and set spark.shuffle.service.host to the IP. So it seems shuffle pods need sticky network identify.

[weixiuli]

May avoid this issus by using hostnetwork ?