[bitnami/etcd] Stop relying on files for state

[pckhoi]

Description of the change

The current etcd container and chart have a few major problems:

1. It relies on files outside of the data directory which could contain conflicting information compared to the data dir and the cluster
2. Removing the member during pre-stop hook is problematic. I guess this was added to support scaling down the cluster. If so, this logic is leaky. There are 2 cases where this breaks down:
   1. If the pod was killed for reasons other than replicas update then the next time the pod starts, it will not be able to start from the existing data dir which means it must throw away the data dir and start from scratch.
   2. If the cluster is scaled down and the PVC is retained, the next time the cluster is scaled up, the new member will encounter a non-empty data dir which it must discard
3. If the member was removed, the container chokes up on non-empty data dir in most cases except when recovering from a snapshot
4. It might attempt to add a new member even when an old member with the same name already exists. This is caused by relying on files for state
5. It runs etcdctl member update for unclear reasons when the data dir is not empty and there is a member ID
6. It relies on ETCD_INITIAL_CLUSTER_STATE to know whether the cluster is new which could be inaccurate

This PR add the following changes:

• Add preupgrade.sh which should be run in a Helm pre-upgrade hook. When the cluster is scaled down, it detects and removes obsolete members with etcdctl member remove.
• Remove prestop.sh
• Stop storing/checking member ID from the member_id file. Instead, the remote member ID is read from the cluster with etcdctl member list, and the local member ID is checked for conflict during startup.
• Stop storing/checking member removal state from member_removal.log. Check with etcdctl member list instead.
• If the data dir is not empty, check if the member still belongs to the cluster (remote ID and local ID are the same). If there is a conflict, remove the data dir, remove the old member, add a new member, and start the member from scratch.
• Remove environment variable ETCD_DISABLE_STORE_MEMBER_ID
• Remove environment variable ETCD_DISABLE_PRESTOP
• Environment variable ETCD_INITIAL_CLUSTER_STATE becomes read-only

Benefits

• Not relying on files outside of the data directory means there is only a single source of truths (or only as many as there are live members in the cluster plus the data dir), which makes most operations more reliable
• Removing obsolete members in Helm pre-upgrade hook means the etcdctl member remove command tends to be executed against a healthy cluster
• If the pod was killed for reasons other than replica changes, it can rejoin the cluster on its own while keeping all its data intact
• The container no longer chokes up on a non-empty data dir, even when the old member is removed

Possible drawbacks

• If during initialization there is a network outage and the current member can't connect to other members, it will think that it must start a new cluster. That said, I don't think there is any good solution in this case except manual recovery.
• I have not tested this set of changes outside of Helm/K8s

Applicable issues

• fixes #16069

Additional information

Related changes in the Helm chart: bitnami/charts#31161 and bitnami/charts#31164

[pckhoi]

I'm planning to open a complementary PR in the charts repo. I will try to add more tests there.

[juan131]

Hi @pckhoi

Thanks so much for this amazing contribution! It'd definitely help on making the Bitnami etcd chart more stable.

I think the main concern/challenge with your changes would be providing a solution for users who may scale down the cluster via kubectl scale sts/etcd --replicas X (or via some HorizontalPodAutoscaler that may also scale down the cluster without Helm's control via hooks). Correct me if I'm wrong but this use case won't be covered, right?

[pckhoi]

@juan131 you're correct that the autoscaling use case isn't covered. People use Etcd for its consistency rather than for handling large, fluctuating traffic so I think autoscaling to handle large traffic is a niche use case.

As for manual scaling, running helm upgrade makes more sense if the cluster is installed via Helm. If people are scaling with kubectl scale then they're probably not using Helm which makes it difficult to operate. So no, deploying/upgrading without Helm is also not supported.

[juan131]

Thanks for confirming so @pckhoi ! In that case, I'd add a warning at the "Upgrading" section alerting about what these changes imply (I mean, warning users to use exclusively Helm to scale the cluster):

• https://github.com/bitnami/charts/tree/main/bitnami/etcd#upgrading

We could even add it in the chart NOTES:

• https://github.com/bitnami/charts/blob/main/bitnami/etcd/templates/NOTES.txt

[pckhoi]

Sure, I will do that.

[pckhoi]

@juan131 I have updated https://github.com/bitnami/charts/tree/main/bitnami/etcd#upgrading. As for https://github.com/bitnami/charts/blob/main/bitnami/etcd/templates/NOTES.txt, I don't see anything that needs to be updated.

[pckhoi]

Thanks! I have addressed all the suggestsions.

[juan131]

@pckhoi I think this PR looks great now! Could you please check my comments in the associated chart PR? Thanks in advance.

[juan131]

I've been doing more tests today and we'll have to change this block. This always throws etcd start logs to stdout regardless BITNAMI_DEBUG is set or not. Alternative:

Suggested change

	am_i_root && start_command=("run_as_user" "$ETCD_DAEMON_USER" "${start_command[@]}")
	[[ -f "$ETCD_CONF_FILE" ]] && start_command+=("--config-file" "$ETCD_CONF_FILE")
	$start_command > >(tee -a "$tmp_file") 2>&1 &
	pid=$!
	debug "Started etcd in background with PID $pid"
	while read -r line; do
	echo "$line" # Stream the output
	if [[ "$line" =~ (established TCP streaming connection with remote peer|the member has been permanently removed from the cluster|ignored streaming request; ID mismatch|\"error\":\"cluster ID mismatch\") ]]; then
	kill "$pid"
	wait "$pid" 2>/dev/null
	debug "Stopped etcd"
	break
	fi
	done < <(tail -f "$tmp_file")
	am_i_root && start_command=("run_as_user" "$ETCD_DAEMON_USER" "${start_command[@]}")
	[[ -f "$ETCD_CONF_FILE" ]] && start_command+=("--config-file" "$ETCD_CONF_FILE")
	"${start_command[@]}" > "$tmp_file" 2>&1 &
	while read -r line; do
	debug_execute echo "$line"
	if [[ "$line" =~ (established TCP streaming connection with remote peer|the member has been permanently removed from the cluster|ignored streaming request; ID mismatch|\"error\":\"cluster ID mismatch\") ]]; then
	etcd_stop
	debug "Stopped etcd"
	break
	fi
	done < <(tail -f "$tmp_file")

By the way, it's not necessary to save the PID, you can use etcd_stop

[pckhoi]

Updated.

[juan131]

small fix:

Suggested change

	read -r -a obsolete_members <<<"$(comm -23 <(echo "$current" | awk -F: '{print $1}' | sort) <(echo "$expected" | sort))"
	read -r -a obsolete_members <<<"$(comm -23 <(echo "$current" | awk -F: '{print $1}' | sort) <(echo "$expected" | sort) | tr -s '\n' ' ')"

[pckhoi]

This change seems to break the script for me.

[pckhoi]

Actually, something else is breaking. Let me investigate more.

[juan131]

What problems are you experiencing? It works for me

Without this change only one member is removed given the obsolete_members wasn't an array but a string with the 1st obsolete member to remove, so when I did a test scaling down from 5 to 3 replicas it only removed one.

[pckhoi]

Sorry, your suggestion works. I just had to fix the loop as well. Everything should work now.