Idea: Enhance NiFiKOp to drive standalone NiFi clusters

[mh013370]

Overview

Proposal #98 suggested creating a new CRD to enable NiFiKOp to deploy and configure a set of unclustered, or standalone, NiFi nodes. In that discussion and through experimenting with actually making the change, I found that this use case didn't warrant creating a new CRD to accomplish the ultimate goal:

• It would be great if NiFiKOp could deploy and configure a set of standalone NiFi nodes for cases where you don't need clustering and could benefit from many independent standalone NiFi nodes.

Why

The first question is likely why you might want to do this.

There are benefits and drawbacks of clustering in NiFi.

Benefits of clustering

• You may interact with every node in the NiFi cluster via the Cluster Coordinator (e.g. the zookeeper-elected leader). All REST API requests are internally replicated to every node by the coordinator.
• You gain coordination features like:
  • flow election, guaranteeing that every node is running the exact same flow
  • primary node only processors, guaranteeing that only one node in the cluster will run a particular processor. Useful for data pulls or scenarios where you only want 1 node to do some work.
  • As of NiFi 1.16.0 , flows can now be edited while one cluster node is disconnected (previously not possible). Disconnected nodes can now rejoin the cluster, updating their flow to match whatever the cluster gives it. Even if that flow is drastically different.

Drawbacks of clustering

• Upgrades are challenging, requiring a careful process to disconnect nodes from the cluster and then draining, stopping, upgrading, and then restarting them.
• It's not possible to upgrade a cluster with zero downtime. Nor is it possible to run a cluster with a mixed set of NiFi versions.
• In large cluster scenarios, nodes tend to disconnect from the cluster in production and it can be painful getting the node re-added to the cluster.

Benefits of unclustered NiFis

• You can trivially rolling upgrade each NiFi node
• You can support a mixed set of NiFi versions in the "cluster"
• There's no complicated leader election or cluster connectivity problems
• Easily horizontally and vertically scalable

Drawbacks of unclustered NiFis

• There's no flow election, so there will be periods of time during flow upgrades where nodes in the "cluster" will be running different flows.
• You lose the ability to drain/offload flowfiles from one NiFi to another. This could be problematic if the NiFi can't successfully process a particular flowfile and gets hung up.
• In order to command & control a "cluster" RESTfully, you need to issue REST requests to each node instead of just the cluster coordinator.

Standalone Cluster Design

We can extend the existing NifiCluster CRD to enable nifikop to deploy and configure standalone NiFi nodes. First, let's review how the NiFi cluster coordinator works.

NiFi Cluster Coordinator

In a clustered NiFi scenario, NiFi uses Apache Zookeeper to elect a single cluster coordinator. This enables a client to interact with the entire cluster through a single node. When the cluster coordinator receives a REST request, it internally replicates that request to each other node in the NiFi cluster.

In order to guarantee that newly created components (e.g. process groups, users, user groups, parameter contexts, etc.) get the same UUID, the cluster coordinator sets the following header on each replicated request: X-Cluster-Id-Generation-Seed.

And then when a new Id is generated for a component, it uses the value of that header as a seed.

So for standalone clusters, we can have NiFiKOp act like the elected cluster coordinator to replicate requests to each of the standalone NiFi nodes, ensuring that each component created gets the same ID.

NiFiKOp Changes

1. We need a new cluster type. There are already internal and external clusters. There will be a new internal-standalone cluster type.

2. For internal-standalone cluster types, NiFiKOp needs to act like the elected cluster coordinator. This means that it needs to interact with every node for every REST call that typically goes to the cluster coordinator. And when components are created, the operator must set the X-Cluster-Id-Generation-Seed header so that the component has the same UUID on each node.

3. For each CRD that can be attached to a NifiCluster, we will track the seed used to create the component. This is so that if nodes are added in the future or there's a failure during setup, we can re-generate the UUID for the component. For example, for NifiRegistryClient, we add NifiRegistryClient.Status.IdSeed which will use the value of the X-Cluster-Id-Generation-Seed header used to create the component.

[mh013370]

I've got a functional prototype of this implementation here: https://github.com/michael81877/nifikop/tree/add-unclustered-type

[mh013370]

I've proved that with these changes, nifikop can deploy a 2+ node standalone cluster and configure at least NifiRegistryClients, NifiParameterContexts, and PrometheusReportingTasks on each of the nodes. I will be testing the changes for NifiUsers, NifiDataflows, and NifiUserGroups soon.