Skip to content
This repository has been archived by the owner on Aug 22, 2024. It is now read-only.

Latest commit

 

History

History
640 lines (487 loc) · 20.7 KB

README.markdown

File metadata and controls

640 lines (487 loc) · 20.7 KB
Raw

zenoss-kubernetes-agent

An agent that collects metrics from a Kubernetes cluster and sends them to Zenoss.

Note: this agent is no longer being actively developed. The zenoss-agent-kubernetes package is now based upon zdatamon's kubernetesagent datasource, and future work is being done there.

Contents:

Dashboards

The following example dashboards were created using the data sent to Zenoss by this agent. You can use these examples as-is, or adapt them to your own needs.

Kubernetes: Multi-Cluster View

This dashboard's scope can be updated to include multiple clusters by adding the agent for each cluster to the dashboard scope's Sources list.

Kubernetes: Multi-Cluster View

The tiles for this dashboard are configured as follows.

Nodes by Cluster

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.cluster.nodes.total
  • Aggregator: none

Pods by Cluster

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.cluster.pods.total
  • Aggregator: none

CPU Usage by Cluster

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.cluster.cpu.ms
  • Aggregator: none

Memory Usage by Cluster

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.cluster.memory.bytes
  • Aggregator: none

Agent CPU Usage

  • Type: Single Metric
  • Chart type: line
  • Metric: k8s.pod.cpu.ms (entity search: zenoss-agent-kubernetes)
  • Chart label: CPU Milliseconds

Agent Memory Usage

  • Type: Single Metric
  • Chart type: line
  • Metric: k8s.pod.memory.bytes (entity search: zenoss-agent-kubernetes)
  • Chart label: Memory Bytes

Pods by Namespace

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.namespace.pods.total
  • Aggregator: none

CPU Usage by Namespace

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.namespace.cpu.ms
  • Aggregator: none

Memory by Namespace

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.namespace.memory.bytes
  • Aggregator: none

Kubernetes: Single Cluster View

This dashboard's scope is intended to be set to one specific Kubernetes cluster by adding only the agent for that cluster to the dashboard scope's Sources list.

Kubernetes: Single Cluster View

The tiles for this dashboard are configured as follows.

Total Nodes

  • Type: MultiMetric
  • Chart type: bar
  • Legend: none
  • Metric name: k8s.cluster.nodes.total
  • Aggregator: none

Total Pods

  • Type: MultiMetric
  • Chart type: bar
  • Legend: none
  • Metric name: k8s.cluster.pods.total
  • Aggregator: none

Total Containers

  • Type: MultiMetric
  • Chart type: bar
  • Legend: none
  • Metric name: k8s.cluster.containers.total
  • Aggregator: none

Pods by Namespace

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.namespace.pods.total
  • Aggregator: none

Containers by Namespace

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.namespace.containers.total
  • Aggregator: none

CPU Usage by Namespace

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.namespace.cpu.ms
  • Aggregator: none

Memory Usage by Namespace

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.namespace.memory.bytes
  • Aggregator: none

CPU Usage by Node

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.node.cpu.ms
  • Aggregator: none

Memory Usage by Node

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.node.memory.bytes
  • Aggregator: none

CPU Usage by Pod

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.pod.cpu.ms
  • Aggregator: none

Memory Usage by Pod

  • Type: MultiMetric
  • Chart type: line
  • Legend: right
  • Metric name: k8s.pod.memory.bytes
  • Aggregator: none

Deploying

The agent is intended to be deployed within the Kubernetes cluster it will be monitoring. The following sections document how to deploy the agent to a cluster either via the native kubectl tool, or with Helm.

There are many ways to deploy resources into a Kubernetes cluster. The following steps should be adaptable to your chosen method of deploying resources.

Prerequisites

The Kubernetes Cluster must have the following prerequisites for the agent to function.

  1. Kubernetes 1.8+
  2. Kubernetes Metrics Server

Deploying with kubectl

  1. Ensure your kubectl is configured to use the correct context.

    kubectl config current-context

  2. Create a Secret containing your Zenoss API key.

    One of the parameters we're going to need to configure for the agent is the Zenoss API key it will use to publish data to Zenoss. We could configure this directly as an environment variable for the agent in its Deployment, but this is insecure. The preferred option for this kind of thing is to create a Kubernetes Secret to use in the agent's Deployment.

    Be sure to replace <API_KEY> with your Zenoss API key.

    kubectl -n kube-system create secret generic zenoss --from-literal=api-key=<API_KEY>

  3. Create a zenoss-agent-kubernetes.yml file with the following contents.

    Be sure to replace <CLUSTER_NAME> with a unique name for the cluster into which you're deploying the agent. This can be just about anything you like, but something like a fully-qualified DNS name (doesn't need to resolve) can help to make it unique. For example, when naming my Google Kubernetes Engine clusters I tend to use <cluster-name>.<project-name>.

    apiVersion: v1
kind: ServiceAccount
metadata:
  name: zenoss-agent-kubernetes
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:zenoss-agent-kubernetes 
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["nodes", "pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["nodes", "namespaces", "pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions", "apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: zenoss-agent-kubernetes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:zenoss-agent-kubernetes
subjects:
- kind: ServiceAccount
  name: zenoss-agent-kubernetes
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zenoss-agent-kubernetes
  namespace: kube-system
  labels:
    app: zenoss-agent-kubernetes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zenoss-agent-kubernetes
  template:
    metadata:
      labels:
        app: zenoss-agent-kubernetes
    spec:
      serviceAccountName: zenoss-agent-kubernetes
      containers:
      - name: zenoss-agent-kubernetes
        image: docker.io/zenoss/zenoss-agent-kubernetes:latest
        env:
        - name: CLUSTER_NAME
          value: <CLUSTER_NAME>
        - name: ZENOSS_API_KEY
          valueFrom:
            secretKeyRef:
              name: zenoss
              key: api-key

  4. Apply the template.

    kubectl apply -f zenoss-agent-kubernetes.yml

Updating the Agent

To reconfigure the resources you would edit zenoss-agent-kubernetes.yml then run the following command. Kubernetes will identify what was changed from the last time you applied the template, and affect just those changes.

kubectl apply -f zenoss-agent-kubernetes.yml

Removing the Agent

To remove all of the resources you run the following command.

kubectl delete -f zenoss-agent-kubernetes.yml

Deploying with Helm

You must first have Helm running on your cluster. See the Helm Quickstart Guide for more information on getting started with Helm.

The following example installs the agent with the minimum required configuration. Replace <K8S_CLUSTER_NAME> with a unique name for the cluster. This is the name the cluster will be identified as in Zenoss. Replace <ZENOSS_API_KEY> with a Zenoss API key.

$ helm repo add zenoss https://zenoss.github.io/charts/
$ helm install zenoss/zenoss-agent-kubernetes \
    --name my-release \
    --set zenoss.clusterName=<K8S_CLUSTER_NAME> \
    --set zenoss.apiKey=<ZENOSS_API_KEY>

This command deploys zenoss-agent-kubernetes on the Kubernetes cluster in the default configuration. See the zenoss-agent-kubernetes chart documentation for additional configuration options that are available.

Configuration

The agent is configured with the following environment variables.

Environment Default Required Description
CLUSTER_NAME yes Kubernetes cluster name
ZENOSS_API_KEY yes Zenoss API key
ZENOSS_ADDRESS api.zenoss.io:443 no Zenoss API address
ZENOSS_NAME default no Name for API endpoint
ZENOSS_DISABLE_TLS false no Disable TLS
ZENOSS_INSECURE_TLS false no Disable certificate verification

It is also possible to configure the agent to send the same data to multiple Zenoss endpoints. This isn't commonly done, but could potentially be useful if you'd like to send the same data to separate tenants.

To send data to multiple Zenoss endpoints you would set the following environment variables instead of ZENOSS_ADDRESS and ZENOSS_API_KEY.

  • ZENOSS1_API_KEY
  • ZENOSS1_ADDRESS
  • ZENOSS1_NAME
  • ZENOSS2_API_KEY
  • ZENOSS2_ADDRESS
  • ZENOSS2_NAME
  • etc.

You can configure up to 9 (ZENOSS9_) endpoints this way. The ZENOSS_NAME environment variable gives a name to each endpoint, and is only used for logging purposes. Setting ZENOSS*_ADDRESS is optional. It will default to api.zenoss.io:443 just like ZENOSS_ADDRESS.

Data

Once deployed into a Kubernetes cluster, the agent will send data about the following entities types to Zenoss.

All models and metrics sent by the agent will have the following metadata.

Field Value
source-type zenoss.agent.kubernetes
source <clusterName>*

* Anytime <clusterName> is referenced throughout this data it refers to the value configured via the agent's CLUSTER_NAME environment variable.

Cluster

The agent will send a cluster model to Zenoss each time it starts.

Cluster Dimensions

Dimension Value
k8s.cluster <clusterName>

Cluster Metadata

Field Value
name <clusterName>
type k8s.cluster

Cluster Metrics

Metric Name Type Units
k8s.cluster.nodes.total GAUGE nodes
k8s.cluster.pods.total GAUGE pods
k8s.cluster.containers.total GAUGE containers
k8s.cluster.cpu.ms* GAUGE milliseconds
k8s.cluster.memory.bytes* GAUGE bytes

* Cluster CPU and memory metrics are a sum of the same metrics for all containers in the cluster.

Node

The agent will send a node model to Zenoss each time it receives node information from the Kubernetes API. Specifically this is the /api/v1/watch/nodes API endpoint. The agent will receive information about all nodes when it starts, and again for each node anytime the node's properties change.

Node Dimensions

Dimension Value
k8s.cluster <clusterName>
k8s.node <nodeName>

Node Metadata

Field Value
name <nodeName>
type k8s.node
impactToDimensions k8s.cluster=<clusterName>

Node Metrics

Metric Name Type Units
k8s.node.cpu.ms GAUGE milliseconds
k8s.node.memory.bytes GAUGE bytes

Node CPU and memory metrics are those directly reported for the node.

Namespace

The agent will send a namespace model to Zenoss each time it receives namespace information from the Kubernetes API. Specifically this is the /api/v1/watch/namespaces API endpoint. The agent will receive information about all namespaces when it starts, and again for each namespace anytime the namespace's properties change.

Namespace Dimensions

Dimension Value
k8s.cluster <clusterName>
k8s.namespace <namespaceName>

Namespace Metadata

Field Value
name <namespaceName>
type k8s.namespace
impactFromDimensions k8s.cluster=<clusterName>

Namespace Metrics

Metric Name Type Units
k8s.namespace.pods.total GAUGE pods
k8s.namespace.containers.total GAUGE containers
k8s.namespace.cpu.ms* GAUGE milliseconds
k8s.namespace.memory.bytes* GAUGE bytes

* Namespace CPU and memory metrics are a sum of the same metrics for all containers in the namespace.

Pod

The agent will send a pod model to Zenoss each time it receives pod information from the Kubernetes API. Specifically this is the /api/v1/watch/pods API endpoint. The agent will receive information about all pods when it starts, and again for each pod anytime the pod's properties change.

Pod Dimensions

Dimension Value
k8s.cluster <clusterName>
k8s.namespace <namespaceName>
k8s.pod <podName>

Pod Metadata

Field Value
name <podName>
type k8s.pod
impactFromDimensions k8s.cluster=<clusterName>,k8s.namespace=<namespaceName>, k8s.cluster=<clusterName>,k8s.node=<nodeName>

Pod Metrics

Metric Name Type Units
k8s.pod.containers.total GAUGE containers
k8s.pod.cpu.ms* GAUGE milliseconds
k8s.pod.memory.bytes* GAUGE bytes

* Pod CPU and memory metrics are a sum of the same metrics for all containers in the pod.

Container

The agent will send container models to Zenoss each time it receives pod information from the Kubernetes API. Specifically this is the /api/v1/watch/pods API endpoint. The agent will receive information about all pods (and their containers) when it starts, and again for each container anytime the container's pod's properties change.

Container Dimensions

Dimension Value
k8s.cluster <clusterName>
k8s.namespace <namespaceName>
k8s.pod <podName>
k8s.container <containerName>

Container Metadata

Field Value
name <containerName>
type k8s.container
impactToDimensions k8s.cluster=<clusterName>,k8s.namespace=<namespaceName>,k8s.pod=<podName>

Container Metrics

Metric Name Type Units
k8s.container.cpu.ms GAUGE milliseconds
k8s.container.memory.bytes GAUGE bytes

Container CPU and memory metrics are those directly reported for the container.

Deployment

The agent will send a deployment model to Zenoss each time it receives deployment information from the Kubernetes API. The agent will receive information about all deployments when it starts, and again for each deployment anytime the deployment's properties change.

Deployment Dimensions

Dimension Value
k8s.cluster <clusterName>
k8s.namespace <namespaceName>
k8s.deployment <deploymentName>

Deployment Metadata

Field Value
name <deploymentName>
type k8s.deployment
impactFromDimensions k8s.cluster=<clusterName>,k8s.namespace=<namespaceName>, k8s.cluster=<clusterName>,k8s.namespace=<namespaceName>,k8s.pod=<podName>, etc.

Deployment Metrics

Metric Name Type Units
k8s.deployment.generation GAUGE number
k8s.deployment.generation.observed GAUGE number
k8s.deployment.replicas GAUGE number
k8s.deployment.replicas.updated GAUGE number
k8s.deployment.replicas.ready GAUGE number
k8s.deployment.replicas.available GAUGE number
k8s.deployment.replicas.unavailable GAUGE number

Related Entities

The impactFromDimensions and impactToDimensions metadata fields described in Data are sent to Zenoss to create relationships between entities. These relationships can be seen in the Zenoss Smart View as Related Entities.

Specifically you should expect to see the following related entities for each type of entity published to Zenoss by this agent.

  • Cluster: Nodes in the cluster.
  • Node: No related entities.
  • Namespace: Cluster, and nodes in the cluster by extension.
  • Pod: Containers, namespace, and cluster and nodes in the cluster by extension.
  • Container: No related entities.
  • Deployment: Pods, namespace, and containers, cluster, and nodes by extension.