cdviz

keywords: cdevents, sdlc status: wip

A set of components to provide a view of which version of services are deployed and which environment, what test ran,...

• Collect every events (cdevents) related to the deployment of artifacts (service, ...), tasks, tests,...
• Dashboard to query and visualize events and states over time

Architecture Overview

cdviz-collector

Goals:

• to create cdevents by polling some sources (folder on fs, S3, AWS ECR, ...)
• to receive cdevents from http, kafka, nats
• to send (broadcast) cdevents to various destination database, http, kafka, nats
• to expose some metrics (TBD)

cdviz-collector is configured via a config file + override by environment variable.

flowchart LR
  classDef future stroke-dasharray: 5 5

  q>in memory queue of cdevents]

  subgraph sources
    src_http(HTTP)
    src_fs_content(FS folder with cdevents)
    src_fs_activity(FS folder activity):::future
    src_s3_content(S3 with cdevents)
    src_s3_activity(S3 activity):::future
    src_kafka(Kafka):::future
    src_nats(NATS):::future
    src_ecr(AWS ECR):::future
    src_misc(...):::future
  end
  src_http --> q
  src_fs_content --> q
  src_fs_activity --> q
  src_s3_content --> q
  src_s3_activity --> q
  src_kafka --> q
  src_nats --> q
  src_ecr --> q
  src_misc --> q

  subgraph sinks
    sink_stdout(stdout)
    sink_db(DB)
    sink_http(HTTP)
    sink_kafka(Kafka):::future
    sink_nats(NATS):::future
  end
  q --> sink_stdout
  q --> sink_http
  q --> sink_db
  q --> sink_kafka
  q --> sink_nats

Loading

Q&A

Why do dashboard tools, like Grafana, have access (read-only) to the DB (PostgreSQL), and go through an API (micro)service (like `cdviz-collector`)?

• access to the data is the value, not the service
• allow dashboards to use the full query power of SQL to query data, and to plug any analytics tools
• allow Data Ops to split the DB with read-only replicas if needed,...
• no need to create and provide a new custom (and always incomplete, frustrated) query language
  • no need to maintain a custom query language on the service side
  • no need to maintain a connector (datasource) for callers (analytics tools,...)
  • no need to re-invent (or to "encapsulate") what DB managers do better
• enforce the read-only view of the DB to be like a public API
  • require to configure access control
  • require to document and expose table structure (like an API) and to provide samples of queries, maybe function,...
• service can provide helper endpoints for some complex queries or additional "views"

Related projects

Maybe with some overlap:

• sassoftware/event-provenance-registry: The Event Provenance Registry (EPR) is a service that manages and stores events and tracks event-receivers and event-receiver-groups.
• RFC : CDEvents-translator design review by rjalander · Pull Request #42 · cdevents/community

Roadmap

• See ROADMAP.md
• See Issues · davidB/cdviz and use it to discuss features, ideas, bugs,...