README.md

📚 Documentation

Table of contents

• Development experience

  • Setup and install
  • Tutorials
  • Concepts
    • Core design principles
    • Core abstractions
    • Blocks
  • Data pipeline management
    • UI overview
  • Kernels
    • Spark and PySpark
  • Testing
    • Data validation (WIP)
  • Templates

• Operating in production

  • Deploying to cloud
    • Using Terraform
    • AWS
  • Configuring production settings
    • Compute resource
    • Runtime variables
  • Triggering pipelines
    • Trigger pipeline from AWS Event (WIP)
  • Monitoring and alerting
    • Pipeline run updates in Slack (WIP)
  • Integrating with other tools
    • Airflow
    • Prefect

Templates

io_config.yaml

This YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

Example