DATA_INFRA.md

Reading

• Machine Learning: The High Interest Credit Card of Technical Debt
• The Care & Feeding of Data Scientists
• Designing Data-Intensive Applications
• Storytelling with Data: A Data Visualization Guide for Business Professionals

Distributed Frameworks

• spark
• https://github.com/ray-project/ray
• https://dask.org/

Warehouse

• redshift
• snowflake
• bigquery
• vertica

Data Lake

• databricks delta
• https://github.com/apache/iceberg
• https://github.com/apache/hudi
• https://www.dremio.com/

Change data capture / streaming

• https://www.attunity.com/
• https://streamsets.com/
• https://github.com/debezium/debezium

data versioning

• https://projectnessie.org/

OLAP

• https://pinot.apache.org/
• https://kylin.apache.org/

Streaming

• kafka
• flink
• beam

real-time/document-search

• elasticsearch
• rockset

Notebooks / exploration

• jupyter
• https://zeppelin.apache.org/
• https://github.com/ironmussa/Optimus
• https://colab.research.google.com/
• https://www.saturncloud.io/
• https://rmarkdown.rstudio.com/
• https://www.querybook.org/
• https://deepnote.com/
• https://cnvrg.io/
• dominodatalab

Labeling

• https://www.snorkel.org/
• https://www.figure-eight.com/
• https://scale.ai
• https://www.labelbox.com

Notebook management

• https://github.com/nteract/papermill

Orchestration

• airflow
• luigi
• oozie
• https://github.com/dagster-io/dagster
• https://github.com/PrefectHQ/prefect
• https://www.getdbt.com/product/

Dataflow/transformation managers

• streamset
• nifi

unstructured/semi-structured data prep

• https://www.trifacta.com/

Solutions

• snowplow
• https://www.prefect.io/
• https://www.ascend.io/
• https://www.datmo.com/
• TFX-OSS

data quality monitoring

• https://www.anomalo.com/
• https://github.com/awslabs/deequ

anomaly detection as a service

• https://www.anodot.com/

event data

• segment
• Rudderstack
• Metarouter
• snowplow

A/B testing

• https://www.optimizely.com/
• https://www.abtasty.com/
• https://github.com/YahooArchive/mendel

Behavioral analytics

• https://www.fullstory.com/
• https://www.hotjar.com/

Feature flagging

• https://launchdarkly.com/
• https://www.split.io/
• https://github.com/checkr/flagr

Catalogue / metadata

• https://github.com/airbnb/knowledge-repo
• https://github.com/lyft/amundsen
• https://eng.uber.com/databook/
• https://medium.com/netflix-techblog/metacat-making-big-data-discoverable-and-meaningful-at-netflix-56fb36a53520
• https://cloud.google.com/data-catalog/
• https://github.com/linkedin/datahub
• airbnb: democratizing data, airbnb: scaling knowledge

BI

• superset
• looker
• periscope

etl / ingestion-as-a-service

• fivetran
• singer.io
• talend
• https://github.com/airbytehq/airbyte
• https://github.com/linkedin/brooklin

reverse-etl: piping data around for salesOps/marketingOps/CRM etc

• https://tray.io/
• https://www.snaplogic.com/
• https://www.polytomic.com/
• https://www.getcensus.com/

Sales analytics

• https://www.chorus.ai/
• https://www.gong.io/

timeseries forecasting

• https://github.com/linkedin/greykite

Neat utils

• query CSVs w/ SQL: cq
• jq

Generating fake data

• https://github.com/lk-geimfari/mimesis

privacy / compliance

• https://www.datagrail.io/
• immuta
• privacera
• ethyca
• osano