-
Notifications
You must be signed in to change notification settings - Fork 130
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
73 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,73 @@ | ||
--- | ||
date: 2024-05-12 | ||
time: 20h:00min | ||
duration: "1:45:17" | ||
title: "Data engineer 101" | ||
tags: ["dev"] | ||
category: "dev" | ||
isNext: false | ||
youtube: https://www.youtube.com/live/mxV9Bx1ZsZg?si=5QnDE6RCcNOBuW1T | ||
published: true | ||
featured: false | ||
--- | ||
|
||
Data engineering is a critical field in data science that involves preparing the "big data" infrastructure to be analyzed by data scientists. In this episode we are discussing the differences and how important each is with our guests. | ||
|
||
## Guests | ||
|
||
- [Mahmoud Fettal](https://twitter.com/mahmoudfettal) | ||
|
||
- [Salim Jannah](https://www.linkedin.com/in/salim-janah) | ||
|
||
- [Omaima Khalil](https://twitter.com/BadQuinn3) | ||
|
||
|
||
## Notes | ||
|
||
0:00:00 - Introduction and welcoming | ||
|
||
0:02:50 - What is data engineering? | ||
|
||
0:08: 43 - What are the key skills required for a data engineer? | ||
|
||
0:16:40 - How does data engineering differ from data science? | ||
|
||
0:20:00 - Data analyst vs data engineer vs data scientist | ||
|
||
0:22:41 - What are the common tools used in data engineering? | ||
|
||
0:28:57 - What are data pipelines? | ||
|
||
0:34:54 - What challenges do data engineers face? | ||
|
||
0:42:12 - Q&A | ||
|
||
0:53:42 - How important is real -time data processing in data engineering? | ||
|
||
1:02:35 - What is a data lake, and how does it differ from a data warehouse? | ||
|
||
1:12:52 - How do data engineers use machine learning? | ||
|
||
1:18:01 - Types of projects really involved with Data engineering | ||
|
||
1:32:17 - What future trends should data engineers be aware of? | ||
|
||
1:41:00 - Geeksblabla Picks | ||
|
||
2:18:30 - Conclusion and Goodbye | ||
|
||
|
||
## Links | ||
|
||
- [Apache Airflow vs Mage.ai](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) | ||
|
||
- [Lakehouse paper](https://medium.com/odicis-data-engineering/apache-airflow-vs-mage-ai-in-data-engineering-745c040a05e8) | ||
|
||
- [Open Source Agent for Data Analysis](https://pandas-ai.com/) | ||
|
||
- [Simplifying Data Engineering and Analytics with Delta](https://www.packtpub.com/product/simplifying-data-engineering-and-analytics-with-delta/9781801814867) | ||
|
||
|
||
## Prepared and Presented by | ||
|
||
- [Meriem Zaid](https://twitter.com/_iMeriem) |