README.md

Python for Data Engineering Course

01: Python Basics

• Introduction
• First Python program
• Basic Data types (int, float, str, bool)
• Variables, Constants and operators
• Control flow (if statements, loops)
  • if Statement
  • for Loop
  • while Loop
  • Control flow examples
• Data structures (lists, tuples, dictionaries, sets)
  • Lists
  • Tuples
  • Dictionaries
  • Sets
• String manipulation
• Modular Programming (introduction)
  • Functions
  • Modules
• Error handling
• Assignments: Python basics assignments

02: Data Manipulation, Analysis and Visualization

• Introduction to Data Analysis
• Setting Up Jupyter Notebook and Pandas
• Data Manipulation with Pandas
  • Introduction to Pandas
  • Basics statistics in data analysis (descriptive statistics, shape of distribution, inferential statistics)
  • Pandas Series
  • Pandas DataFrame
  • Reading and writing data with Pandas
  • Accessing and selecting data
  • Data manipulation (filtering, sorting, grouping)
  • Data Cleaning
  • Merge, Join and Concatenating
• Numerical computations with NumPy
  • NumPy array operations (slicing, indexing)
  • NumPy array manipulation
  • Working with random numbers in NumPy
  • Analyzing sales data with NumPy
• Data visualization Basics
  • Line plot, Scatter plot, and Bar plot
  • Histogram, Box plot, and Violin plot
• Assignment 1 : Analyzing E-commerce Orders using Pandas
• Assignment 2 : Exploratory Data Analysis (EDA) using Pandas, NumPy, and Matplotlib

04: Web Scraping and APIs

• Introduction to Web Scraping and APIs
• Basics of HTML, CSS, and JavaScript
  • Introduction to HTML
  • Introduction to CSS
  • Introduction to JavaScript
• Getting Started with Web Scraping Using Python
  • HTTP Requests
  • HTTP Response
  • Fetch Web Pages Using Requests Library
  • Parsing HTML with BeautifulSoup
• Dealing with Pagination and Infinite Scrolling
• Working with APIs
  • Handling API Response
  • Error Handling and Status Codes
• Assignment 1: Scraping Jobinja website, fetch last year jobs information and save into a CSV file.

05: Working with Data Sources, Storages, and Serialization

• Introduction to I/O (Input/Output)
  • Common File Formats
• Importance of File I/O (Input/Output)
• Basic Concepts of File I/O
  • File Paths and Directories
  • Opening and Closing Files
• Text Files
• CSV Files
• JSON (JavaScript Object Notation) Files
• XML Files
• Excel Files
• Binary Files
• Database File Formats (SQLite)
• Parquet and pyArrow

06: Object-Oriented Programming

• Introduction to Object-Oriented Paradigm
  • Definition of Object-Oriented
  • Historical Background
  • Understanding Object-Oriented Development
  • Object-Oriented Analysis, Design, and Programming
  • Fundamental Principles of Object-Oriented Paradigms
• Object-OrientedAnalysis(OOA)
  • Software Modeling Techniques
  • Object-Oriented Analysis (OOA)
  • UML Diagrams
    • UML Diagrams in Different Phases of Software Development
    • Use Case Diagram
• Object-Oriented Design (OOD)
  • Core Concepts of OOD
  • Fundamental Principles of OOD
    • Abstraction
    • Encapsulation
    • Inheritance
    • Polymorphism
  • Stages of Object-Oriented Design (OOD)
    • System Design
    • Object Design
  • Key Processes in Object-Oriented Design (OOD)
  • Controlling Events and Boundary Conditions
  • Documentation and Communication
  • Introduction to Design Patterns
• Introduction to Object-Oriented Programming (OOP)
  • Python Classes and Objects
  • Object-Oriented Programming Principles in Python
    • 1. Encapsulation
      • Encapsulation in Data Pipelines
      • Encapsulation in Python's Data Structures
      • Custom Data Structures with Encapsulation
      • Encapsulation and Performance
    • 2. Inheritance
      • What is Inheritance?
      • Basic Syntax
      • Accessing Parent Class Methods
      • Multiple Inheritance in Python
      • Mixin
      • Inheritance Example: Put all togather
      • Inheritance in data pipelines
    • 3. Abstraction
      • Introduction to Abstraction and Its Applications
      • What is Abstraction?
      • Creating an Abstract Base Class
      • Abstract Methods
      • Abstract Properties
      • Abstract Method vs Abstract Properties
      • Abstract Methods with Default Implementations
      • Using Abstract Base Classes with Multiple Inheritance
    • 4. Polymorphism in Python
      • Basic Example of Polymorphism
      • Method Overriding
      • Polymorphism with Inheritance
      • Duck Typing
      • Method Overloading and Polymorphism
      • Polymorphism with Operator Overloading
      • Abstract Base Classes and Polymorphism
    • Encapsulation vs Inheritance vs Abstraction vs Polymorphism
    • Dunder Methods
    • Class Relationships
• Object-Oriented Design Principles
  • SOLID principles
  • DRY (Don't Repeat Yourself)
  • KISS (Keep It Simple, Stupid)
  • YAGNI (You Aren't Gonna Need It)
  • Separation of Concerns (SoC)
  • Composition Over Inheritance
  • Law of Demeter (LoD)

07: SQL & NoSQL Databases with Python

Working with SQL Databases

• Introduction to SQL and relational databases
• SQL basics (SELECT, FROM, WHERE, JOIN)
• Creating and managing databases, tables, and indexes
• CRUD operations (Create, Read, Update, Delete)
• Connecting to databases
• Executing SQL queries
• Fetching and manipulating data with SQL
• Using SQLAlchemy for database interaction

Working with NoSQL Databases

• Understanding NoSQL databases (e.g., MongoDB, Redis)
• Connecting to NoSQL databases
• Querying and manipulating data in NoSQL databases
• Handling document-based and key-value data models

Assignment: Ass7

08: Building Data Pipelines

Data Pipelines

• ETL (Extract, Transform, Load)
• Understanding data pipelines and their components
• Designing and architecting data pipelines
• Implementing data ingestion, transformation, and loading (ETL)

Assignment: Ass8

Capstone Project

Project Development

• Apply all the concepts learned in a real-world data engineering project
• Work with various data sources including web data and APIs
• Implement ETL pipelines, data processing, and analysis using Python libraries and tools