Data Engineer Bootcamp Notes

Welcome to my personal notes for the Data Engineer Handbook Bootcamp.

This repository is my learning journal, containing summaries, key concepts, and lab solutions for the 6-week bootcamp. It complements my forked Data Engineer Handbook repo.

Repository Structure

data-engineer-notes/
├── README.md
├── resources.md
├── assets/
├── images/
├── week00/
│   ├── summary.md
│   ├── key-concepts.md
│   ├── lab-notes.md
│   └── lab00/
│       ├── solution.ipynb
│       └── ...     # Artifacts from bootcamp materials
├── week01/
│   └── (similar structure)
├── ...
└── week06/
    └── (similar structure)

Week Notes

Each week contains:

Summary: Key takeaways from the week
Key Concepts: Detailed explanations and examples of core ideas
Lab Notes: Observations, detailed notes, and troubleshooting during labs
Labs: Solutions for each lab

Links

Original Handbook: DataExpert-io/data-engineer-handbook
My Fork: pizofreude/data-engineer-handbook

Learning Progress

💻 Daily Practice System

This repository now includes a comprehensive practice tracking system to organize daily coding practice across multiple platforms:

practice/ - Platform-organized coding problems (LeetCode, StrataScratch, HackerRank, NeetCode, Codewars, etc.)
concepts/ - Reference notes on data structures, algorithms, SQL patterns, and system design
interview-prep/ - Interview-specific preparation materials (behavioral, technical, system design)
logs/ - Daily practice logs and progress tracking with statistics dashboard

Quick Start

# Create today's log entry
./scripts/new-day.sh

# Start a new problem
# ./scripts/create-problem.sh <platform> <difficulty> "problem-name"
./scripts/create-problem.sh leetcode medium "problem-name"

# Create a concept note
# ./scripts/link-concept.sh "concept-name" <category>
./scripts/link-concept.sh "Window Functions" sql-patterns

# Generate weekly stats
python scripts/generate-stats.py

✅ Manual Update Checklist: For Every Problem You Solve

Here's your streamlined checklist for logging each problem.

📋 The 5-Step Workflow

Step 1: Start Your Day ⏱️ 30 seconds

Run once per day (first thing in the morning):

./scripts/new-day.sh

✅ Done! No manual edits needed for this step.

Step 2: Scaffold the Problem ⏱️ 30 seconds

For each new problem you're about to solve:

./scripts/create-problem.sh <platform> <difficulty> "<problem-slug>"

Examples:

./scripts/create-problem.sh codewars easy "absolute-value-log-base"
./scripts/create-problem.sh leetcode medium "rank-scores"
./scripts/create-problem.sh stratascratch hard "revenue-analysis"

✅ Done! Folder created, template copied, ready to code.

Step 3: Write Your Solution ⏱️ 10-30 minutes (solving time)

Navigate to the problem folder:

cd practice/<platform>/<difficulty>/<problem-slug>

Open and write your solution:

code solution. sql    # For SQL problems
# OR
code solution.py     # For Python/algorithm problems

What to do:

✏️ Paste your working solution code
✏️ Add comments explaining key logic (optional but recommended)
💾 Save the file

Example:

-- Calculate absolute value and logarithm base 64
SELECT
  ABS(number1) AS abs,
  LOG(64, number2) AS log
FROM decimals;

Step 4: Document Your Solution ⏱️ 10-15 minutes

Open the notes file:

code notes.md

You need to manually update these sections:

A. Metadata (Top of file)

# [Problem Name]  ← CHANGE THIS

## 📋 Metadata
- **Platform:** [Platform name]  ← CHANGE THIS
- **Difficulty:** [Easy/Medium/Hard] (Optional:  add platform rating like "7 kyu")  ← CHANGE THIS
- **Date Solved:** 2026-01-03  ← ✅ ALREADY FILLED BY SCRIPT
- **Time Spent:** XX minutes  ← CHANGE THIS
- **Status:** [✅ Solved | 🔄 Revisit | ❌ Stuck]  ← CHANGE THIS

Example:

# Absolute Value and Log to Base

## 📋 Metadata
- **Platform:** Codewars
- **Difficulty:** Easy (7 kyu)
- **Date Solved:** 2026-01-03  ← Script filled this
- **Time Spent:** 15 minutes
- **Status:** ✅ Solved

B. Links

## 🔗 Links
- [Problem URL]  ← PASTE THE ACTUAL URL HERE

Example:

## 🔗 Links
- https://www.codewars.com/kata/594a8f2f7ca3c692a4000041/train/sql

C. Topics & Tags (Check the boxes)

## 📚 Topics & Tags
- [ ] SQL
- [ ] Window Functions
- [ ] Joins
- [ ] CTEs
- [ ] Python
- [ ] Dynamic Programming

Check the relevant ones:

##📚 Topics & Tags
- [x] SQL  ← Put 'x' inside
- [x] Mathematical Functions
- [ ] Window Functions
- [ ] Joins

D. Problem Statement

## 📝 Problem Statement
[Paste the problem description here]

### Example Input/Output
```markdown
Input: 
Output:

What to do:

✏️ Copy-paste the problem description from the platform
✏️ Add example input/output (if provided)

E. Approach

## 💡 Approach

### Initial Thoughts
[What was your first idea?  What patterns did you recognize?]

### Solution Strategy
1. Step 1
2. Step 2
3. Step 3

What to do:

✏️ Write your thought process (2-3 sentences)
✏️ List the steps you took (bullet points)

Example:

## 💡 Approach

### Initial Thoughts
Straightforward application of SQL math functions:  ABS for absolute value, LOG for logarithm with custom base.

### Solution Strategy
1. Use `ABS(number1)` to get absolute values
2. Use `LOG(64, number2)` for logarithm base 64
3. Alias columns as required (`abs`, `log`)

F. Solution

## 🖥️ Solution

### Attempt 1 (Initial)
```sql
-- Your first solution here

Result: [Passed/Failed/Timeout]


**What to do:**
- ✏️ Paste your solution code (can be same as `solution.sql`)
- ✏️ Note if it passed or failed

**If you optimized it, add:**
```markdown
### Attempt 2 (Optimized) ⭐
```sql
-- Improved solution

Result: ✅ Passed with better performance


---

#### **G. Complexity Analysis**
```markdown
## ⚡ Complexity Analysis
- **Time Complexity:** O(?)
- **Space Complexity:** O(?)

What to do:

✏️ Fill in the Big O notation
✏️ If you don't know, write: "Time: O(n) - single pass through table"

Example:

## ⚡ Complexity Analysis
- **Time Complexity:** O(n) - single pass through table
- **Space Complexity:** O(n) - result set same size as input

H. Key Learnings

## 🎓 Key Learnings
1. 
2. 
3.

What to do:

✏️ Write 2-4 things you learned (this is THE MOST IMPORTANT SECTION!)

Example:

## 🎓 Key Learnings
1. **ABS()** - Returns absolute value (distance from zero)
2. **LOG(base, value)** - PostgreSQL syntax for custom base logarithm
3. PostgreSQL uses `LOG(base, value)` while MySQL uses `LOG(value) / LOG(base)`
4. Base-64 logarithm:  `LOG(64, 4096) = 2` because 64² = 4096

I. Related Concepts (Optional)

## 🏷️ Related Concepts
See:  `concepts/sql-patterns/[concept-file]. md`

What to do:

✏️ If you created a concept note, link it here
⏭️ Skip if you haven't created a concept yet

Example:

## 🏷️ Related Concepts
See:  `concepts/sql-patterns/sql-mathematical-functions.md`

Step 5: Update Daily Log ⏱️ 5 minutes

Open today's log:

cd ../../../../    # Return to repo root
code logs/2026/01-january. md

Find today's date section and fill in:

A. Time Spent

### Friday, January 03, 2026
⏱️ Time:   X hours  ← CHANGE THIS

Example:

⏱️ Time:  1.5 hours

B. Problems Completed

#### ✅ Completed
1.

Add each problem with:

Problem name and difficulty
Key topics
Link to your solution
One-line key learning

Example:

#### ✅ Completed
1. **Codewars - Absolute Value and Log to Base** (Easy/7kyu)
   - Topics: ABS(), LOG(), Mathematical functions
   - [Solution](../../practice/codewars/easy/absolute-value-log-base/)
   - Key learning: PostgreSQL LOG(base, value) syntax differs from MySQL

2. **LeetCode 178 - Rank Scores** (Medium)
   - Topics: Window functions, DENSE_RANK
   - [Solution](../../practice/leetcode/medium/178-rank-scores/)
   - Key learning:  DENSE_RANK vs RANK vs ROW_NUMBER differences

C. Learnings

#### 💡 Learnings
-

Write 2-4 broader learnings from today:

Example:

#### 💡 Learnings
- Mathematical functions in SQL are database-specific (PostgreSQL vs MySQL syntax)
- Always check for NULL values when using LOG() with user input
- ABS() is useful for calculating distances and differences
- Created concept note:  `concepts/sql-patterns/sql-mathematical-functions.md`

D. Tomorrow's Plan

#### 🎯 Tomorrow
- [ ]

Plan 2-3 things for tomorrow:

Example:

#### 🎯 Tomorrow
- [ ] LeetCode 180 - Consecutive Numbers (Window functions practice)
- [ ] StrataScratch - Revenue analysis problem
- [ ] Review:  Self-joins pattern

Step 6: Commit & Push ⏱️ 1 minute

git status

# Add your changes
git add practice/<platform>/<difficulty>/<problem-slug>/
git add logs/2026/01-january. md

# If you created a concept note, add it too
git add concepts/

# Commit with descriptive message
git commit -m "✅ [Platform]:  [Problem Name] - [Key Topic]"

# Push to GitHub
git push

Example commit messages:

git commit -m "✅ Codewars:  Absolute Value and Log to Base - SQL math functions"
git commit -m "✅ LeetCode 178: Rank Scores - Window functions"
git commit -m "✅ StrataScratch: Revenue Analysis - CTEs and aggregations"

📊 Weekly: Update Stats Dashboard ⏱️ 5 minutes

Run every Sunday (or end of week):

python scripts/generate-stats.py

Copy the output:

## 📊 All-Time Stats

| Platform      | Easy | Medium | Hard | Total |
|---------------|------|--------|------|-------|
| Codewars      | 5    | 2      | 0    | 7     |
| Leetcode      | 12   | 8      | 1    | 21    |
| **Total**     | **17** | **10** | **1** | **28** |

📅 Last Updated: 2026-01-05 20:30

Paste it into:

code logs/README.md

Replace the old stats section with the new output.

Also update:

## 🔥 Current Streaks
- **Daily Practice:** X days  ← UPDATE THIS MANUALLY

Commit:

git add logs/README.md
git commit -m "📊 Update weekly practice stats"
git push

✅ Quick Reference Checklist

Print this and keep it next to you:

□ Step 1: ./scripts/new-day.sh (once per day)

For each problem: 
□ Step 2: ./scripts/create-problem.sh <platform> <difficulty> "<slug>"
□ Step 3: Write solution in solution.sql or solution.py
□ Step 4: Fill in notes.md:
    □ Change title
    □ Update metadata (platform, difficulty, time, status)
    □ Paste problem URL
    □ Check topic tags
    □ Paste problem statement
    □ Write approach & strategy
    □ Paste solution code
    □ Add complexity analysis
    □ Write key learnings (MOST IMPORTANT!)
    □ Link concept note (if created)

□ Step 5: Update logs/2026/01-january.md:
    □ Time spent today
    □ Add problem to "Completed" list
    □ Write today's learnings
    □ Plan tomorrow's focus

□ Step 6: git add → commit → push

Weekly: 
□ Sunday: Run generate-stats.py
□ Update logs/README.md & the monthly log + practice/README.md with new stats

💡 Time-Saving Tips

Minimal Version (10 min per problem)

If you're short on time, focus on:

✅ Solution code (solution.sql)
✅ Key learnings in notes.md
✅ Daily log entry

Skip the rest for now, come back later to fill in.

Batch Update

If you solve multiple problems:

Scaffold all problems first
Solve all problems
Update all notes.md files
Update daily log once (list all problems)
Single commit at the end

Use Snippets/Shortcuts

Create editor snippets for repetitive sections like complexity analysis, common tags, etc.

🎯 Summary: What You MUST Do Manually

File	What to Update
solution.sql	Your code
notes.md	Title, metadata, URL, approach, learnings
logs/YYYY/MM-month.md	Time, problems list, learnings, tomorrow's plan
logs/README.md	Weekly stats (copy from script output)

Everything else is automated! 🎉

Features

Automation Scripts: Quickly scaffold new problems and logs with templates
Platform-Agnostic: Automatically discovers and tracks any coding platform
Comprehensive Templates: Detailed templates for problems, concepts, and daily logs
Progress Tracking: Statistics generation and progress dashboards
Knowledge Base: Structured concept notes linked to practice problems

See practice/README.md for detailed usage instructions and workflow.

Name		Name	Last commit message	Last commit date
Latest commit History 159 Commits
assignments		assignments
concepts		concepts
images/week00		images/week00
interview-prep		interview-prep
logs		logs
practice		practice
scripts		scripts
templates		templates
week00		week00
week01		week01
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
aie_interview.md		aie_interview.md
de_interview.md		de_interview.md
distributed_computing.md		distributed_computing.md
ml_ai.md		ml_ai.md
pipelines.md		pipelines.md
python-dsa-notes.md		python-dsa-notes.md
resources.md		resources.md

License

pizofreude/data-engineer-notes

Folders and files

Latest commit

History

Repository files navigation

Data Engineer Bootcamp Notes

Repository Structure

Week Notes

Links

Learning Progress

💻 Daily Practice System

Quick Start

✅ Manual Update Checklist: For Every Problem You Solve

📋 The 5-Step Workflow

Step 1: Start Your Day ⏱️ 30 seconds

Step 2: Scaffold the Problem ⏱️ 30 seconds

Step 3: Write Your Solution ⏱️ 10-30 minutes (solving time)

Step 4: Document Your Solution ⏱️ 10-15 minutes

A. Metadata (Top of file)

B. Links

C. Topics & Tags (Check the boxes)

D. Problem Statement

E. Approach

F. Solution

H. Key Learnings

I. Related Concepts (Optional)

Step 5: Update Daily Log ⏱️ 5 minutes

A. Time Spent

B. Problems Completed

C. Learnings

D. Tomorrow's Plan

Step 6: Commit & Push ⏱️ 1 minute

📊 Weekly: Update Stats Dashboard ⏱️ 5 minutes

✅ Quick Reference Checklist

💡 Time-Saving Tips

Minimal Version (10 min per problem)

Batch Update

Use Snippets/Shortcuts

🎯 Summary: What You MUST Do Manually

Features

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Contributors 3

Uh oh!

Languages

Packages