Section outline

• Select section Day 1 – Big Data and HDFS Fundamentals

  Day 1 – Big Data and HDFS Fundamentals

  Collapse all Expand all

  • Big Data Concepts, Challenges

  • Hadoop & HDFS Architecture

  • Data Management in Hadoop

  • Lab: HDFS Commands, VM Setup
• Select section Day 2 – Apache Hive & Spark Core

  Day 2 – Apache Hive & Spark Core

  • Hive Architecture, ETL with Hive

  • Working with CSV, JSON, Parquet

  • Spark Overview & Deployment Modes

  • RDD Operations, Regex, Pair RDD

  • Lab: Hive ETL, Spark RDD Examples
• Select section Day 3 – Spark Internals & DataFrames

  Day 3 – Spark Internals & DataFrames

  • Spark DAG, Shuffle, Stages, Job Metrics

  • Performance Tuning: Memory, Executors, Caching

  • Setting up Spark on YARN, Kubernetes

  • Intro to DataFrames, Catalyst, Tungsten

  • Lab: Metrics, Caching, DataFrame Operations
• Select section Day 4 – Advanced DataFrames & Delta Lake

  Day 4 – Advanced DataFrames & Delta Lake

  • Spark SQL, HiveContext, JDBC Integration

  • Joins, Bucketing, Analytical Queries

  • BI Tool Integration

  • Delta Lake: ACID Transactions, Time Travel

  • Lab: Delta Table Management, Format Conversions
• Select section Day 5 – Structured Streaming & Kafka

  Day 5 – Structured Streaming & Kafka

  • Structured Streaming Concepts

  • Micro-Batch Triggers, Late Data, Joins

  • Kafka Architecture & Multi-node Setup

  • Spark-Kafka Integration for Real-time Apps

  • Lab: Twitter Stream Analysis, Kafka Receiver