Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

Databricks framework to validate Data Quality of pySpark DataFrames

Benchmarks for data processing systems: Pathway, Spark, Flink, Kafka Streams

Apache Spark 3 - Structured Streaming Course Material

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

A data engineering project (Twitter monitor app)

Final Project for IoT: Big Data Processing and Analytics class. Analyzing U.S nationwide temperature from IoT sensors in real-time

For a series of posts on Amazon MSK, Amazon EKS, and Amazon EMR

Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO

Custom PySpark Data Sources

Toolbox for building Generative AI applications on top of Apache Spark.

Counting Tweets Per User in Real-Time

A data and analytics engineering platform designed for real-time sports betting analytics.

Apache Spark Guide

RealTime StockStream is a streamlined, simulation system for processing live stock market data. It uses Apache Kafka for data input, Apache Spark for data handling, and Apache Cassandra for data storage, making it a powerful yet easy-to-use tool for financial data analysis

Solution for IBM Data Engineering Professional Certificate

Databricks. Incremental data processing, task orchestration, and production job monitoring.

Video processing (webcam) in real time using Kafka and Spark.

Repository of notebooks and related collateral used in the Databricks Demo Hub, showing how to use Databricks, Delta Lake, MLflow, and more.

A real time analytics dashboard to analyze the trending hashtags and @ mentions at any location using kafka and spark streaming.