Open source platform for the machine learning lifecycle
-
Updated
May 5, 2025 - Python
#
apache-spark
Apache Spark is an open source distributed general-purpose cluster-computing framework. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.
Here are 491 public repositories matching this topic...
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
python airflow spark apache-spark scheduler s3 data-engineering data-lake warehouse redshift data-migration livy etl-framework apache-airflow emr-cluster etl-pipeline etl-job data-engineering-pipeline airflow-dag goodreads-data-pipeline
-
Updated
Mar 9, 2020 - Python
PySpark + Scikit-learn = Sparkit-learn
-
Updated
Dec 31, 2020 - Python
(Deprecated) Scikit-learn integration package for Apache Spark
-
Updated
Dec 3, 2019 - Python
pyspark methods to enhance developer productivity 📣 👯 🎉
-
Updated
Mar 6, 2025 - Python
A command-line tool for launching Apache Spark clusters.
-
Updated
Dec 13, 2024 - Python
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
data-science machine-learning apache-spark deep-learning hadoop tensorflow keras optimization-algorithms data-parallelism distributed-optimizers
-
Updated
Jul 25, 2018 - Python
PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
-
Updated
Oct 15, 2024 - Python
A boilerplate for writing PySpark Jobs
-
Updated
Jan 21, 2024 - Python
Train and run Pytorch models on Apache Spark.
-
Updated
May 11, 2023 - Python
Code for "Efficient Data Processing in Spark" Course
-
Updated
Oct 1, 2024 - Python
Easy to use library to bring Tensorflow on Apache Spark
-
Updated
Oct 11, 2023 - Python
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
-
Updated
Sep 3, 2024 - Python
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
docker big-data cassandra apache-spark data-storage postgresql data-engineering apache-kafka data-processing data-pipeline real-time-analytics containerization apache-zookeeper apache-airflow etl-pipeline
-
Updated
Feb 14, 2025 - Python
Dataproc templates and pipelines for solving in-cloud data tasks
-
Updated
Apr 2, 2025 - Python
Apache Spark 3 - Structured Streaming Course Material
-
Updated
Aug 19, 2023 - Python
Apache (Py)Spark type annotations (stub files).
-
Updated
Aug 17, 2022 - Python
Real-Time Financial Market Data Processing and Prediction application
python apache-spark mariadb pytorch neural-networks scrapy apache-kafka financial-data structured-streaming
-
Updated
Jul 23, 2023 - Python
Astronomy Broker based on Apache Spark
-
Updated
May 2, 2025 - Python
Solution Accelerators for Serverless Spark on GCP, the industry's first auto-scaling and serverless Spark as a service
-
Updated
May 3, 2024 - Python
Created by Matei Zaharia
Released May 26, 2014
- Followers
- 428 followers
- Repository
- apache/spark
- Website
- spark.apache.org
- Wikipedia
- Wikipedia