Running BigData Applications with Microservices and Airflow
2 min readMar 29, 2019
- Build a Kafka cluster for middleware and message passing.
- Build Microservice with Kafka using Springboot
- Build Spark job repository for jobs to be exposed as service
- Setup Airflow and read parameter Json files from Microservice application.
Kafka cluster start :
create some topic in kafka which can be used to pass json message.
SpringBoot Kafka Microservice :
Spark Jobs as Service :
Job reads the kafka topic and prints the message also writes it to parameter file in Airflow execution directory.
Airflow setup and Dag creation through parameter file :
This Dag will run and execute Spark jobs to work upon your data in datalake. We can create dynamic pipelines and execute our pipeline from the front-end with ease.