JaFebruary 15, 2020
Migration of Apache Spark to Kubernetes
In this project, we migrated Spark applications running on VMs to Kubernetes.
We tailored the system configuration more towards Kubernetes, as the client was already Kubernetes cluster-oriented.
Design Phase
Initially, we gained an understanding of the existing system and gradually migrated it.
We also ensured compatibility with running Spark applications in containers, making necessary code adjustments to ensure no impact on existing functionality.
We discussed the design utilizing the Spark-on-Kubernetes Operator available at this link.
Setup/Migration Phase
We scheduled weekly meetings for progress updates and conducted discussions on service stoppage and deadlines during migration. For services with high SLA requirements, we handled the migration with extra caution.
Delivery
Upon completion of all development tasks, we tested the functionalities within the defined scope, shared various configuration details, delivered the source code, and deployed the application.
Thank you for your cooperation!