DevOps general Distributed Systems Infrastructure Messaging and Job Queues (RabbitMQ/Redis/...) pythonSee in schedule Download/View Slides
Many of us are using Kubernetes in production. A Kubernetes Operator is a way to automate packaging, deploying, and managing of a Kubernetes Application. It is a software alternative to a human operator who has deep knowledge of how to set up, deploy, and manage a particular piece of infrastructure and what to do if it isn’t behaving correctly. Let’s see how we can automate all of this while staying in the Python ecosystem.
It will be helpful to know some basic concepts of Kubernetes(Deployments, Services, Pods, Configmap etc.) and Celery(docs.celeryproject.org) to get the most out of this talk.
Talk is divided into four phases.
Phase I - Problems and Opportunities
We're going to see some simple examples/problems where a lot of manual effort is involved so as to connect audience to the problem.
We're going to discuss problems with configuration management, database cluster setup and introduce the focus problem of the talk which is around automating the setup of a Celery cluster.
Phase II - Incrementally Approaching the Solution
We're going to incrementally approach the automation each of the manual steps involved in running a Celery cluster in Production. We're going to discuss the extension capabilities in Kubernetes using CRDs and Custom Controllers which are going to help us manage our Celery cluster automagically.
Phase III - Celery Operator in action
We're going to see the code of custom controller and the whole operator in action. We create the newly defined celery resource and see how the operator works on bringing up the worker and flower deployments and handles autoscaling based on queue length.
Phase IV - Conclusion and Q&A
We're going to talk about different use-cases and what is world doing with Operators. We'll discuss the next steps for the Celery operator and some resources to help build operators. We'll end the talk with a Q&A.
Slides for the talk are available on - https://bit.ly/europython20-ppt
Celery Operator POC I built for this talk is open source - https://github.com/brainbreaker/Celery-Kubernetes-Operator
Type: Talk (45 mins); Python level: Beginner; Domain level: Intermediate
I'm a Software Engineer working with Grofers - largest online grocery shopping platform in India. I'm an open-source evangelist with decent experience building production-ready systems with Python and Kubernetes at a scale of million daily active users. I'm a Google Summer of Code'17 scholar and have contributed to codebases of LibreOffice, Mozilla and others in the past.
When I'm not coding, I give back to the community by organizing regular support-group circles, volunteer drives and providing emotional counselling to cancer patients and caregivers.