The most popular Container Orchestration solutions available in the market are Kubernetes, Swarm and Mesos. I have used Kubernetes and Swarm, but never got a chance to use Mesos or DC/OS. There were a bunch of questions I had about Mesos and DC/OS and I never got the time to explore that. Recently, I saw the announcement about Mesosphere opensourcing DC/OS and I found this as a perfect opportunity for me to try out Opensource DC/OS. In this blog, I have captured the answers to questions I had regarding Mesos and DC/OS. In the next blog, I will cover some hands-on that I did with Opensource DC/OS.
What is the relationship between Apache Mesos, Opensource DC/OS and Enterprise DC/OS?
Apache Mesos is the Opensource distributed orchestrator for Container as well as non-Container workloads. Both Opensource DC/OS and Enterprise DC/OS are built on top of Apache Mesos. Opensource DC/OS adds Service discovery, Universe package for different frameworks, CLI and GUI support for management and Volume support for persistent storage. Enterprise DC/OS adds enterprise features around security, performance, networking, compliance, monitoring and multi-team support that the Opensource DC/OS project does not include. Complete list of differences between Opensource and Enterprise DC/OS are captured here.
What does Mesosphere do and how it is related to Apache Mesos?
Mesosphere company has products that are built on top of Apache Mesos. Lot of folks working in Mesosphere contribute to both Apache Mesos and Opensource DC/OS. Mesosphere has the following products currently:
- DC/OS Enterprise – Orchestration solution
- Velocity- CI, CD solution
- Infinity – Big data solution
Why DC/OS is called OS?
Sometimes folks get confused thinking Mesos being a Container optimized OS like CoreOS, Atomic. Mesos is not a Container optimized OS. Similar to the way Desktop OS provides resource management in a single host, DC/OS provides cluster management across entire cluster. Mesos master(including first level scheduler) and agent are perceived as kernel components and user space components include frameworks, user space applications, dns and load balancers. The kernel provides primitives for the frameworks.
What are Mesos frameworks and why they are needed?
Following diagram from Mesos Architecture page shows the highlevel architecture blocks:
Mesos provides a two level scheduler. First level scheduler in Mesos Master gives resources to each frameworks and each framework takes care of allocating resource within its jobs. Each framework takes care of specific constraints associated with the particular application. This allows for independent development of frameworks without modifying primary scheduler within Mesos Master. Mesos has frameworks in categories like Long running generic tasks, Short running batch processing tasks, Big data processing, Distributed filesystem/database. Following are some examples of Mesos frameworks:
- Long lived jobs – Marathon, Aurora
- Short lived jobs or batch jobs – Chronos, Jenkins
- Big data – Hadoop, Spark, Storm
- Database – Cassandra
Following link captures all available frameworks. By default, Mesos includes a strict priority resource allocation module and a modified fair sharing resource allocation module in Mesos Master. Users can add their own allocation modules if more flexibility is desired.
High level comparison between Mesos vs Kubernetes vs Docker Swarm?
Following table shows the high level Comparison of the 3 popular Orchestration solutions. As new features get developed, this table needs to be updated accordingly.
|Workload type||Container and Non-Container||Container||Container|
|Deployment unit||Process or Container||Container||Pod|
|Container runtime||Using native namespaces or Docker||Docker||Docker, Rkt|
|Scheduling hierarchy||Two level Scheduler. Even Kubernetes and Swarm scheduler can run on top of Mesos scheduler||Single level scheduler||Single level scheduler|
|Docker Integration||Loosely coupled||Tighly coupled||Loosely coupled|
|Production quality||Mesos has been used in production deployments like twitter, Airbnb, eBay, and Netflix for quite some time||Starting to get deployed in production||Starting to get deployed in production|
|Usecase||Heterogenous workload using different frameworks||Homogenous workload||Homogenous workload|
At this point, Kubernetes has the highest momentum but all 3 of the above have significant market share. I think all the 3 Orchestration solutions will stay for different Use-cases.
What are the DC/OS Enterprise components?
Following are the primary components:
- Mesos Master, Slave – Master runs the primary services and Slave nodes run the user workloads.
- Frameworks like marathon, chronos
- Zookeeper for coordination
- Service discovery using Mesos dns, minuteman layer4 load balancer
- Networking across nodes. There is ongoing integration with CNI and plugins like Calico
- CLI and GUI for management
- Multi-tenant support
- Security and compliance including LDAP authentication
- Enterprise support
How to write own Mesos framework?
One reason to write own framework is to give fine grained control over what gets run where and at what time. Its possible that certain set of tasks should be executed together with some constraints and these can be taken care by framework.
Framework implementor needs to implement Scheduler and Executor. Scheduler registers with Mesos Master and takes care of scheduling the Jobs. Executor runs in each slave and takes care of running the jobs in slave. For example, Marathon framework has a Docker executor for running Docker Containers. Following link explains reasons to use Frameworks and following links(1, 2) explains how to write a new Framework.
For most usecases, a generic framework like Marathon would suffice. If there are specific needs, then it is suggested to write a new framework. Since frameworks share a lot of functionality, it is not clear to me on the code reuse between frameworks.