How To Build Kubernetes Platform
Architecture
Architecture Diagram
Non-Prod Environment
Prod Environment
Cluster Networking
Kubernetes supports for third-party netwroking the cluster via CNI plugin. for more infromation, please see Cluster Networking.According to this chinese blog, Principles and Solutions of Kubernetes Networking from Yourongyun the three top of CNI providers for kubernetes cluster based on VMs are Project Calico, Flannel, Weave Net. And also, here is comparison of variable networking solutions, https://github.com/xelatex/homepage/blob/master/source/_posts/Battlefield-Calico-Flannel-Weave-and-Docker-Overlay-Network.mdIf just only considering performance, Project Calico should be perferred.
Container Repository
Two options as follows,Option 1, Private Repository, such as Harbor. Option 2, Repositories from Cloud Providers, such as AWS ECR.
Setup
We can refer to Picking the Right Solution in Kubernetes offical document to select a setup solution. Considering our goal of building a kuberletes platform based on VMs in our on-premises data center.
Minikube For Dev Env
Minikube can be engaed to create a kubernets development in a local local single machine. For more information about Minikube, please read Running Kubernetes Locally via Minikube.
Kubeadm For Non-Prod Env
Using Kubeadm, we can build a kubernetes cluster for non-prod environment, which runs master key components as containers.For more Kubeadm information, please see Using kubeadm to Create a Cluster.
Other IaaC For Prod Env
For creating a kubernetes cluster on VMs for Prod Env, We need to a infrastructure automation tool, such as Ansible, to have this done. Here is a reference of Creating Kubernets Cluster via Ansible
Devops
CI/CD
Source Code Management
For small and agile web projects, such SaaS applications, considering adopting GitHub Flow. For more information, please refer to GitHub Flow.For desk or client applications, such PC desk application, ISO/Android App, or being different time windows for delivery and release of applications, considering GitLab Flow. For more information please refer to Gitlab Flow.
PipeLine
Monitoring
Dashboard
Using Kubernetes Dashboard as Web-based UI for Kubernets clusters to manage the cluster itself along with its attendant resources.For more information, please see Web-UI(Dashboard).
Metrics
Here is official suggested solution as follows. For more informaton, please see Tools for Monitoring Compute, Storage, and Network ResourcesGrafana + Heapster / Prometheus + cAdvisor + InfluxDBHeapster as a metircs aggregator and processorInfluxDB time series database for storageGrafana as a dashboarding and alerting solutioncAdvisor has been built in Kubelet, which collects host metrics like CPU, disk space, and memory utilization, in addition to container metrics.And also, here is a practical example, How to Utilize the “Heapster + InfluxDB + Grafana” Stack in Kubernetes for Monitoring Pods.
Logging
APM
Security
Pod Service Account
For more information, please see the User Guide to Service Accounts.
Pod Security Policy
For more information, please see Pod Security Policies
Auth
Authentication
For more information, please see AuthenticatingSupport SSO integration, such as SAML,AD OpenID, Auth2?
Authenrization
For more information, please see https://kubernetes.io/docs/admin/authorization/ABAC/RBAC
Multi-Tenancy
HypernetesFor more information, please see Hypernetes: Bringing Security and Multi-tenancy to Kubernetes
Application Architecture
Microservice
Here is a chinese blog about how to select open source tools for building a micorservice.https://mp.weixin.qq.com/s/bsuveX-E6E2fKZ24mj03nQ
Servcie Mesh
Training
Knowledge
Kubernets
DocumentKubernetes Handbook (Chinese Version)ETCDFor more information, please see ETCD Document.
Containter
Docker
For moe information, please see https://docs.docker.com/.CRI-OFor more information, please see http://cri-o.io/.
OCI
OCI is a container specification named Open Container Initiative, consisting of OCI Runtime Specification and OCI Image Format