kops treats a cluster as a declarative object stored in S3 — you edit the spec, then tell it to make reality match, which is exactly the Kubernetes model applied to the cluster itself.
kops (Kubernetes Operations) creates and manages self-managed Kubernetes clusters on AWS: it provisions the VMs, runs the control plane on machines you own, and manages upgrades.
YOU STATE STORE (S3) AWS
─── ──────────────── ───
kops create → cluster spec → ASGs, EC2, ELB,
kops edit → (YAML, versioned) Route53, EBS, IAM,
kops update → security groups
│
kops validate ←───────────┴─────────────→ control plane runs on
desired vs actual YOUR instances
# kops — check the current release rather than copying a version from a blog
curl -Lo kops https://github.com/kubernetes/kops/releases/latest/download/kops-linux-amd64
chmod +x kops && sudo mv kops /usr/local/bin/
kops version
aws --version
# What WOULD be created — read this, it is a long list
kops update cluster --name ${NAME} --state=${KOPS_STATE_STORE}
# Actually build it, and write a kubeconfig entry
kops update cluster --name ${NAME} --state=${KOPS_STATE_STORE} --yes --admin
kops now creates, roughly in order:
IAM roles and instance profiles
↓
VPC · subnets (public + private) · NAT gateways · route tables
↓
Security groups
↓
Route53 records: api.k8s.example.com
↓
Launch templates + Auto Scaling Groups
↓ ASGs boot instances; each runs nodeup on first boot
Control-plane instances: etcd, apiserver, scheduler, controller-manager
↓
Worker instances: kubelet joins via the API endpoint from DNS
↓
Addons: CNI, CoreDNS, and whatever the spec declares
Everything is an edit to the spec followed by an update. Same loop, always:
kops edit cluster ──→ changes the spec in S3
kops edit ig nodes ──→ changes an instance group
│
↓
kops update cluster --yes ──→ changes AWS resources
│
↓
kops rolling-update cluster --yes ──→ replaces instances, one at a time
(only if instances must change)
# Scale the workers: edit minSize/maxSize
kops edit ig --name ${NAME} nodes
kops update cluster --name ${NAME} --yes
# Scaling an ASG needs no rolling update; changing the AMI or instance type does
Why does a control plane need an odd number of nodes?
Because etcd uses Raft, and Raft requires a majority (a quorum) to commit a write. With N members the quorum is floor(N/2) + 1.
Members
Quorum
Failures tolerated
1
1
0
2
2
0
3
2
1
4
3
1
5
3
2
Two key observations:
Two is strictly worse than one. Same zero tolerance, twice the hardware that can fail — so the probability of an outage doubles for no benefit.
Four is no better than three. Both tolerate one failure, but four costs more and makes every write wait on a larger quorum.
What happens when quorum is lost: etcd goes read-only. The API server can still serve reads, so kubectl get works — but no writes succeed, so no deploys, no scaling, no rescheduling. Running pods keep running, which makes the failure quiet and easy to misdiagnose.
kops or EKS? · نستخدم kops ولا EKS؟
Ask what constraint forces the choice, then answer.
kops
EKS
Control plane
Yours — your instances, your problem
AWS runs it, per-hour fee
Upgrades
You run them, one minor at a time
AWS-assisted, still your responsibility for nodes
etcd backups
Yours to verify
AWS manages
Customisation
Full — any flag, any component
Limited to what EKS exposes
Time to first cluster
15 minutes plus DNS setup
Minutes, or one Terraform module
Ongoing cost
EC2 only
EC2 plus the control-plane fee
Who to call at 3am
You
AWS support
Choose EKS by default. The control-plane fee is far less than the salary cost of operating etcd yourself, and "we run our own control plane" is undifferentiated work.
Choose kops when you need control EKS does not expose, you are not on AWS in the same account model, or you have a regulatory reason to own every component.
You lose the S3 state store. What can you still do?
Almost nothing, from kops' point of view. The cluster keeps running — the control plane and kubelets do not consult S3 — but you have lost the ability to manage it: no kops update, no kops rolling-update, no upgrades, no scaling through kops.
You also lose the cluster CA material kops keeps there, and the etcd backups it writes there.
Recovery options, worst to best:
S3 versioning enabled → restore the deleted objects. This is why versioning is step one.
Reconstruct the spec by hand from the running cluster and AWS resources — possible, tedious, and error-prone.
Build a new cluster and migrate workloads.
The lesson being tested is that the state store is a critical, stateful dependency of your tooling, exactly like Terraform state — and that it needs versioning, encryption, restricted IAM and a tested restore, decided before the first kops create.