Containerd supports namespaces at the container runtime level. These namespaces are entirely different from the Kubernetes namespaces. Containerd namespaces are used to provide isolation to different applications that might be using containerd like docker, kubelet, etc. Below are two well-known namespaces.
K8s.io : contains all the containers started from the CRI plugin by kubelet, irrespective of the namespace in Kubernetes
moby : comprises all containers started by docker
Since containerd allows different apps to use different namespaces, we must provide k8s.io as a namespace when interacting with containerd directly, to manage containers started as part of PMK stack. All containers started in PMK stack are added to thek8s.io namespace. Even the containers started by PMK directly (i.e., etcd, bouncer and kube-proxy as added to k8s.io namespace) need the apiserver to be able to access the etcd container and similar containers for the same reasons.
Docker | Nerdctl | crictl | ctr |
|---|
docker ps
| /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io ps
| /opt/pf9/pf9-kube/bin/crictl -r unix:///run/containerd/containerd.sock ps
| ctr -n k8s.io containers list
|
docker inspect
Example docker inspect etcd
| /opt/pf9/pf9-kube/bin/nerdctl inspect
Example /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io inspect etcd
for more detail info use -mode=native /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io inspect -mode=native etcd
| /opt/pf9/pf9-kube/bin/crictl -r unix:///run/containerd/containerd.sock inspect <container ID>
| ctr -n k8s.io containers info <container ID>
|
docker exec
| /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io exec -it etcd /bin/sh
| /opt/pf9/pf9-kube/bin/crictl -r unix:///run/containerd/containerd.sock exec -ti <container ID> sh
| get the container ID from list of containers check if the container has a task associated with it (not all containers have a task associated. For such containers nerdctl or crictl might need to be used to exec)
ctr -n k8s.io tasks ls
Exec into container using ID
ctr -n k8s.io tasks exec--exec-id <arbitrary string to associate to this task> <container ID> /bin/sh
|
docker images
| /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io images
| /opt/pf9/pf9-kube/bin/crictl -r unix:///run/containerd/containerd.sock images
| ctr -n k8s.io images list
|
docker network
| /opt/pf9/pf9-kube/bin/nerdctln -n k8s.io network ls
| N/A | N/A |
docker system info
| /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io system info
| N/A | N/A |
docker logs
| /opt/pf9/pf9-kube/bin/nerdctl -n k8s.io logs etcd
/opt/pf9/pf9-kube/bin/crictl -r unix:///run/containerd/containerd.sock logs <container-id>
| /opt/pf9/pf9-kube/bin/crictl -r unix:///run/containerd/containerd.sock logs <container ID>
| N/A |
|
|
|
|
/var/log/containers/
/var/log/pods/
/etc/containerd/config.toml
One of the questions often raised was, “what happens if the cluster is partially migrated, i.e., some nodes are running with containerd runtime while others are running with docker runtime?”
Pods continued to run as usual when the cluster is partially migrated. Moreover, the pods were able to communicate across nodes irrespective of the container runtime on the source and destination nodes where the pods are running.
root@ip-10-0-2-16:~# kubectl --kubeconfig /etc/pf9/kube.d/kubeconfigs/admin.yaml get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-6799fc88d8-2cl9l 1/1 Running 0 3m22s 10.20.123.19 ip-10-0-2-168.us-west-2.compute.internal <none> <none>
nginx-6799fc88d8-4rj7s 1/1 Running 0 3m22s 10.20.215.76 ip-10-0-2-16.us-west-2.compute.internal <none> <none>
nginx-6799fc88d8-g9jx9 1/1 Running 0 3m22s 10.20.132.198 ip-10-0-1-90.us-west-2.compute.internal <none> <none>
nginx-6799fc88d8-q2wvs 1/1 Running 0 3m22s 10.20.132.197 ip-10-0-1-90.us-west-2.compute.internal <none> <none>
nginx-6799fc88d8-vd4cp 1/1 Running 0 3m22s 10.20.123.18 ip-10-0-2-168.us-west-2.compute.internal <none> <none>
root@ip-10-0-2-16:~# kubectl --kubeconfig /etc/pf9/kube.d/kubeconfigs/admin.yaml get no -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-1-90.us-west-2.compute.internal Ready worker 27m v1.21.3 10.0.1.90 34.221.171.199 Ubuntu 20.04.2 LTS 5.4.0-1041-aws docker://19.3.11
ip-10-0-2-16.us-west-2.compute.internal Ready master 30m v1.21.3 10.0.2.16 34.220.60.240 Ubuntu 20.04.2 LTS 5.4.0-1041-aws docker://19.3.11
ip-10-0-2-168.us-west-2.compute.internal Ready worker 27m v1.21.3 10.0.2.168 54.200.68.102 Ubuntu 20.04.2 LTS 5.4.0-1041-aws containerd://1.4.6
## On the worker nodes that is using docker runtime
root@ip-10-0-1-90:~# /opt/pf9/pf9-kube/bin/crictl exec -ti 1a4fb967877cb bash
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]. As the default settings are now deprecated, you should set the endpoint instead.
## Verify east-west connectivity across pods of the nginx deployment
root@nginx-6799fc88d8-q2wvs:/# for i in 10.20.123.19 10.20.215.76 10.20.132.198 10.20.132.197 10.20.123.18; do curl http://$i; done
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
# Verify north-south connectivity
root@nginx-6799fc88d8-q2wvs:/# curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
