Direct Access to the Kubernetes API Using Authentication Credentials
Instead of using kubectl in proxy mode, we can provide the location and credentials directly to the HTTP client. This approach can be used if you are using a client that may get confused by proxies, but it is less secure than using the kubectl proxy due to the risk of MITM attacks. To mitigate this risk, it is recommended that you import the root certificate and verify the identity of the API server when using this method.
When thinking about accessing the cluster using credentials, we need to understand how authentication is configured and what authentication plugins are enabled in our cluster. Several authentication plugins can be used, which allow different ways of authenticating with the server:
- Client certificates
- ServiceAccount bearer tokens
- Authenticating proxy
- HTTP basic auth
Note
Note that the preceding list includes only some of the authentication plugins. You can learn more about authentication at https://kubernetes.io/docs/reference/access-authn-authz/authentication/.
Let's check what authentication plugins are enabled in our cluster by looking at the API server running process using the following command and looking at the flags passed to the API server:
kubectl exec -it kube-apiserver-minikube -n kube-system -- /bin/sh -c "apt update ; apt -y install procps ; ps aux | grep kube-apiserver"
This command will first install/update procps (a tool used to inspect processes) within the API server, which is running as a pod on our Minikube server. Then, it will get the list of processes and filter it by using the kube-apiserver keyword. You will get a long output, but here is the part that we are interested in:
Figure 4.48: Getting the details flags passed to the API server
The following two flags from this screenshot tell us some important information:
- --client-ca-file=/var/lib/minikube/certs/ca.crt
- --service-account-key-file=/var/lib/minikube/certs/sa.pub
These flags tell us that we have two different authentication plugins configured—X.509 client certificates (based on the first flag) and ServiceAccount tokens (based on the second flag). We will now learn how to use both of these authentication methods for communicating with the API server.
Method 1: Using Client Certificate Authentication
X.509 certificates are used for authenticating external requests, which is the current configuration in our kubeconfig file. The --client-ca-file=/var/lib/minikube/certs/ca.crt flag indicates the certificate authority that is used to validate client certificates, which will authenticate with the API server. An X.509 certificate defines a subject, which is what identifies a user in Kubernetes. For example, the X.509 certificate used for SSL by https://www.google.com/ has a subject containing the following information:
Common Name = www.google.com
Organization = Google LLC
Locality = Mountain View
State = California
Country = US
When an X.509 certificate is used for authenticating a Kubernetes user, the Common Name of the subject is used as the username for the user, and the Organization field is used as the group membership of that user.
Kubernetes uses a TLS protocol for all of its API calls as a security measure. The HTTP client that we have been using so far, curl, can work with TLS. Earlier, kubectl proxy took care of communicating over TLS for us, but if we want to do it directly using curl, we need to add three more details to all of our API calls:
- --cert: The client certificate path
- --key: The private key path
- --cacert: The certificate authority path
So, if we combine them, the command syntax should look as follows:
curl --cert <ClientCertificate> --key <PrivateKey> --cacert <CertificateAuthority> https://<APIServerAddress:port>/api
In this section, we will not create these certificates, but instead, we will be using the certificates that were created when we bootstrapped our cluster using Minikube. All the relevant information can be taken from our kubeconfig file, which was prepared by Minikube when we initialized the cluster. Let's see that file:
kubectl config view
You should get the following response:
Figure 4.49: The API server IP and authentication certificates in kubeconfig
The final command should look like the following: you can see that we can explore the API:
curl --cert ~/.minikube/client.crt --key ~/.minikube/client.key --cacert ~/.minikube/ca.crt https://192.168.99.110:8443/api
You should get the following response:
Figure 4.50: Response from API server
So, we can see that the API server is responding to our calls. You can use this method to achieve everything that we have done in the previous section using kubectl proxy.
Method 2: Using a ServiceAccount Bearer Token
Service accounts are meant to authenticate processes running within the cluster, such as pods, to allow internal communication with the API server. They use signed bearer JSON Web Tokens (JWTs) to authenticate with the API server. These tokens are stored in Kubernetes objects called Secrets, which are a type of entities used to store sensitive information, such as the aforementioned authentication tokens. The information stored inside a Secret is Base64-encoded.
So, each ServiceAccount has a corresponding secret associated with it. When a pod uses a ServiceAccount to authenticate with the API server, the secret is mounted on the pod and the bearer token is decoded and then mounted at the following location inside a pod: /run/secrets/kubernetes.io/serviceaccount. This can then be used by any process in the pod to authenticate with the API server. Authentication by use of ServiceAccounts is enabled by a built-in module known as an admission controller, which is enabled by default.
However, ServiceAccounts alone are not sufficient; once authenticated, Kubernetes also needs to permit any actions for that ServiceAccount (which is the authorization phase). This is managed by Role-Based Access Control (RBAC) policies. In Kubernetes, you can define certain Roles, and then use RoleBinding to bind those Roles to certain users or ServiceAccounts.
A Role defines what actions (API verbs) are allowed and which API groups and resources can be accessed. A RoleBinding defines which user or ServiceAccount can assume that Role. A ClusterRole is similar to a Role, except that a Role is namespace-scoped, while a ClusterRole is a cluster-scoped policy. The same distinction is true for RoleBinding and ClusterRoleBinding.
Note
You will learn more about secrets in Chapter 10, ConfigMaps and Secrets; more on RBAC in Chapter 13, Runtime and Network Security in Kubernetes; and admission controllers in Chapter 16, Kubernetes Admission Controllers.
Every namespace contains a ServiceAccount called default. We can see that by using the following command:
kubectl get serviceaccounts --all-namespaces
You should see the following response:
Figure 4.51: Examining default ServiceAccounts for each namespace
As mentioned earlier, a ServiceAccount is associated with a secret that contains the CA certificate of the API server and a bearer token. We can view the ServiceAccount-associated secret in the default namespace, as follows:
kubectl get secrets
You should get the following response:
NAME TYPE DATA AGE
default-token-wtkk5 kubernetes.io/service-account-token 3 10h
We can see that we have a secret named default-token-wtkk5 (where wtkk5 is a random string) in our default namespace. We can view the content of the Secret resource by using the following command:
kubectl get secrets default-token-wtkk5 -o yaml
This command will get the object definition as it is stored in etcd and display it in YAML format so that it is easy to read. This will produce an output as follows:
Figure 4.52: Displaying the information stored in a secret
Note from the preceding secret that namespace, token, and the CA certificate of the API server (ca.crt) are Base64-encoded. You can decode it using base64 --decode in your Linux terminal, as follows:
echo "<copied_value>" | base64 --decode
Copy and paste the value from ca.crt or token in the preceding command. This will output the decoded value, which you can then write to a file or a variable for later use. However, in this demonstration, we will show another method to get the values.
Let's take a peek into one of our pods:
kubectl exec -it <pod-name> -- /bin/bash
This command enters the pod and then runs a Bash shell on it. Then, once we have the shell running inside a pod, we can explore the various mount points available in the pod:
df -h
This will give an output similar to the following:
Figure 4.53: The mount point for the bearer token
The mount point can be explored further:
ls /var/run/secrets/kubernetes.io/serviceaccount
You should see an output similar to the following:
ca.crt namespace token
As you can see here, the mount point contains the API server CA certificate, the namespace this secret belongs to, and the JWT bearer token. If you are trying these commands on your terminal, you can exit the pod's shell by entering an exit.
If we try to access the API server using curl from inside the pod, we would need to provide the CA path and the token. Let's try to list all the pods in the pod's namespace by accessing the API server from inside a pod.
We can create a new Deployment and start a Bash terminal with the following procedure:
kubectl run my-bash --rm --restart=Never -it --image=ubuntu -- bash
This may take a few seconds to start up, and then you will get a response similar to this:
If you don't see a command prompt, try pressing enter.
root@my-bash: /#
This will start up a Deployment running Ubuntu and immediately take us inside the pod and open up the Bash shell. The --rm flag in this command will delete the pod after all the processes inside the pod are terminated—that is, after we leave the pod using the exit command. But for now, let's install curl:
apt update && apt -y install curl
This should produce a response similar to this:
Figure 4.54: Installing curl
Now that we have installed curl, let's try to list the pods using curl by accessing the API path:
curl https://kubernetes/api/v1/namespaces/$NAMESPACE/pods
You should see the following response:
Figure 4.55: Trying to access the API without TLS
Notice that the command has failed. This happened since Kubernetes forces all communication to use TLS, which usually rejects insecure connections (without any authentication tokens). Let's add the --insecure flag, which will allow an insecure connection with curl, and observe the results:
curl --insecure https://kubernetes/api/v1/namespaces/$NAMESPACE/pods
You should get a response as follows:
Figure 4.56: Anonymous request to the API server
We can see that we were able to reach the server using an insecure connection. However, the API server treated our request as anonymous since there was no identity provided to our command.
Now, to make commands easier, we can add the namespace, CA certificate (ca.crt), and the token to variables so that the API server knows the identity of the service account generating the API request:
CACERT=/run/secrets/kubernetes.io/serviceaccount/ca.crt
TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)
NAMESPACE=$(cat /run/secrets/kubernetes.io/serviceaccount/namespace)
Note that here we can use the values directly as they are in plaintext (not encoded) when looking from inside a pod, compared to having to decode them from a Secret. Now, we have all the parameters ready. When using bearer token authentication, the client should send this token in the header of the request, which is the authorization header. This should look like this: Authorization: Bearer <token>. Since we have added the token into a variable, we can simply use that. Let's run the curl command to see whether we can list the pods using the identity of the ServiceAccount:
curl --cacert $CACERT -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/$NAMESPACE/pods
You should get the following response:
Figure 4.57: Request to the API server using the default ServiceAccount
Notice that we were able to reach the API server, and the API server verified the "system:serviceaccount:default:default" identity, which is represented in this format: system:<resource_type>:<namespace>:<resource_name> However, we still got a Forbidden error because ServiceAccounts do not have any permissions by default. We need to manually assign permissions to our default ServiceAccount in order to be able to list pods. This can be done by creating a RoleBinding and linking it to the view ClusterRole.
Open another terminal window, ensuring that you don't close the terminal session running the my-bash pod (because the pod will be deleted and you will lose your progress if you close it). Now, in the second terminal session, you can run the following command to create a rolebinding defaultSA-view to attach the view ClusterRole to the ServiceAccount:
kubectl create rolebinding defaultSA-view \
--clusterrole=view \
--serviceaccount=default:default \
--namespace=default
Note
The view ClusterRole should already exist for your Kubernetes cluster, as it is one of the default ClusterRoles available for use.
As you might recall from the previous chapter, this is an imperative approach to creating resources; you will learn how to create manifests for RBAC policies in Chapter 13, Runtime and Network Security in Kubernetes. Note that we have to specify the ServiceAccount as <namespace>:<ServiceAccountName>, and we have a --namespace flag since a RoleBinding can only apply to the ServiceAccounts within that namespace. You should get the following response:
rolebinding.rbac.authorization.k8s.io/defaultSA-view created
Now, go back to the terminal window where we accessed the my-bash pod. With the necessary permissions set, let's try our curl command again:
curl --cacert $CACERT -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/$NAMESPACE/pods
You should get the following response:
Figure 4.58: Successful response from the API server
Our ServiceAccount can now authenticate with the API server, and it is authorized to list pods in the default namespace.
It is also valid to use ServiceAccount bearer tokens outside the cluster. You may want to use tokens instead of certificates as an identity for long-standing jobs since the token does not expire as long as the ServiceAccount exists, whereas a certificate has an expiry date set by the certificate-issuing authority. An example of this is CI/CD pipelines, where external services commonly use ServiceAccount bearer tokens for authentication.
Activity 4.01: Creating a Deployment Using a ServiceAccount Identity
In this activity, we will bring together all that we have learned in this chapter. We will be using various operations on our cluster and using different methods to access the API server.
Perform the following operations using kubectl:
- Create a new namespace called activity-example.
- Create a new ServiceAccount called activity-sa.
- Create a new RoleBinding called activity-sa-clusteradmin to attach the activity-sa ServiceAccount to the cluster-admin ClusterRole (which exists by default). This step is to ensure that our ServiceAccount has the necessary permissions to interact with the API server as a cluster admin.
Perform the following operations using curl with bearer tokens for authentication:
- Create a new NGINX Deployment with the identity of the activity-sa ServiceAccount.
- List the pods in your Deployment. Once you use curl to check the Deployment, if you have successfully gone through the previous steps, you should get a response that looks something like this:
Figure 4.59: Expected response when checking the Deployment
- Finally, delete the namespace with all associated resources. When using curl to delete a namespace, you should see a response with phase set to terminating for the status field of the namespace resource, as in the following screenshot:
"status": {
"phase": "Terminating"
Note
The solution to this activity can be found at the following address: https://packt.live/304PEoD.