A demo to show how to use Workload Identity to call Google Cloud API. In this demo we will call the Translate API from a GKE application(pod) using Workload Identity.

• Google Cloud Account
• With a Service Account with roles:
  • Kubernetes Engine Admin - to create GKE cluster
  • Service Account roles used to create/update/delete Service Account
    • iam.serviceAccounts.actAs
    • iam.serviceAccounts.get
    • iam.serviceAccounts.create
    • iam.serviceAccounts.delete
    • iam.serviceAccounts.update
    • iam.serviceAccounts.get
    • iam.serviceAccounts.getIamPolicy
    • iam.serviceAccounts.setIamPolicy Or simply you can add Service Account Admin and Service Account User roles
  • Compute Network Admin - to create the VPC networks
• Google Cloud SDK
• terraform
• kubectl
• Taskfile

• kustomize(Optional)
• direnv(Optional)

Clone the sources,

git clone https://github.com/kameshsampath/workload-identity-gke-demo.git && cd "$(basename "$_" .git)"
export DEMO_HOME="$PWD"

When working with Google Cloud the following environment variables helps in setting the right Google Cloud context like Service Account Key file, project etc., You can use direnv or set the following variables on your shell,

export GOOGLE_APPLICATION_CREDENTIALS="the google cloud service account key json file to use"
export CLOUDSDK_ACTIVE_CONFIG_NAME="the google cloud cli profile to use"
export GOOGLE_CLOUD_PROJECT="the google cloud project to use"
export KUBECONFIG="$DEMO_HOME/.kube"

(e.g.)

export CLOUDSDK_ACTIVE_CONFIG_NAME=personal
export GOOGLE_APPLICATION_CREDENTIALS=~/.ssh/my-sa-key.json
export GOOGLE_CLOUD_PROJECT=my-awesome-project
export KUBECONFIG="$DEMO_HOME/.kube"

TIP If you are using direnv you can then create file .envrc.local and add the environment variables. They can then be loaded using direnv allow .

You can find more information about gcloud cli configurations at https://cloud.google.com/sdk/docs/configurations.

As you may need to override few terraform variables that you don't want to check in to VCS, add them to a file called .local.tfvars and set the following environment variable to be picked up by terraform runs,

export TFVARS_FILE=.local.tfvars

NOTE: All .local.tfvars file are git ignored by this template.

Check the Inputs section for all possible variables that are configurable.

An example .local.tfvars that will use a Google Cloud project my-awesome-project, create a two node GKE cluster named wi-demo in region asia-south1 with Kubernetes version 1.24. from stable release channel. The machine type of each cluster node will be e2-standard-4. The demo will be deployed in Kubernetes namespace demo-apps, will use lingua-greeter as the Kubernetes Service Account.

app_ksa            = "lingua-greeter"
app_namespace      = "demo-apps"
cluster_name       = "wi-demo"
configure_app_workload_identity = false
gke_num_nodes      = 2
kubernetes_version = "1.24."
machine_type       = "e2-standard-4"
project_id         = "my-awesome-project"
region             = "asia-south1"
release_channel    = "stable"

NOTE: For rest of the section we assume that your tfvars file is called .local.tfvars

As part of the demo, let us deploy a Kubernetes application called lingua-greeter. The application exposes a REST API /:lang , that allows you to translate a text Hello World! into the language :lang using Google Translate client.

NOTE: The :lang is the BCP 47 language code, https://en.wikipedia.org/wiki/IETF_language_tag.

The deployment is done using task. You can list all available tasks using the command,

task --list

We will use terraform to create all the Google Cloud resources like GKE, Service Account, Kubernetes manifests etc.,

task init

The terraform apply will create the following Google Cloud resources,

• A Kubernetes cluster on GKE with Workload Identity enabled
• A Google Cloud VPC that will be used with GKE

task create_cluster

To see Workload Identity in action we will deploy the application in two parts,

• Application(workload) not enabled for Workload Identity
• Application(workload) enabled for Workload Identity

Create the namespace demo-apps to deploy the lingua-greeter application,

kubectl create ns demo-apps

Run the following command to deploy the application,

kubectl apply -n demo-apps -k $DEMO_HOME/app/config

Wait for application to be ready,

kubectl rollout status -n demo-apps deployment/lingua-greeter --timeout=60s

Get the application service LoadBalancer IP,

kubectl get svc -n demo-apps lingua-greeter

TIP: You can also deploy the application using the command task deploy_app

If the EXTERNAL-IP is <pending> then wait for the IP to be assigned. It will take few minutes for the EXTERNAL-IP to be assigned. You can use the following command to wait until External-IP is assigned,

  while [ -z $(kubectl get svc -n demo-apps lingua-greeter -ojsonpath="{.status.loadBalancer.ingress[*].ip}") ]; do sleep .3; done;

export SERVICE_IP=$(kubectl get svc -n demo-apps lingua-greeter -ojsonpath="{.status.loadBalancer.ingress[*].ip}")

Call the service to return the translation of Hello World! in Tamil(ta),

curl "http://$SERVICE_IP/ta"

The service should fail with a message,

{"message":"Internal Server Error"}

When you check the logs of the lingua-greeter pod, you should see a message like,

time="2023-01-25T10:26:50Z" level=error msg="googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\nMore details:\nReason: authError, Message: Invalid Credentials\n"

As it describes you don't have authentication credentials to call the API. All Google Cloud API requires GOOGLE_APPLICATION_CREDENTIALS to allow client to authenticate itself before calling the API. If you check the deployment manifest we dont have one configured.

• Create a Service Account(SA) that has permissions to call Google Translation API, in our demo we call that SA as translator
• Add translator SA with role roles/cloudtranslate.user
• Add an IAM binding policy to translator SA, with the role roles/iam.workloadIdentityUser and a member "serviceAccount:$GOOGLE_CLOUD_PROJECT.svc.id.goog[demo-apps/lingua-greeter]" (default workload identity SA)

Edit your .local.tfvars file and update the app_use_workload_identity to be true.Save the .local.tfvars and run the following command to create the SA, role and IAM policy binding resources,

task use_workload_identity

The command ran earlier should also generate an updated lingua-greeter Kubernetes Service Account manifest $DEMO_HOME/k8s/sa.yaml, that is annotated to impersonate the translator Google SA,

apiVersion: v1
kind: ServiceAccount
metadata:
  name: lingua-greeter
  namespace: demo-apps
  annotations:
    iam.gke.io/gcp-service-account: "translator@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com"

Run the following command to update the Kubernetes SA lingua-greeter to use the Google IAM service Account using Workload Identity mechanics,

kubectl apply -n demo-apps -f "$DEMO_HOME/k8s/sa.yaml"

Call the service again, the service should succeed with a response,

{"text":"Hello World!","translation":"வணக்கம் உலகம்!","translationLanguage":"ta"}

NOTE: Sometimes it may take few seconds for the pods to refresh the metadata, in such cases try to call the service after few seconds.

For more information check out Workload Identity.

Apache License