Creating an Internal Load Balancer in AWS EKS using Istio

This article discusses the need and steps to create an internal load balancer in AWS for an EKS cluster using Istio

Terminology

EKS is the Managed Kubernetes Service available on AWS and applications running inside the cluster are usually accessed from outside the cluster via an Ingress Gateway which sits at the edge of the cluster. This gateway uses a load balancer which can be a Classic (CLB), Application (ALB), or Network (NLB) load balancer provided by AWS.

Istio is an open-source independent service mesh that provides the fundamentals you need to successfully run a distributed microservice architecture. The default installation of Istio creates this ingress gateway automatically allowing you to access services from outside the cluster as shown below.

Need

There may be requirements to expose applications running inside the k8s cluster internally only i.e. not exposed via the internet and/or accessible only to certain security groups. For example, if there are other backends running in a separate k8s cluster they may need to access microservices in your cluster and you may not want to expose those microservices to anything apart from that backend. In order to do so, you need to create another ingress gateway exposing an internal load balancer with attached security groups.

At the time of writing, Istio has no option to specify the number of ingress gateways you need (external or internal). You have to generate the manifest files using Istio’s helm chart using helm or istioctl in order to create more than one ingress gateway.

Steps

Step 1: Generate yaml files to be used for creating the resources needed.

Resources that are needed for the ingress gateway:

• HorizontalPodAutoscaler
• Deployment
• Gateway
• PodDisruptionBudget
• Service
• ServiceAccount

These can be obtained using istioctl or helm based on already existing ingress gateway that comes with Istio. At a high level, we need to make a copy of the above resources, update the names, and apply the yaml file created.

istioctl

istioctl manifest generate > manifest.yaml

If you search for Resources for IngressGateway component in the manifest.yaml you will get to the section that contains the YAML and resources related to the ingress gateway.

helm

helm template install/kubernetes/helm/istio > manifest.yaml

You need to find the 6 resources mentioned above that are generated for default ingress gateway.

We will be deploying the second instance of the ingress gateway into the same namespace (istio-system), and therefore we will need to:

• Change the name of the second ingress gateway — istio-internal-ingressgateway sounds clear enough.
• Correctly label the pods for the istio-internal-ingressgateway and update other resources accordingly (e.g., the Kubernetes service)

In each resource, we have to change at least the resource name and labels — we will simply be changing istio-ingressgateway to istio-internal-ingressgateway and ingressgateway to internal-ingressgateway value. Note that some resources might also refer to other resources — e.g., HorizontalPodAutoscaler references the deployment, so you have to change the name there as well.

Step 2: Add annotations for the internal load balancer

Different settings can be applied to a load balancer service in AWS using annotations. You can find the complete list here. The ones we need are:*

service.beta.kubernetes.io/aws-load-balancer-internal: Used on the service to indicate that we want an internal ELB.

service.beta.kubernetes.io/aws-load-balancer-security-groups: Used to specify the security groups to be added to ELB created. This replaces all other security groups previously assigned to the ELB.

Service yaml file should look like as below:

Step 3: Deploy the resources

After copying the needed configuration for resources into a yaml file like internal-gateway.yaml we need to apply it

kubectl apply -f internal-gateway.yaml

Step 4 (Optional) : Add entry in Route 53 for the generated load balancer

You can obtain the load balancer URL that gets created from the previous step and add an alias name for it in Route 53.

kubectl get svc istio-internal-ingressgateway -n istio-system

Step 5: Create Gateway and Virtual service to allow routing to services

You can create a gateway and virtual service to route traffic to the appropriate service.

Note

Correct tags should be added to your Amazon VPC subnets so that only private subnets are attached to the load balancer created.

1. Open the AWS VPC console.

2. On the navigation pane, choose Subnets.

3. Choose the Tags tab for each subnet, and then confirm that a tag exists. See the following example:

Key: kubernetes.io/cluster/yourEKSClusterName
Value: shared

4. For your public subnets, confirm that the following tag exists:

Key: kubernetes.io/role/elb
Value: 1

Note: To see if a subnet is a public subnet, check the route table associated with the subnet. A public subnet has a route to an internet gateway (igw-xxxxxxxxxxx). A private subnet has a route to the internet through a NAT gateway or NAT instance, or no route to the internet at all.

Important: You must have the tag in step 4 to create an internet-facing load balancer service.

5. For your private subnets, confirm that the following tag exists:

Key: kubernetes.io/role/internal-elb
Value: 1

Important: You must have the tag in step 5 to create an internal-facing load balancer service.