In this blog, I will talk about different options for getting traffic from external world into GKE cluster. The options described are:
- Network load balancer(NLB)
- Http load balancer with ingress
- Http load balancer with Network endpoint groups(NEG)
- nginx Ingress controller
- Istio ingress gateway
For each of the above options, I will deploy a simple helloworld service with 2 versions and show access from outside world to the 2 versions of the application. The code is available in my github project here. You can clone and play with it if needed. In the end, I will also discuss about choosing the right option based on the requirement.
Pre-requisites
For all the examples below, we need to create a GKE cluster with some basic configurations. Following command sets up a 3 node GKE cluster with VPC native mode:
gcloud container clusters create demo-cluster --num-nodes=3 --zone=us-central1-b --enable-ip-aliasFollowing command installs Istio add-on to the cluster. This is needed for Istio ingress gateway option. Istio can also be installed as a helm package.
gcloud beta container clusters update demo-cluster --project sreemakam-demo --zone us-central1-b\
--update-addons=Istio=ENABLED --istio-config=auth=MTLS_PERMISSIVEFollowing command installs nginx controller using Helm. Before doing this step, it is needed to install Helm client and server part.
Install helm:
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller
Install nginx controller:
helm install --name nginx-ingress stable/nginx-ingress --set rbac.create=true --set controller.publishService.enabled=trueNetwork load balancer(NLB)
NLB works at L4 level. To expose multiple versions, we need to deploy multiple NLBs. Each NLB will expose a public IP address.
Following command deploys the application:
kubectl apply -f nlbFollowing command shows the 2 NLBs that are created:
gcloud compute forwarding-rules list
NAME REGION IP_ADDRESS IP_PROTOCOL TARGET
a209565130eda11ea90fe42010a80019 us-central1 104.154.216.240 TCP us-central1/targetPools/a209565130eda11ea90fe42010a80019
a20ec964c0eda11ea90fe42010a80019 us-central1 34.69.21.178 TCP us-central1/targetPools/a20ec964c0eda11ea90fe42010a80019Following outputs shows the services, deployments and pods:
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello LoadBalancer 10.113.1.198 34.69.21.178 8080:31482/TCP 59m
hello2 LoadBalancer 10.113.3.7 104.154.216.240 8080:31550/TCP 59m
kubernetes ClusterIP 10.113.0.1 <none> 443/TCP 8d
kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello 2/2 2 2 59m
hello2 2/2 2 2 59m
kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-bc787595d-4k27t 1/1 Running 0 59m
hello-bc787595d-smvdl 1/1 Running 0 94m
hello2-7494666cc7-sk2sv 1/1 Running 0 59m
hello2-7494666cc7-kl2pq 1/1 Running 0 59mTo access the hello v1 of the service, we need to do:
curl 34.69.21.178:8080
To access the hello v1 of the service, we need to do:
curl 104.154.216.240:8080
Http load balancer with ingress
Http load balancer works at L7. The ingress object in Kubernetes creates Global load balancer in GCP.
Following command deploys the application:
kubectl apply -f ingress
Following command shows the forwarding rules for HTTP load balancer:
$ gcloud compute forwarding-rules list
NAME REGION IP_ADDRESS IP_PROTOCOL TARGET
k8s-fw-default-fanout-ingress--fcdc22763fbe5b3a 34.102.181.115 TCP k8s-tp-default-fanout-ingress--fcdc22763fbe5b3aFollowing outputs shows the services, deployments, pods and ingress:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello NodePort 10.113.1.0 <none> 8080:31555/TCP 11h
hello2 NodePort 10.113.13.246 <none> 8080:30233/TCP 11h
kubernetes ClusterIP 10.113.0.1 <none> 443/TCP 8d
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello 2/2 2 2 11h
hello2 2/2 2 2 11h
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-bc787595d-qjrmr 1/1 Running 0 11h
hello-bc787595d-ra3ar 1/1 Running 0 11h
hello2-7494666cc7-5h588 1/1 Running 0 11h
hello2-7494666cc7-faeaw 1/1 Running 0 11h
$ kubectl get ingress
curl NAME HOSTS ADDRESS PORTS AGE
fanout-ingress * 34.102.181.115 80 20mTo access the hello v1 of the service, we need to do:
curl 34.102.181.115/v1To access the hello v2 of the service, we need to do:
curl 34.102.181.115/v2
HTTP Load balancer with NEG
Network endpoint groups are groups of Network endpoints which can be tied to Load balancer as backends. NEGs are useful for Container native load balancing where each Container can be represented as endpoint to the load balancer. With Container native load balancing, the advantages are better load distribution, removal of extra hop that reduces latency and better health check. Without Container native load balancing, load balancer would distribute packets to each node and iptables in each node would further distribute the packets to each pod/container and this adds the extra hop.
Following command deploys the application with HTTP load balancer and NEG:
kubectl apply -f ingress-negThe only additional configuration needed to setup Ingress with NEG is to add the following annotation to the ingress service manifest file:
annotations:
cloud.google.com/neg: '{"ingress": true}'Following command shows the forwarding rules for HTTP load balancer:
$ gcloud compute forwarding-rules list
NAME REGION IP_ADDRESS IP_PROTOCOL TARGET
k8s-fw-default-fanout-ingress--fcdc22763fbe5b3a 34.102.181.115 TCP k8s-tp-default-fanout-ingress--fcdc22763fbe5b3aFollowing outputs shows the services, deployments, pods and ingress:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello NodePort 10.113.10.187 <none> 8080:32046/TCP 2m38s
hello2 NodePort 10.113.4.7 <none> 8080:32420/TCP 2m38s
kubernetes ClusterIP 10.113.0.1 <none> 443/TCP 8d
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello 2/2 2 2 2m45s
hello2 2/2 2 2 2m46s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-bc787595d-64k8f 1/1 Running 0 2m50s
hello-bc787595d-fsw31 1/1 Running 0 2m50s
hello2-7494666cc7-ldvz8 1/1 Running 0 2m51s
hello2-7494666cc7-s314f 1/1 Running 0 2m51s
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
fanout-ingress * 34.102.181.115 80 19mFollowing outputs shows the network endpoint groups and the endpoints associated with the network endpoint groups. There are 2 network endpoint groups, 1 for hello v1 service and another for hello v2 service. Each NEG has 2 pods that are part of the service.
gcloud compute network-endpoint-groups list
NAME LOCATION ENDPOINT_TYPE SIZE
k8s1-fcdc2276-default-hello-8080-ae85b431 us-central1-b GCE_VM_IP_PORT 2
k8s1-fcdc2276-default-hello2-8080-6ef9b812 us-central1-b GCE_VM_IP_PORT 2
$ gcloud compute network-endpoint-groups list-network-endpoints --zone us-central1-b k8s1-fcdc2276-default-hello-8080-ae85b431
INSTANCE IP_ADDRESS PORT
gke-demo-cluster-default-pool-cf9e9717-vx2r 10.48.0.45 8080
gke-demo-cluster-default-pool-cf9e9717-bdfk 10.48.1.29 8080
$ gcloud compute network-endpoint-groups list-network-endpoints --zone us-central1-b k8s1-fcdc2276-default-hello2-8080-6ef9b812
INSTANCE IP_ADDRESS PORT
gke-demo-cluster-default-pool-cf9e9717-vx2r 10.48.0.44 8080
gke-demo-cluster-default-pool-cf9e9717-g02w 10.48.2.30 8080To access hello version v1, we need to do:
curl 34.102.181.115/v1
To access hello version v2, we need to do:
curl 34.102.181.115/v2Istio Ingress gateway
Istio provides service mesh functionality. In addition to providing a mesh between services, Istio also provides Ingress gateway functionality that takes care of traffic control and routing features for traffic entering the mesh from outside world.
Istio injects an Envoy proxy container into each pod which takes care of traffic management and routing without the individual applications being aware of it. The injection can either be done automatically or manually. Following command setups up automatic proxy injection:
kubectl label namespace default istio-injection=enabledI have used 2 approaches to deploy the service using Istio.
Istio Approach 1:
In this approach, I have used Virtual service to demux the 2 versions of helloworld to separate services. This is not an optimal approach as we cannot use the advantages of Istio service mesh. I have put this approach just for reference.
In the pre-requisites section, I have mentioned the steps to add Istio to the GKE cluster. Following command deploys the application with Istio ingress gateway.
kubectl apply -f istioIstio ingress gateway sets up GCP NLB. NLB provides connectivity from external world and Istio ingress gateway takes care of the http routing rules. Following command shows the Istio deployments in istio-system namespace:
$ kubectl get deployments --namespace istio-system
NAME READY UP-TO-DATE AVAILABLE AGE
istio-citadel 1/1 1 1 8d
istio-galley 1/1 1 1 8d
istio-ingressgateway 1/1 1 1 8d
istio-pilot 1/1 1 1 8d
istio-policy 1/1 1 1 8d
istio-sidecar-injector 1/1 1 1 8d
istio-telemetry 1/1 1 1 8d
promsd 1/1 1 1 8dFollowing command shows the NLB created by Istio ingress gateway:
$ gcloud compute forwarding-rules list
NAME REGION IP_ADDRESS IP_PROTOCOL TARGET
af95f4a780fa711ea9ae142010a80011 us-central1 35.232.71.209 TCP us-central1/targetPools/af95f4a780fa711ea9ae142010a80011Following outputs shows the services, deployments, pods and ingress:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello NodePort 10.113.3.95 <none> 8080:32133/TCP 5h58m
hello2 NodePort 10.113.2.177 <none> 8080:31628/TCP 5h58m
kubernetes ClusterIP 10.113.0.1 <none> 443/TCP 9d
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello 2/2 2 2 5h50m
hello2 2/2 2 2 5h50m
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-bc787595d-h7h6j 2/2 Running 0 5h50m
hello-bc787595d-sq7z4 2/2 Running 0 5h50m
hello2-7494666cc7-t7bnq 2/2 Running 0 5h50m
hello2-7494666cc7-tx9q2 2/2 Running 0 5h50mIn the above command, there are 2 containers in each pod. The first container is the application container and the second one is the proxy container.
To access the application, do the following:
export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
curl http://$GATEWAY_URL/v1
curl http://$GATEWAY_URL/v2In my case, I have specified “Hosts” field as “*” in the gateway and virtualservice, so I did not specify the host when doing the curl. If you have specified a hostname, use the curl -H option to access the service.
Istio Approach 2
In this approach, the 2 different versions of hello service is represented as subsets in Virtualservice manifest and using destinationrule, we can redirect to the right pod in the kubernetes cluster.
Virtualservice has 2 subsets, 1 for version v1 and another for version v2. All traffic management configuration happens through Virtualservice. The “host” field in virtual service maps to the kubernetes service name. Virtual service directs traffic to the right destination and the destination rule maps the subset to the appropriate labels and also sets up load balancing logic.
Following command sets up the application with virtual service pointing to the following mapping(/v1->hello version v1, v2->hello version v2, default-hello version v2):
kubectl apply -f istio1Following command shows the rules in Virtual service:
Http:
Match:
Uri:
Prefix: /v1
Route:
Destination:
Host: hello
Subset: v1
Match:
Uri:
Prefix: /v2
Route:
Destination:
Host: hello
Subset: v2
Route:
Destination:
Host: hello
Subset: v2Now, lets apply another virtual service that maps all traffic to version v1. This will map all traffic to hello version v1.
kubectl apply -f istio1/traffic-mgmt/istio-service-v1.yaml
Following command shows the rules in Virtual service:
Http:
Route:
Destination:
Host: hello
Subset: v1Now, lets apply another virtual service that maps 20% of traffic to hello version v1 and 80% of traffic to hello version v2.
kubectl apply -f istio1/traffic-mgmt/istio-service-v1-20-v2-80.yaml
Following command shows the rules in Virtual service for the above configuration:
Http:
Route:
Destination:
Host: hello
Subset: v1
Weight: 20
Destination:
Host: hello
Subset: v2
Weight: 80Istio with GCP http load balancer
By default, Istio ingress gateway does not allow usage of GCP HTTP load balancer. There could be scenarios where its an advantage to use GCP HTTP load balancer like needing to use GCP managed certificates, integrate GCP load balancer with cloud armor, CDN etc. For these types of cases, it makes more sense to use GCP global load balancer and point it to Istio ingress gateway. In this case, GCP global load balancer would do the https termination while service level redirection will be done by Istio ingress gateway. The flow would look something like this:
I referred this example to get it working, but I was not able to get it working completely. Following are the steps that I used:
- Modify the default Istio ingress controller to use “Nodeport” type and NEG.
- Write a custom healthcheck. Since the default GCP LB health check sends to port 80 “/” and Istio health check is at a different point(
port 15020on the/healthz/ready), we need to write a virtual service to do the remapping. This service is specified in healthcheck.yaml. - Deploy Istio gateway, virtualservice and the kubernetes ingress, services, deployments and pods.
Modify Istio ingress controller
Apply the following patch in istio-http-lb/gateway-patch directory to the Istio ingress controller:
kubectl -n istio-system patch svc istio-ingressgateway --type=json -p="$(cat istio-ingressgateway-patch.json)" --dry-run=true -o yaml | kubectl apply -f -Deploy services
kubectl apply -f istio-http-lbFor some reason, the health check was still failing. If someone has got this working, please ping me.
nginx Ingress controller
Ingress Kubernetes manifest file defaults to Global load balancer in GKE. GKE also supports integration with third party load balancers like nginx using annotation in the Ingress file. The first step would be to install nginx controller in the GKE cluster. As mentioned in the pre-requisites, this can be done using helm.
Following command shows the nginx services running after deploying nginx controller using helm. nginx-ingress-controller is the controller service, nginx-ingress-default-backend service provides the health check.
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-controller LoadBalancer 10.113.7.70 34.70.249.19 80:31545/TCP,443:31234/TCP 28h
nginx-ingress-default-backend ClusterIP 10.113.1.89 <none> 80/TCP 28hFollowing annotation in the Ingress manifest file sets up usage of nginx controller.
annotations:
kubernetes.io/ingress.class: nginxFollowing command deploys the application with nginx ingress controller:
kubectl apply -f nginxFollowing outputs shows the services, deployments, pods and ingress:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello NodePort 10.113.3.175 <none> 8080:30157/TCP 29h
hello2 NodePort 10.113.1.11 <none> 8080:30693/TCP 29h
kubernetes ClusterIP 10.113.0.1 <none> 443/TCP 46h
nginx-ingress-controller LoadBalancer 10.113.7.70 34.70.249.19 80:31545/TCP,443:31234/TCP 28h
nginx-ingress-default-backend ClusterIP 10.113.1.89 <none> 80/TCP 28h
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello 2/2 2 2 29h
hello2 2/2 2 2 29h
nginx-ingress-controller 1/1 1 1 29h
nginx-ingress-default-backend 1/1 1 1 29h
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-bc787595d-cwm52 1/1 Running 0 94m
hello-bc787595d-smvdl 1/1 Running 0 94m
hello2-7494666cc7-nkl8w 1/1 Running 0 94m
hello2-7494666cc7-sb9hm 1/1 Running 0 94m
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
ingress-resource * 34.70.249.19 80 29hFollowing command shows the NLB created by nginx ingress controller:
$ gcloud compute forwarding-rules list
NAME REGION IP_ADDRESS IP_PROTOCOL TARGET
aa83c60c4103d11ea9ae142010a80011 us-central1 34.70.249.19 TCP us-central1/targetPools/aa83c60c4103d11ea9ae142010a80011To access hello version v1, we need to do:
curl 34.70.249.19/v1
To access hello version v2, we need to do:
curl 34.70.249.19/v2How to choose?
Now that we have covered different ways to get traffic into the cluster, let’s talk about when to use each type. NLB is typically used for L4 based services. In the example application deployed above, we deployed 2 NLBs since NLB cannot do routing based on URL. HTTP load balancer with ingress is used for typical L7 services. Container native load balancing is built on top of HTTP load balancer and HTTP load balancer with NEG provides better distribution and health check, so this is preferred. If Istio is used as service mesh, it makes sense to use Istio ingress gateway as similar traffic rules can be used for traffic entering the mesh as well as within the mesh. Typically, Istio is front-ended by GCP NLB. In cases where we need to front-end with GCP HTTP LB, it is also possible as described above. There are some scenarios where customer is more comfortable with third party load balancer like nginx either because of familiarity or because of the complex routing/rewrite rules that nginx supports.
In the future blog, I will talk about internal communication between services inside GKE cluster. In that blog, I will discuss clusterip, Istio service mesh, Traffic director and ILB.
Sreenivas Makam's Blog 
1 thought on “Ingress into GKE Cluster”