Uncategorized

Exposing Pivotal Container Service via a reverse proxy

Posted by

At work we provide various types of environments that our users can select from a self-service portal. Pivotal Container Service (PKS) is one of these, and users often request that these environments be exposed externally to the internet. They need to be able to access the PKS API, Kubernetes API and various other services that often share the same ports. Since these these environments have a single public IP available, we use an NGINX reverse proxy to expose the services.

Below you’ll find the NGINX config files that facilitate what’s described above. The files can be separate and placed in the NGINX conf.d directory or combined and placed in the nginx.conf file.

In the config files you’ll see the use of the $host variable. You can read more about it here. Basically it tells NGINX to look at the host header of a request and proxy the request to the DNS entry for the host header. For example, if a user goes to nginx.pks.example.com, NGINX will perform a DNS lookup on nginx.pks.example.com and send the request to it.

Each environment also has self-service DNS that allows the users to create DNS records for their domain. If a user creates a new PKS cluster that gets the IP 10.100.0.10, they will create a DNS record for this IP. When they access the newly created DNS record, NGINX will resolve it to the 10.100.0.10 IP and send the request to it.

Since you can’t specify the certs of the Kubernetes API server during deploying with PKS, the cert on the load balancer will most likely be different from than the Kubernetes API cert, and will not automatically be loaded into your kubeconfig when you run pks get-credentials. You can place the Kubernetes API cert into your kubeconfig but running:

openssl s_client -connect cluster1.pks.example.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > kube.pem
kubectl config set-cluster cluster1 –certificate-authority=kube.pem –embed-certs=true

Or disabling TLS validation:

kubectl config set-cluster cluster1 –insecure-skip-tls-verify=1

I used Ubuntu 18.04.2 LTS and nginx/1.14.0 for this exercise. Lastly, I’m not an NGINX expert so if you find anything that could be improved, please leave a comment. Also, please don’t throw this into a production environment without testing and knowing what everything does. It hasn’t been tested under heavy loads as most environments are used for POCs, demos, etc.

general.conf

# General
ssl_certificate /etc/ssl/cert.pem;
ssl_certificate_key /etc/ssl/key.pem;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout  10m;
ssl_ciphers HIGH:!aNULL:!MD5;
resolver 127.0.0.53;
proxy_http_version 1.1;

# Allows "kubectl exec" and "kube log -f" to work.
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;

# For Ops Manager redirects
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

pks-api.conf

# Kube API 
server {
     listen  9021 ssl http2 default_server;
     location / {
         proxy_pass https://$host:9021;
     }
 }

kube-pks-auth.conf

# Kube Auth and PKS UAA
server {
     listen 8443 ssl http2 default_server;
     location / {
         proxy_pass https://$host:8443;
     }
 }

kube-http-lb.conf

# HTTP Kube load balancers
server {
     listen 80;
     server_name *.pks.example.com;
     location / {
         proxy_pass http://$host:80;
     }
 }

kube-https-lb.conf

# HTTPS Kube load balancers
server {
     listen 443 ssl http2;
     server_name *.pks.example.com;
     location / {
         proxy_pass https://$host:443;
     }
 }

kube-http-ingress.conf

# HTTP Ingress
server {
     listen 80;
     server_name *.cluster1.pks.example.com;
     location / {
         # LB in NSX-T responsible for HTTP ingress
         proxy_pass http://10.195.71.167:80;
         proxy_set_header Host $http_host;
     }
 }

kube-https-ingress.conf

# HTTPS Ingress
server {
     listen 443 ssl http2;
     server_name *.cluster1.pks.example.com;
     location / {
         # LB in NSX-T responsible for HTTPS ingress
         proxy_pass https://10.195.71.167:443;
         proxy_set_header Host $http_host;
     }
 }

Leave a comment