In this post we will setup an HTTPS ingress with VMware PKS and go through some troubleshooting steps.

Deployment and Service

We will use an NGINX server to test with. Let’s start by creating an NGINX deployment:

kubectl run nginx –image nginx

Expose the server via a service:

kubectl expose deployment nginx –port 80

Certificates

In order to create an HTTPS Ingress, we need to create a Kubernetes TLS secret that we can use in the ingress.

In my lab I have a CA who’s certificate I’ve already uploaded into NSX-T manager under System > Trust > Certificates:

I used this CA to create a certificate (wildcard.pem) and key (wildcard-key.pem) that will be used below. The certificate’s CN is *.pks.vmware.local. Be aware that as of PKS 1.2 and NSX-T 2.3, the certificate validation only checks the certificate’s CN and not any DNS alternate names. This will be addressed in the future, but for now make sure your CN name is set appropriately. Here I’m going to create a TLS secret named pks-wildcard using the wildcard certificate and wildcard certificate key files in my current directory:

kubectl create secret tls pks-wildcard –cert wildcard.pem –key wildcard-key.pem

Let’s look at the pks-wildcard secret:

kubectl describe secret pks-wildcard

Name: pks-wildcard
Namespace: default
Labels: <none>
Annotations: <none>

Type: kubernetes.io/tls

Data
====
tls.crt: 1407 bytes
tls.key: 1675 bytes

Don’t worry about the tls.crt and tls.key names under Data. The certificate data will always look like this in the secret.

Ingress

Now we can use the TLS secret in our ingress. I’ll use the ingress defined below:

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  tls:
  - secretName: pks-wildcard
  hosts:
  - nginx.pks.vmware.local
  rules:
  - host: nginx.pks.vmware.local
    http:
      paths:
      - backend:
        serviceName: nginx
        servicePort: 80

Create the ingress:

kubectl apply -f ingress.yaml

The ingress will get applied on the pks-<cluster UUID>-https_terminated virtual server in NSX-T manager:

Let’s verify this by getting the ingress’s IP using kubectl:

kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
ingress-demo nginx.pks.vmware.local 10.20.0.15,100.64.64.23 80, 443 11m

Now we can create a DNS A record that maps *.pks.vmware.local to the virtual server’s IP. Verify by running nslookup or dig:

nslookup *.pks.vmware.local
Server: 192.168.1.254
Address: 192.168.1.254#53

Name: *.pks.vmware.local
Address: 10.20.0.15

If we open a browser, go to https://nginx.pks.vmware.local and verify the certificate, we will see that the correct certificate has been applied.

When we created the ingress the NCP process added the wildcard certificate to NSX-T’s certificate store. If you go to NSX-T Manager > System > Trust > Certificates, you’ll see the following certs:

Wildcard certificate that is managed by the ingress and NCP
The default certificated used when there are no matching certificates
My lab’s certificate authority

Take note of the wildcard and default certificate IDs and then go to NSX-T Manager > Networking > Virtual Servers > PKS cluster UUID-https_terminated > LB Profiles:

You will see that the Certificates field matches the ID of the wildcard certificate, and that the Default Certificate field matches the default certificate (nsx-lb).

When accessing your site by entering a name such as nginx.pks.vmware.local, NCP will verify that the domain name (pks.vmware.local) matches a certificate in the NSX-T Manager’s certificate store. For example, if you go to nginx.vmware.local, it will match the wildcard certificate *.pks.vmware.local.

If you enter a name that doesn’t end in pks.vmware.local, you’ll get the default certificate (nsx-lb). For example, if you go to https://10.20.0.15, you’ll see that this certificate is presented: