Delete an orphaned PKS cluster

You may run into a situation where your PKS cluster is orphaned or you’re unable to delete it with the pks delete-cluster command. I’m going to show how to manually clean up the PKS cluster in the database.

In the background I’ve reproduced this issue by deleting the BOSH deployment that backs the PKS cluster. When I attempt to delete it with:

pks delete-cluster cluster1

I receive the error: Error: {}. However, if I run pks clusters, the cluster is still listed. The Kubernetes cluster VMs no longer exist in vCenter so we just need to remove the PKS cluster from the MariaDB running on the PKS VM. Let’s start by sshing to the PKS VM. We will need two pieces of information to do this:

1. The environment name: 192.168.30.52
2. The PKS deployment ID: pivotal-container-service-c4ffe959d20cd65146d6

Larger Image

Once we have this we can ssh into the PKS VM by running the following command:

bosh -e 192.168.30.51 -d pivotal-container-service-c4ffe959d20cd65146d6 ssh

Since there is only a single instance in the deployment, we don’t need to specify the instance name.

Once we are logged in, we can access the database by running:

PKS version 1.1 and later:

sudo -i
/var/vcap/packages/mariadb/bin/mysql -u root -p -S /var/vcap/sys/run/mysql/mysqld.sock

Pre 1.1:

sudo -i
/var/vcap/packages/mariadb/bin/mysql -u root -p

This will prompt us for a password, which we can get by logging into CF Ops Manager, selecting the PKS tile and selecting:

1. Credentials tab
2. Link to Credentials for the Cf Mysql Mysql Admin password

This will take us to a page where we can find the password (secret):

Paste this value into the password prompt and press enter.

Once we are logged into the the database server, we can select the PKS database by running:

use pks;

Show the available tables:

show tables;

delete from cluster where name = ‘cluster1’;

Now when you run pks clusters the cluster will be gone.

Below is an image for all the SQL commands:

Larger image