Delete an orphaned PKS cluster

You may run into a situation where your PKS cluster is orphaned or you’re unable to delete it with the pks delete-cluster command. I’m going to show how to manually clean up the PKS cluster in the database.

In the background I’ve reproduced this issue by deleting the BOSH deployment that backs the PKS cluster. When I attempt to delete it with:

pks delete-cluster cluster1

I receive the error: Error: {}. However, if I run pks clusters, the cluster is still listed. The Kubernetes cluster VMs no longer exist in vCenter so we just need to remove the PKS cluster from the MariaDB running on the PKS VM. Let’s start by sshing to the PKS VM. We will need two pieces of information to do this:

  1. The environment name: 192.168.30.52
  2. The PKS deployment ID: pivotal-container-service-c4ffe959d20cd65146d6

Larger Image

pks-instance.png

Once we have this we can ssh into the PKS VM by running the following command:

bosh -e 192.168.30.51 -d pivotal-container-service-c4ffe959d20cd65146d6 ssh

Since there is only a single instance in the deployment, we don’t need to specify the instance name.

Once we are logged in, we can access the database by running:

PKS version 1.1 and later:

sudo -i
/var/vcap/packages/mariadb/bin/mysql -u root -p -S /var/vcap/sys/run/mysql/mysqld.sock

Pre 1.1:

sudo -i
/var/vcap/packages/mariadb/bin/mysql -u root -p

This will prompt us for a password, which we can get by logging into CF Ops Manager, selecting the PKS tile and selecting:

  1. Credentials tab
  2. Link to Credentials for the Cf Mysql Mysql Admin password

sqldb-creds.png

This will take us to a page where we can find the password (secret):

sqldb-pass.png

Paste this value into the password prompt and press enter.

Once we are logged into the the database server, we can select the PKS database by running:

use pks;

Show the available tables:

show tables;

db-tables.png

delete from cluster where name = ‘cluster1’;

Now when you run pks clusters the cluster will be gone.

Below is an image for all the SQL commands:

Larger image

db-actions.png



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s