Upgrading from TKG 1.3 to 1.4 (including extensions) on vSphere

Tanzu Kuberentes Grid (TKG) 1.4 was just released on September 9th, 2021 and I couldn’t wait to take it for a spin. You can read all about the new features in the Release Notes and be sure to check out the updated documentation as there are several changes to how things work from 1.3.

My first task with exploring TKG 1.4 was to upgrade my existing 1.3.0 cluster and deployed extensions. This was a fairly painless process as you’ll see in the following post. For details on how my 1.3.0 environment was deployed, please see my previous posts, Installing Tanzu Kubernetes Grid 1.3 on vSphere with NSX Advanced Load Balancer, How to configure external-dns with Microsoft DNS in TKG 1.3 (plus Harbor and Contour) and Working with TKG Extensions and Shared Services in TKG 1.2.

Download updated components

You can download updated tanzu CLI binaries, node OS OVA files, signed kubectl, crashd and velero binaries from customerconnect.vmware.com.

Install the 1.4 tanzu CLI binary and additional executables

You can first verify your current tanzu CLI binary version via the tanzu version command:

tanzu version

version: v1.3.0
buildDate: 2021-03-16
sha: 06ddc9a

After downloading the updated tanzu CLI binary (tar/gz format), you will need to extract and install it on your local system:

tar -xvf tanzu-cli-bundle-linux-amd64.tar

cli/
cli/core/
cli/core/v1.4.0/
cli/core/v1.4.0/tanzu-core-linux_amd64
cli/core/plugin.yaml
cli/login/
cli/login/v1.4.0/
cli/login/v1.4.0/tanzu-login-linux_amd64
cli/login/plugin.yaml
cli/cluster/
cli/cluster/v1.4.0/
cli/cluster/v1.4.0/tanzu-cluster-linux_amd64
cli/cluster/plugin.yaml
cli/package/
cli/package/v1.4.0/
cli/package/v1.4.0/tanzu-package-linux_amd64
cli/package/plugin.yaml
cli/management-cluster/
cli/management-cluster/v1.4.0/
cli/management-cluster/v1.4.0/tanzu-management-cluster-linux_amd64
cli/management-cluster/plugin.yaml
cli/pinniped-auth/
cli/pinniped-auth/v1.4.0/
cli/pinniped-auth/v1.4.0/tanzu-pinniped-auth-linux_amd64
cli/pinniped-auth/plugin.yaml
cli/kubernetes-release/
cli/kubernetes-release/v1.4.0/
cli/kubernetes-release/v1.4.0/tanzu-kubernetes-release-linux_amd64
cli/kubernetes-release/plugin.yaml
cli/manifest.yaml
cli/ytt-linux-amd64-v0.34.0+vmware.1.gz
cli/kapp-linux-amd64-v0.37.0+vmware.1.gz
cli/imgpkg-linux-amd64-v0.10.0+vmware.1.gz
cli/kbld-linux-amd64-v0.30.0+vmware.1.gz
cli/vendir-linux-amd64-v0.21.1+vmware.1.gz
cd cli/

ubuntu@cli-vm:~/cli$ ls
cluster                                 kubernetes-release  pinniped-auth
core                                    login               vendir-linux-amd64-v0.21.1+vmware.1.gz
imgpkg-linux-amd64-v0.10.0+vmware.1.gz  management-cluster  ytt-linux-amd64-v0.34.0+vmware.1.gz
kapp-linux-amd64-v0.37.0+vmware.1.gz    manifest.yaml
kbld-linux-amd64-v0.30.0+vmware.1.gz    package
sudo install core/v1.4.0/tanzu-core-linux_amd64 /usr/local/bin/tanzu

You can now check to see that the tanzu CLI binary version has been updated:

tanzu version

!  Configuration is now stored in /home/ubuntu/.config/tanzu. Legacy configuration directory /home/ubuntu/.tanzu is deprecated and will be removed in a future release.
!  To complete migration, please remove legacy configuration directory /home/ubuntu/.tanzu and adjust your script(s), if any, to point to the new location.
version: v1.4.0
buildDate: 2021-08-30
sha: c9929b8f

Be sure to make note of the new configuration directory as noted in the tanzu version output.

You can follow a similar process to gunzip and install the other utilities located in the cli folder (imgpkg, kapp, kbld, vendir, ytt):

for file in $(ls *.gz)
do
gunzip $file
sudo install ${file::-3} /usr/local/bin/$(echo $file | awk -F - '{print $1}')
done

Lastly, you can follow a similar process to update the kubectl binary to the latest version:

gunzip kubectl-linux-v1.21.2+vmware.1.gz

sudo install kubectl-linux-v1.21.2+vmware.1 /usr/local/bin/kubectl

kubectl version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2+vmware.1", GitCommit:"54e7e68e30dd3f9f7bb4f814c9d112f54f0fb273", GitTreeState:"clean", BuildDate:"2021-06-28T22:17:36Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4+vmware.1", GitCommit:"d475bbd9e7cd66c6db7069cb447766daada65e3b", GitTreeState:"clean", BuildDate:"2021-02-22T22:15:46Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

Update the tanzu CLI plugins

The tanzu CLI plugins that were installed in a previous version also need to be updated but this a simple operation:

tanzu plugin list

  NAME                LATEST VERSION  DESCRIPTION                                                        REPOSITORY  VERSION  STATUS
  alpha               v1.3.1          Alpha CLI commands                                                 core                 not installed
  cluster             v1.3.1          Kubernetes cluster operations                                      core        v1.3.0   upgrade available
  kubernetes-release  v1.3.1          Kubernetes release operations                                      core        v1.3.0   upgrade available
  login               v1.3.1          Login to the platform                                              core        v1.3.0   upgrade available
  management-cluster  v1.3.1          Kubernetes management cluster operations                           core        v1.3.0   upgrade available
  pinniped-auth       v1.3.1          Pinniped authentication operations (usually not directly invoked)  core        v1.3.0   upgrade available
tanzu plugin clean
tanzu plugin list

  NAME                LATEST VERSION  DESCRIPTION                                                        REPOSITORY  VERSION  STATUS
  alpha               v1.3.1          Alpha CLI commands                                                 core                 not installed
  cluster             v1.3.1          Kubernetes cluster operations                                      core                 not installed
  kubernetes-release  v1.3.1          Kubernetes release operations                                      core                 not installed
  login               v1.3.1          Login to the platform                                              core                 not installed
  management-cluster  v1.3.1          Kubernetes management cluster operations                           core                 not installed
  pinniped-auth       v1.3.1          Pinniped authentication operations (usually not directly invoked)  core                 not installed

Note: This next command needs to be run from the cli directory noted previously.

tanzu plugin install --local . all
tanzu plugin list

  NAME                LATEST VERSION  DESCRIPTION                                                        REPOSITORY  VERSION  STATUS
  alpha               v1.3.1          Alpha CLI commands                                                 core                 not installed
  cluster             v1.3.1          Kubernetes cluster operations                                      core        v1.4.0   installed
  kubernetes-release  v1.3.1          Kubernetes release operations                                      core        v1.4.0   installed
  login               v1.3.1          Login to the platform                                              core        v1.4.0   installed
  management-cluster  v1.3.1          Kubernetes management cluster operations                           core        v1.4.0   installed
  package                             Tanzu package management                                                       v1.4.0   installed
  pinniped-auth       v1.3.1          Pinniped authentication operations (usually not directly invoked)  core        v1.4.0   installed

Deploy a new node OS template to vSphere

Since the upgrade includes support for newer Kubernetes versions, you’ll want to deploy the OVA files corresponding to the versions you’ll want/need (you’ll need at least the 1.21.2 OVA as it is required for the management cluster).

I like to take a snapshot of the VM before converting it to a template so that I can deploy my nodes as linked-clones and save time and disk space. Keep in mind that if you do this, you will not be able to resize the node disk size during cluster creation.

You’ll want to repeat this for as many different Kubernetes versions (or nodes OS types) that you might want to deploy.

Validate current management and workload cluster

With the prerequisites out of the way, you can move on to inspecting your current installation prior to upgrading.

If you’re not already accessing your management cluster, you’ll need to use the tanzu login command to get started:

tanzu login

? Select a server tkg-mgmt            ()
✔  successfully logged in to management cluster using the kubeconfig tkg-mgmt
tanzu management-cluster get

  NAME      NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES
  tkg-mgmt  tkg-system  running  1/1           1/1      v1.21.2+vmware.1  management


Details:

NAME                                                         READY  SEVERITY  REASON  SINCE  MESSAGE
/tkg-mgmt                                                    True                     26m
├─ClusterInfrastructure - VSphereCluster/tkg-mgmt            True                     160d
├─ControlPlane - KubeadmControlPlane/tkg-mgmt-control-plane  True                     26m
│ └─Machine/tkg-mgmt-control-plane-cqrs4                     True                     33m
└─Workers
  └─MachineDeployment/tkg-mgmt-md-0
    └─Machine/tkg-mgmt-md-0-5895d84d4c-ljmr9                 True                     22m


Providers:

  NAMESPACE                          NAME                    TYPE                    PROVIDERNAME  VERSION  WATCHNAMESPACE
  capi-kubeadm-bootstrap-system      bootstrap-kubeadm       BootstrapProvider       kubeadm       v0.3.23
  capi-kubeadm-control-plane-system  control-plane-kubeadm   ControlPlaneProvider    kubeadm       v0.3.23
  capi-system                        cluster-api             CoreProvider            cluster-api   v0.3.23
  capv-system                        infrastructure-vsphere  InfrastructureProvider  vsphere       v0.7.10
tanzu cluster list --include-management-cluster

  NAME      NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES       PLAN
  tkg-wld   default     running  1/1           2/2      v1.20.4+vmware.1  <none>      dev
  tkg-mgmt  tkg-system  running  1/1           1/1      v1.20.4+vmware.1  management  dev

You can see from the previous output that I have single control plane node management cluster and workload clusters present, both running Kubernetes version 1.20.4.

Upgrade the management cluster

Note: I you’re using NSX Advanced Load Balancer (NSX ALB) as I am, be sure that you are on a supported version. The supported versions for TKG 1.4 are 20.1.3 and 20.1.6. If you need to upgraded, check out my previous post, Upgrading NSX Advanced Load Balancer.

There are a few parameters that you can pass to the tanzu management-cluster upgrade command but I didn’t need any of them since my installation is fairly simple:

tanzu management-cluster upgrade

Upgrading management cluster 'tkg-mgmt' to TKG version 'v1.4.0' with Kubernetes version 'v1.21.2+vmware.1'. Are you sure? [y/N]: y

A new control plane VM will be created from the 1.21 node OS image that was uploaded previously.

Once it’s powered on, it will be joined to the existing cluster (using kubeadm) and then additional control plane and worker nodes will be created and joined to the cluster. The original nodes will be powered off and deleted.

You can see that the new and old control plane nodes are both present part way through the upgrade:

kubectl get nodes

NAME                             STATUS   ROLES                  AGE     VERSION
tkg-mgmt-control-plane-6vd6z     Ready    control-plane,master   160d    v1.20.4+vmware.1
tkg-mgmt-control-plane-cqrs4     Ready    control-plane,master   3m12s   v1.21.2+vmware.1
tkg-mgmt-md-0-766c78c69c-7r66n   Ready    <none>                 160d    v1.20.4+vmware.1

And once the new control plane node is fully joined to the cluster, the old control plane node is deactivated and deleted:

kubectl get nodes

NAME                             STATUS                     ROLES                  AGE    VERSION
tkg-mgmt-control-plane-6vd6z     Ready,SchedulingDisabled   control-plane,master   160d   v1.20.4+vmware.1
tkg-mgmt-control-plane-cqrs4     Ready                      control-plane,master   5m9s   v1.21.2+vmware.1
tkg-mgmt-md-0-766c78c69c-7r66n   Ready                      <none>                 160d   v1.20.4+vmware.1
kubectl get nodes
NAME                             STATUS   ROLES                  AGE     VERSION
tkg-mgmt-control-plane-cqrs4     Ready    control-plane,master   7m17s   v1.21.2+vmware.1
tkg-mgmt-md-0-766c78c69c-7r66n   Ready    <none>                 160d    v1.20.4+vmware.1

This same process is repeated for the worker nodes in the management cluster. When the upgrade is complete, you should see all control plane and worker nodes running at the new version:

kubectl get nodes

NAME                             STATUS   ROLES                  AGE     VERSION
tkg-mgmt-control-plane-cqrs4     Ready    control-plane,master   14m     v1.21.2+vmware.1
tkg-mgmt-md-0-5895d84d4c-ljmr9   Ready    <none>                 3m45s   v1.21.2+vmware.1

And the same tanzu commands that were run earlier to inspect the clusters will show that the management cluster is now running on a newer version:

tanzu management-cluster get

  NAME      NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES
  tkg-mgmt  tkg-system  running  1/1           1/1      v1.21.2+vmware.1  management


Details:

NAME                                                         READY  SEVERITY  REASON  SINCE  MESSAGE
/tkg-mgmt                                                    True                     9m49s
├─ClusterInfrastructure - VSphereCluster/tkg-mgmt            True                     160d
├─ControlPlane - KubeadmControlPlane/tkg-mgmt-control-plane  True                     9m49s
│ └─Machine/tkg-mgmt-control-plane-cqrs4                     True                     17m
└─Workers
  └─MachineDeployment/tkg-mgmt-md-0
    └─Machine/tkg-mgmt-md-0-5895d84d4c-ljmr9                 True                     5m49s


Providers:

  NAMESPACE                          NAME                    TYPE                    PROVIDERNAME  VERSION  WATCHNAMESPACE
  capi-kubeadm-bootstrap-system      bootstrap-kubeadm       BootstrapProvider       kubeadm       v0.3.23
  capi-kubeadm-control-plane-system  control-plane-kubeadm   ControlPlaneProvider    kubeadm       v0.3.23
  capi-system                        cluster-api             CoreProvider            cluster-api   v0.3.23
  capv-system                        infrastructure-vsphere  InfrastructureProvider  vsphere       v0.7.10
tanzu cluster list --include-management-cluster

  NAME      NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES       PLAN
  tkg-wld   default     running  1/1           2/2      v1.20.4+vmware.1  <none>      dev
  tkg-mgmt  tkg-system  running  1/1           1/1      v1.21.2+vmware.1  management  dev

You can expand the link below to see an example of the default log output during a management cluster upgrade:

tanzu management-cluster upgrade output
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading management cluster providers...
Checking cert-manager version...
Deleting cert-manager Version="v0.16.1"
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Performing upgrade...
Deleting Provider="cluster-api" Version="" TargetNamespace="capi-system"
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Deleting Provider="bootstrap-kubeadm" Version="" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Deleting Provider="control-plane-kubeadm" Version="" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Deleting Provider="infrastructure-vsphere" Version="" TargetNamespace="capv-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Management cluster providers upgraded successfully...
Upgrading management cluster kubernetes version...
Verifying kubernetes version...
Retrieving configuration for upgrade cluster...
Create InfrastructureTemplate for upgrade...
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1...
Waiting for kubernetes version to be updated for control plane nodes
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1...
Waiting for kubernetes version to be updated for worker nodes...
updating additional components: 'metadata/tkg' ...
updating additional components: 'addons-management/kapp-controller' ...
updating additional components: 'addons-management/standard-package-repo' ...
updating additional components: 'addons-management/tanzu-addons-manager' ...
updating additional components: 'tkr/tkr-controller' ...
updating additional components: 'addons-management/core-package-repo' ...
Waiting for additional components to be up and running...
Waiting for packages to be up and running...
Management cluster 'tkg-mgmt' successfully upgraded to TKG version 'v1.4.0' with kubernetes version 'v1.21.2+vmware.1'

Note: I went through this upgrade process a few times and once saw the following message at the end of the management cluster upgrade process:

Warning: Management cluster is upgraded successfully, but some packages are failing. Failure while waiting for packages to be installed: package reconciliation failed: kapp: Error: Timed out waiting after 30s

Everything was fine in my management cluster and it just appeared to take a little longer than expected for all pods to be up and running again. You can quickly check the status via the kubectl get po -A command.

Update the Pinniped settings

This wasn’t a concern for me as I am using LDAP (AD) authentication in my TKG clusters but if you had configured OIDC authentication prior to upgrading TKG to 1.4, you will need to follow the instructions noted at Update Pinniped Settings for Management Clusters with OIDC Authentication. The is because both Dex and Pinniped were needed for OIDC and LDAP authentication in TKG 1.3 but only Pinniped is needed for OIDC authentication in TKG 1.4 (LDAP still needs Dex). If you’re using LDAP authentication like I am, you can skip this step.

Upgrade the workload cluster(s)

You’ll want to get a list of your clusters first:

tanzu cluster list

  NAME     NAMESPACE  STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES   PLAN
  tkg-wld  default    running  1/1           2/2      v1.20.4+vmware.1  <none>  dev

And you’ll also want to see what available upgrades you have.

tanzu kubernetes-release get

  NAME                       VERSION                  COMPATIBLE  ACTIVE  UPDATES AVAILABLE
  v1.19.12---vmware.1-tkg.1  v1.19.12+vmware.1-tkg.1  True        True    True
  v1.20.8---vmware.1-tkg.2   v1.20.8+vmware.1-tkg.2   True        True    True
  v1.21.2---vmware.1-tkg.1   v1.21.2+vmware.1-tkg.1   True        True    False

There is a new feature with the kubernetes-release option to tanzu that allows you to activate or deactivate specific tkr versions. For example, if I don’t want to see the 1.19.12 version any longer, I can run tanzu kubernetes-release deactivate v1.19.12---vmware.1-tkg.1 to remove it from the list of available upgrades. After making this change, tanzu kuberentes-release get will no longer show it:

tanzu kubernetes-release get

  NAME                      VERSION                 COMPATIBLE  ACTIVE  UPDATES AVAILABLE
  v1.20.8---vmware.1-tkg.2  v1.20.8+vmware.1-tkg.2  True        True    True
  v1.21.2---vmware.1-tkg.1  v1.21.2+vmware.1-tkg.1  True        True    False

If you want to see all tkr versions, you can pass the -a switch to tanzu kubernetes-release get since by default, it only returns compatible, active releases:

 tanzu kubernetes-release get -a

  NAME                             VERSION                        COMPATIBLE  ACTIVE  UPDATES AVAILABLE
  v1.17.16---vmware.2-tkg.1        v1.17.16+vmware.2-tkg.1        False       True    True
  v1.17.16---vmware.2-tkg.2        v1.17.16+vmware.2-tkg.2        False       True    True
  v1.17.16---vmware.3-tkg.1        v1.17.16+vmware.3-tkg.1        False       True    True
  v1.18.16---vmware.1-tkg.1        v1.18.16+vmware.1-tkg.1        False       True    True
  v1.18.16---vmware.1-tkg.2        v1.18.16+vmware.1-tkg.2        False       True    True
  v1.18.16---vmware.3-tkg.1        v1.18.16+vmware.3-tkg.1        False       True    True
  v1.18.17---vmware.1-tkg.1        v1.18.17+vmware.1-tkg.1        False       True    True
  v1.18.17---vmware.2-tkg.1        v1.18.17+vmware.2-tkg.1        False       True    True
  v1.19.12---vmware.1-tkg.1        v1.19.12+vmware.1-tkg.1        True        False   True
  v1.19.8---vmware.1-tkg.1         v1.19.8+vmware.1-tkg.1         False       True    True
  v1.19.8---vmware.1-tkg.2         v1.19.8+vmware.1-tkg.2         False       True    True
  v1.19.8---vmware.3-tkg.1         v1.19.8+vmware.3-tkg.1         False       True    True
  v1.19.9---vmware.1-tkg.1         v1.19.9+vmware.1-tkg.1         False       True    True
  v1.19.9---vmware.2-tkg.1         v1.19.9+vmware.2-tkg.1         False       True    True
  v1.20.4---vmware.1-tkg.1         v1.20.4+vmware.1-tkg.1         False       True    True
  v1.20.4---vmware.1-tkg.2         v1.20.4+vmware.1-tkg.2         False       True    True
  v1.20.4---vmware.3-tkg.1         v1.20.4+vmware.3-tkg.1         False       True    True
  v1.20.5---vmware.1-tkg.1         v1.20.5+vmware.1-tkg.1         False       True    True
  v1.20.5---vmware.2-fips.1-tkg.1  v1.20.5+vmware.2-fips.1-tkg.1  False       True    True
  v1.20.5---vmware.2-tkg.1         v1.20.5+vmware.2-tkg.1         False       True    True
  v1.20.8---vmware.1-tkg.2         v1.20.8+vmware.1-tkg.2         True        True    True
  v1.21.2---vmware.1-tkg.1         v1.21.2+vmware.1-tkg.1         True        True    False

As noted previously, my cluster is running on 1.20.4 so I will check to see what upgrades are available:

tanzu kubernetes-release available-upgrades get v1.20.4---vmware.1

There are no available upgrades for this TanzuKubernetesRelease
  NAME  VERSION

Hmmm, that’s certainly not what I expected. I tried it a little differently (passed in the cluster name) to see if I could get the results I was looking for:

tanzu kubernetes-release available-upgrades get tkg-wld

There are no available upgrades for this TanzuKubernetesRelease
  NAME  VERSION

Same thing. It turns out that this is a known issue as documented in the Known Issues section of the TKG 1.4 Release Notes (look for tanzu kubernetes-release available-upgrades does not show available upgrades for a given Kubernetes version). The workaround is to validate that there is an available version to upgrade to in the tanzu kubernetes-release get output and then pass the version explicitly when performing the upgrade.

One thing I did notice was that I can use a tanzu cluster command to list what upgrades are available:

tanzu cluster available-upgrades get tkg-wld

  NAME                             VERSION                        COMPATIBLE
  v1.20.4---vmware.1-tkg.2         v1.20.4+vmware.1-tkg.2         False
  v1.20.4---vmware.3-tkg.1         v1.20.4+vmware.3-tkg.1         False
  v1.20.5---vmware.1-tkg.1         v1.20.5+vmware.1-tkg.1         False
  v1.20.5---vmware.2-fips.1-tkg.1  v1.20.5+vmware.2-fips.1-tkg.1  False
  v1.20.5---vmware.2-tkg.1         v1.20.5+vmware.2-tkg.1         False
  v1.20.8---vmware.1-tkg.2         v1.20.8+vmware.1-tkg.2         True
  v1.21.2---vmware.1-tkg.1         v1.21.2+vmware.1-tkg.1         True

I’m going to upgrade my cluster to 1.21.2 and since I already have that version saved as a template in vSphere, I should be ready to go at this point:

tanzu cluster upgrade tkg-wld --tkr v1.21.2---vmware.1-tkg.1

Upgrading workload cluster 'tkg-wld' to kubernetes version 'v1.21.2+vmware.1'. Are you sure? [y/N]: y

As with the management cluster, you’ll see the control plane VM(s) getting created first and the old ones deleted and then the worker nodes will follow.

Before the upgrade:

kubectl --context=tkg-wld-admin@tkg-wld get nodes

NAME                            STATUS   ROLES                  AGE    VERSION
tkg-wld-control-plane-bfgdk     Ready    control-plane,master   160d   v1.20.4+vmware.1
tkg-wld-md-0-84c4d7898c-ds2b2   Ready    <none>                 160d   v1.20.4+vmware.1
tkg-wld-md-0-84c4d7898c-vv9g2   Ready    <none>                 160d   v1.20.4+vmware.1

And after the upgrade:

kubectl --context=tkg-wld-admin@tkg-wld get nodes

NAME                            STATUS   ROLES                  AGE    VERSION
tkg-wld-control-plane-nnvtr     Ready    control-plane,master   25m    v1.21.2+vmware.1
tkg-wld-md-0-5c98898f95-9x2mp   Ready    <none>                 16m    v1.21.2+vmware.1
tkg-wld-md-0-5c98898f95-k9fsx   Ready    <none>                 4m35s   v1.21.2+vmware.1

You can expand the link below to see an example of the default log output during a management cluster upgrade:

tanzu cluster upgrade output
Validating configuration...
Verifying kubernetes version...
Retrieving configuration for upgrade cluster...
Create InfrastructureTemplate for upgrade...
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1...
Waiting for kubernetes version to be updated for control plane nodes
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1...
Waiting for kubernetes version to be updated for worker nodes...
updating additional components: 'metadata/tkg' ...
updating additional components: 'addons-management/kapp-controller' ...
updating additional components: 'addons-management/standard-package-repo' ...
Waiting for packages to be up and running...
Cluster 'tkg-wld' successfully upgraded to kubernetes version 'v1.21.2+vmware.1'

If you have applications deployed that use persistent volumes, you’ll see a lot of activity similar to the following in the vSphere Client when the upgrade process gets near the end:

This is expected since all of the virtual disks that back the PVCs have to be reattached to the new worker nodes.

One thing I noticed that is worth mentioning is that even though the upgrade process was finished, my deployed applications were not all available for a few more minutes. Harbor was the longest holdout at being fully functional about 10 minutes after the upgrade completed.

Update the AKO operator

The AKO operator controls the ability to work with NSX Advanced Load Balancer (NSX ALB) for creating and managing load balancer services. In the 1.4 version of TKG, this component is installed as a package. Since we’re upgrading, that package doesn’t exist and needs to be installed (preserving the existing configuration, of course). If you are not using NSX ALB you can skip this section.

Taking a look at the installed packages in the management cluster, you can see that the AKO Operator package is not installed (I have a lot more on Packages in this post):

tanzu package installed list -A

- Retrieving installed packages...
  NAME                  PACKAGE-NAME                     PACKAGE-VERSION  STATUS               NAMESPACE
  antrea                antrea.tanzu.vmware.com                           Reconcile succeeded  tkg-system
  metrics-server        metrics-server.tanzu.vmware.com                   Reconcile succeeded  tkg-system
  pinniped              pinniped.tanzu.vmware.com                         Reconcile succeeded  tkg-system
  tanzu-addons-manager  addons-manager.tanzu.vmware.com                   Reconcile succeeded  tkg-system
  vsphere-cpi           vsphere-cpi.tanzu.vmware.com                      Reconcile succeeded  tkg-system
  vsphere-csi           vsphere-csi.tanzu.vmware.com                      Reconcile succeeded  tkg-system

You will first need to export a few environment variables that will be used by the subsequent tanzu cluster create command.

export _TKG_CLUSTER_FORCE_ROLE="management"
export FILTER_BY_ADDON_TYPE="networking/ako-operator"
export REMOVE_CRS_FOR_ADDON_TYPE="networking/ako-operator"
export NAMESPACE="tkg-system"

You need to issue a tanzu cluster create command with the --dry-run option to create an AKO operator addon secret manifest file.

tanzu cluster create tkg-mgmt --dry-run -f ~/.config/tanzu/tkg/clusterconfigs/rj3duprspd.yaml --vsphere-controlplane-endpoint 1.1.1.1 > ako-operator-addon-manifest.yaml

Note: Replace tkg-mgmt with the name of your management cluster and rj3duprspd.yaml with the name of the cluster configuration file that was created when your management cluster was created. You can leave the 1.1.1.1 address as-is since it is just there as a placeholder and not really used in the generated manifest.

You should end up with a file similar to the following:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    tkg.tanzu.vmware.com/addon-type: networking/ako-operator
  labels:
    clusterctl.cluster.x-k8s.io/move: ""
    tkg.tanzu.vmware.com/addon-name: ako-operator
    tkg.tanzu.vmware.com/cluster-name: tkg-mgmt
  name: tkg-mgmt-ako-operator-addon
  namespace: tkg-system
stringData:
  values.yaml: |
    #@data/values
    #@overlay/match-child-defaults missing_ok=True
    ---
    akoOperator:
      avi_enable: true
      namespace: tkg-system-networking
      cluster_name: tkg-mgmt
      config:
        avi_disable_ingress_class: true
        avi_ingress_default_ingress_controller: false
        avi_ingress_shard_vs_size: ""
        avi_ingress_service_type: ""
        avi_ingress_node_network_list: '""'
        avi_admin_credential_name: avi-controller-credentials
        avi_ca_name: avi-controller-ca
        avi_controller: nsxalb-cluster.corp.tanzu
        avi_username: admin
        avi_password: VMware1!
        avi_cloud_name: Default-Cloud
        avi_service_engine_group: Default-Group
        avi_data_network: K8s-Frontend
        avi_data_network_cidr: 192.168.220.0/23
        avi_ca_data_b64: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZhekNDQTFPZ0F3SUJBZ0lRTWZaeTA4bXV2SVZLZFpWRHo3L3JZekFOQmdrcWhraUc5dzBCQVFzRkFEQkkKTVJVd0V3WUtDWkltaVpQeUxHUUJHUllGZEdGdWVuVXhGREFTQmdvSmtpYUprL0lzWkFFWkZnUmpiM0p3TVJrdwpGd1lEVlFRREV4QkRUMDVVVWs5TVEwVk9WRVZTTFVOQk1CNFhEVEl3TURneE9URTNNakEwTkZvWERUTXdNRGd4Ck9URTNNekF6TlZvd1NERVZNQk1HQ2dtU0pvbVQ4aXhrQVJrV0JYUmhibnAxTVJRd0VnWUtDWkltaVpQeUxHUUIKR1JZRVkyOXljREVaTUJjR0ExVUVBeE1RUTA5T1ZGSlBURU5GVGxSRlVpMURRVENDQWlJd0RRWUpLb1pJaHZjTgpBUUVCQlFBRGdnSVBBRENDQWdvQ2dnSUJBTEtJZFg3NjQzUHp2dFZYbHFOSXdEdU5xK3JoY0hGMGZqUjQxNGorCjFJR1FVdVhyeWtqaFNEdGhQUCs4QkdON21CZ0hUOEFqQVMxYjk1eGM4QjBTMkZobG4zQW9SRTl6MDNHdGZzQnUKRlNCUlVWd0FpZlg2b1h1OTdXemZmaHFQdHhaZkxKWGJoT29tamxrWDZpZmZBczJUT0xVeDJPajR3MnZ5Ymh6agpsY0E3MGFpKzBTbDZheFNvM2xNWjRLa3VaMldnZkVjYURqamozMy9wVjMvYm5GSys3eWRQdHRjMlRlazV4c0k4ClhOTWlySVZ4VWlVVDRZTHk0V0xpUzIwMEpVZmJwMVpuTXZuYlE4SnYxUW5abDlXN1dtQlBjZ3hSNEFBdWIwSzQKdlpMWHU2TVhpYm9UbHprTUIvWXRoQ2tUTmxKY0traEhmNjBZUi9UNlN4MVQybnVweUJhNGRlbzVVR1B6aFJpSgpwTjM3dXFxQWRLMXFNRHBDakFSalM2VTdMZjlKS2pmaXJpTHpMZXlBalA4a2FONFRkSFNaZDBwY1FvWlN4ZXhRCjluKzRFNE1RbTRFSjREclZaQ2lsc3lMMkJkRVRjSFhLUGM3cStEYjRYTTdqUEtORzVHUDFFTVY0WG9odjU4eVoKL3JSZm1LNjRnYXI4QU1uT0tUMkFQNjgxcWRaczdsbGpPTmNYVUFMemxYNVRxSWNoWVQwRFZRbUZMWW9NQmVaegowbDIxUWpiSzBZV25QemE2WWkvTjRtNnJGYkVCNFdYaXFoWVNreHpyTXZvY1ZVZ2Q0QUFQMXZmSE5uRkVzblVSCm5Tc2lnbEZIL3hseU8zY0JGcm1vWkF4YkEyMDkxWEhXaEI0YzBtUUVJM2hPcUFCOFVvRkdCclFwbVErTGVzb0MKMUxaOUFnTUJBQUdqVVRCUE1Bc0dBMVVkRHdRRUF3SUJoakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZApEZ1FXQkJURkF4U3ZZNjRRNWFkaG04SVllY0hCQVV1b2J6QVFCZ2tyQmdFRUFZSTNGUUVFQXdJQkFEQU5CZ2txCmhraUc5dzBCQVFzRkFBT0NBZ0VBamcvdjRtSVA3Z0JWQ3c0cGVtdEduM1BTdERoL2FCOXZiV3lqQXl4U05hYUgKSDBuSUQ1cTV3b3c5dWVCaURmalRQbmhiZjNQNzY4SEc4b0wvKzlDK1ZtLzBsaUZCZCswL0RhYXlLcEFORk1MQgpCVitzMmFkV1JoUXVjTFFmWFB3dW04UnliV3Y4MndrUmtXQ0NkT0JhQXZBTXVUZ2swOFN3Skl5UWZWZ3BrM25ZCjBPd2pGd1NBYWR2ZXZmK0xvRC85TDhSOU5FdC9uNFdKZStMdEVhbW85RVZiK2wrY1lxeXh5dWJBVlkwWTZCTTIKR1hxQWgzRkVXMmFRTXB3b3VoLzVTN3c1b1NNWU42bWlZMW9qa2k4Z1BtMCs0K0NJTFBXaC9mcjJxME8vYlB0YgpUcisrblBNbVo4b3Y5ZXBOR0l1cWh0azVqYTIvSnVZK1JXNDZJUmM4UXBGMUV5VWFlMDJFNlUyVmFjczdHZ2UyCkNlU0lOa29MRkZtaUtCZkluL0hBY2hsbWU5YUw2RGxKOXdBcmVCREgzRThrSDdnUkRXYlNLMi9RRDBIcWFjK0UKZ2VHSHdwZy84T3RCT0hVTW5NN2VMT1hCSkZjSm9zV2YwWG5FZ1M0dWJnYUhncURFdThwOFBFN3JwQ3h0VU51cgp0K3gyeE9OSS9yQldnZGJwNTFsUHI3bzgxOXpQSkN2WVpxMVBwMXN0OGZiM1JsVVNXdmJRTVBGdEdBeWFCeStHCjBSZ1o5V1B0eUVZZ25IQWI1L0RxNDZzbmU5L1FuUHd3R3BqdjFzMW9FM1pGUWpodm5HaXM4K2RxUnhrM1laQWsKeWlEZ2hXN2FudHpZTDlTMUNDOHNWZ1ZPd0ZKd2ZGWHBkaWlyMzVtUWx5U0czMDFWNEZzUlYrWjBjRnA0TmkwPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
        avi_labels: '""'
        avi_disable_static_route_sync: true
        avi_cni_plugin: antrea
        avi_control_plane_ha_provider: false
        avi_management_cluster_vip_network_name: K8s-Frontend
        avi_management_cluster_vip_network_cidr: 192.168.220.0/23
        avi_control_plane_endpoint_port: 6443
type: tkg.tanzu.vmware.com/addon

You can now apply the manifest which will create the AKO Operator addon secret and trigger the installation of the AKO Operator package:

kubectl apply -f ako-operator-addon-manifest.yaml

secret/tkg-mgmt-ako-operator-addon created

You can see that the AKO Operator package and the load-balancer-and-ingress-service package are now installed:

tanzu package installed list -A

- Retrieving installed packages...
  NAME                               PACKAGE-NAME                                        PACKAGE-VERSION  STATUS               NAMESPACE
  ako-operator                       ako-operator.tanzu.vmware.com                                        Reconcile succeeded  tkg-system
  antrea                             antrea.tanzu.vmware.com                                              Reconcile succeeded  tkg-system
  load-balancer-and-ingress-service  load-balancer-and-ingress-service.tanzu.vmware.com                   Reconcile succeeded  tkg-system
  metrics-server                     metrics-server.tanzu.vmware.com                                      Reconcile succeeded  tkg-system
  pinniped                           pinniped.tanzu.vmware.com                                            Reconcile succeeded  tkg-system
  tanzu-addons-manager               addons-manager.tanzu.vmware.com                                      Reconcile succeeded  tkg-system
  vsphere-cpi                        vsphere-cpi.tanzu.vmware.com                                         Reconcile succeeded  tkg-system
  vsphere-csi                        vsphere-csi.tanzu.vmware.com                                         Reconcile succeeded  tkg-system

The load-balancer-and-ingress-service package will automatically be installed to any workload clusters as well:

 tanzu package installed list -A

/ Retrieving installed packages...
  NAME                               PACKAGE-NAME                                        PACKAGE-VERSION  STATUS                                                                NAMESPACE
  antrea                             antrea.tanzu.vmware.com                                              Reconcile succeeded                                                   tkg-system
  load-balancer-and-ingress-service  load-balancer-and-ingress-service.tanzu.vmware.com                   Reconcile failed: Error (see .status.usefulErrorMessage for details)  tkg-system
  metrics-server                     metrics-server.tanzu.vmware.com                                      Reconcile succeeded                                                   tkg-system
  pinniped                           pinniped.tanzu.vmware.com                                            Reconcile succeeded                                                   tkg-system
  vsphere-cpi                        vsphere-cpi.tanzu.vmware.com                                         Reconcile succeeded                                                   tkg-system
  vsphere-csi                        vsphere-csi.tanzu.vmware.com                                         Reconcile succeeded                                                   tkg-system

You can see that it’s not in a healthy state though. If you dig further into it you’ll see a little more detail about why:

tanzu package installed get load-balancer-and-ingress-service -n tkg-system

/ Retrieving installation details for load-balancer-and-ingress-service...
NAME:                    load-balancer-and-ingress-service
PACKAGE-NAME:            load-balancer-and-ingress-service.tanzu.vmware.com
PACKAGE-VERSION:
STATUS:                  Reconcile failed: Error (see .status.usefulErrorMessage for details)
CONDITIONS:              [{ReconcileFailed True  Error (see .status.usefulErrorMessage for details)}]
USEFUL-ERROR-MESSAGE:    kapp: Error: Applying update statefulset/ako (apps/v1) namespace: avi-system:
  Updating resource statefulset/ako (apps/v1) namespace: avi-system:
    StatefulSet.apps "ako" is invalid: spec:
      Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden (reason: Invalid)

The package is trying to make a change to the existing ako statefulset that is now allowed. This is easy enough to work around by deleting the existing stateful set and letting the package framework recreate it.

kubectl -n avi-system delete statefulsets.apps ako

statefulset.apps "ako" deleted


tanzu package installed get load-balancer-and-ingress-service -n tkg-system

- Retrieving installation details for load-balancer-and-ingress-service...
NAME:                    load-balancer-and-ingress-service
PACKAGE-NAME:            load-balancer-and-ingress-service.tanzu.vmware.com
PACKAGE-VERSION:
STATUS:                  Reconcile succeeded
CONDITIONS:              [{ReconcileSucceeded True  }]
USEFUL-ERROR-MESSAGE:

This is a known issue affecting TKG 1.4 and should be addressed in a future version.

One other oddity I noticed was that sometimes my enovy service got a new IP address from NSX ALB. To get back to the original IP address it was as easy as editing the service and specifying loadBalancerIP: <previous IP> and then deleting any envoy virtual services in NSX ALB. A new envoy virtual service will be created after a few minutes with the desired IP address.

Update the extensions

NOTE: These instructions are specific to coming from a TKG 1.3 installation. The extension framework was changed slightly in 1.3.1. If you are using these instructions but are running TKG 1.3.1, you do not need to delete the extensions prior to installing the packages…i.e. you don’t want to run and of the commands similar to kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/ingress/contour/contour-extension.yaml as they will remove the installed app and any stateful data.

From several of my previous posts, you may know that I have installed all of the available TKG extensions into my workload cluster (Cert Manager, Harbor, Contour, External DNS, Fluentbit, Prometheus and Grafana). The extension framework has been removed in TKG 1.4 and all of these applications are now installed as “Packages” The package framework makes great use of the kapp-controller and should ease installation and lifecycle management going forward. You can read more about kapp-controller and packages in general at https://carvel.dev/kapp-controller/.

You can see what extensions are installed via the kubectl get apps command (run from the context of the cluster in question):

kubectl get apps -A

NAMESPACE                        NAME                                DESCRIPTION           SINCE-DEPLOY   AGE
tanzu-system-ingress             contour                             Reconcile succeeded   57s            161d
tanzu-system-logging             fluent-bit                          Reconcile succeeded   57s            160d
tanzu-system-monitoring          grafana                             Reconcile succeeded   80s            160d
tanzu-system-monitoring          prometheus                          Reconcile succeeded   9s             160d
tanzu-system-registry            harbor                              Reconcile succeeded   98s            160d
tanzu-system-service-discovery   external-dns                        Reconcile succeeded   59s            161d
tkg-system                       antrea                              Reconcile succeeded   4m7s           161d
tkg-system                       load-balancer-and-ingress-service   Reconcile succeeded   1m8s           3m4s
tkg-system                       metrics-server                      Reconcile succeeded   41s            161d
tkg-system                       pinniped                            Reconcile succeeded   4m46s          161d
tkg-system                       vsphere-cpi                         Reconcile succeeded   61s            161d
tkg-system                       vsphere-csi                         Reconcile succeeded   3m22s          161d

You’ll need to make a note of value in the NAME column as it will be used later when we install the replacement package.

To determine what packages are available, you can use the new tanzu package command:

tanzu package available list -A

/ Retrieving available packages...
  NAME                                                DISPLAY-NAME                       SHORT-DESCRIPTION                                                                                                                                                                                       NAMESPACE
  cert-manager.tanzu.vmware.com                       cert-manager                       Certificate management                                                                                                                                                                                  tanzu-package-repo-global
  contour.tanzu.vmware.com                            Contour                            An ingress controller                                                                                                                                                                                   tanzu-package-repo-global
  external-dns.tanzu.vmware.com                       external-dns                       This package provides DNS synchronization functionality.                                                                                                                                                tanzu-package-repo-global
  fluent-bit.tanzu.vmware.com                         fluent-bit                         Fluent Bit is a fast Log Processor and Forwarder                                                                                                                                                        tanzu-package-repo-global
  grafana.tanzu.vmware.com                            grafana                            Visualization and analytics software                                                                                                                                                                    tanzu-package-repo-global
  harbor.tanzu.vmware.com                             Harbor                             OCI Registry                                                                                                                                                                                            tanzu-package-repo-global
  multus-cni.tanzu.vmware.com                         multus-cni                         This package provides the ability for enabling attaching multiple network interfaces to pods in Kubernetes                                                                                              tanzu-package-repo-global
  prometheus.tanzu.vmware.com                         prometheus                         A time series database for your metrics                                                                                                                                                                 tanzu-package-repo-global
  addons-manager.tanzu.vmware.com                     tanzu-addons-manager               This package provides TKG addons lifecycle management capabilities.                                                                                                                                     tkg-system
  ako-operator.tanzu.vmware.com                       ako-operator                       NSX Advanced Load Balancer using ako-operator                                                                                                                                                           tkg-system
  antrea.tanzu.vmware.com                             antrea                             networking and network security solution for containers                                                                                                                                                 tkg-system
  calico.tanzu.vmware.com                             calico                             Networking and network security solution for containers.                                                                                                                                                tkg-system
  kapp-controller.tanzu.vmware.com                    kapp-controller                    Kubernetes package manager                                                                                                                                                                              tkg-system
  load-balancer-and-ingress-service.tanzu.vmware.com  load-balancer-and-ingress-service  Provides L4+L7 load balancing for TKG clusters running on vSphere                                                                                                                                       tkg-system
  metrics-server.tanzu.vmware.com                     metrics-server                     Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.                                                                             tkg-system
  pinniped.tanzu.vmware.com                           pinniped                           Pinniped provides identity services to Kubernetes.                                                                                                                                                      tkg-system
  vsphere-cpi.tanzu.vmware.com                        vsphere-cpi                        The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management. Cluster API Provider for vSphere is a concrete implementation of Cluster API for vSphere.  tkg-system
  vsphere-csi.tanzu.vmware.com                        vsphere-csi                        vSphere CSI provider                                                                                                                                                                                    tkg-system

The tanzu package command is welcome addition and doesn’t rely on separate manifest files like the extension framework did. The list of packages is maintained by VMware and updated as new versions or existing packages or new packages are added. To see what packages are currently installed you can run the tanzu package installed list command:

tanzu package installed list -A

/ Retrieving installed packages...


  NAME                               PACKAGE-NAME                     PACKAGE-VERSION       STATUS               NAMESPACE

  antrea                             antrea.tanzu.vmware.com                                Reconcile succeeded  tkg-system
  load-balancer-and-ingress-service  load-balancer-and-ingress-service.tanzu.vmware.com     Reconcile succeeded  tkg-system
  metrics-server                     metrics-server.tanzu.vmware.com                        Reconcile succeeded  tkg-system
  pinniped                           pinniped.tanzu.vmware.com                              Reconcile succeeded  tkg-system
  vsphere-cpi                        vsphere-cpi.tanzu.vmware.com                           Reconcile succeeded  tkg-system
  vsphere-csi                        vsphere-csi.tanzu.vmware.com                           Reconcile succeeded  tkg-system

These are all packages that were installed/upgraded during the upgrade to version 1.4 and all reside in the tkg-system namespace.

cert-manager

You should start with cert-manager as it is needed for most of the other extensions.

First check to see what version is available to install:

tanzu package available list cert-manager.tanzu.vmware.com -A

/ Retrieving package versions for cert-manager.tanzu.vmware.com...
  NAME                           VERSION               RELEASED-AT           NAMESPACE
  cert-manager.tanzu.vmware.com  1.1.0+vmware.1-tkg.2  2020-11-24T18:00:00Z  tanzu-package-repo-global

With the version available, the cert-manager package can be installed. cert-manager was not an app in 1.3, only a deployment, hence it not showing up in the kubectl get apps -A output earlier. The only thing we need to be sure of with the cert-manger package installation is to install the package to the same namespace where cert-manger is currently installed (cert-manager in this example).

tanzu package install cert-manager -p cert-manager.tanzu.vmware.com -n cert-manager -v 1.1.0+vmware.1-tkg.2

/ Installing package 'cert-manager.tanzu.vmware.com'
| Getting namespace 'cert-manager'
| Getting package metadata for 'cert-manager.tanzu.vmware.com'
| Creating service account 'cert-manager-cert-manager-sa'
| Creating cluster admin role 'cert-manager-cert-manager-cluster-role'
| Creating cluster role binding 'cert-manager-cert-manager-cluster-rolebinding'
- Creating package resource
| Package install status: Reconciling


 Added installed package 'cert-manager' in namespace 'cert-manager'

And that’s it for cert-manger…very easy. You can check the kubectl get apps -A output to see that cert-manager is now listed

kubectl get apps -A

NAMESPACE                        NAME                                DESCRIPTION           SINCE-DEPLOY   AGE
cert-manager                     cert-manager                        Reconcile succeeded   24s            9m22s
tanzu-system-ingress             contour                             Reconcile succeeded   57s            161d
tanzu-system-logging             fluent-bit                          Reconcile succeeded   57s            160d
tanzu-system-monitoring          grafana                             Reconcile succeeded   80s            160d
tanzu-system-monitoring          prometheus                          Reconcile succeeded   9s             160d
tanzu-system-registry            harbor                              Reconcile succeeded   98s            160d
tanzu-system-service-discovery   external-dns                        Reconcile succeeded   59s            161d
tkg-system                       antrea                              Reconcile succeeded   4m7s           161d
tkg-system                       load-balancer-and-ingress-service   Reconcile succeeded   1m8s           6m52s 
tkg-system                       metrics-server                      Reconcile succeeded   41s            161d
tkg-system                       pinniped                            Reconcile succeeded   4m46s          161d
tkg-system                       vsphere-cpi                         Reconcile succeeded   61s            161d
tkg-system                       vsphere-csi                         Reconcile succeeded   3m22s          161d

And the tanzu package installed list -A command shows it as well:

tanzu package installed list -A

- Retrieving installed packages...
  NAME                               PACKAGE-NAME                     PACKAGE-VERSION       STATUS               NAMESPACE
  cert-manager                       cert-manager.tanzu.vmware.com    1.1.0+vmware.1-tkg.2  Reconcile succeeded  cert-manager
  antrea                             antrea.tanzu.vmware.com                                Reconcile succeeded  tkg-system
  load-balancer-and-ingress-service  load-balancer-and-ingress-service.tanzu.vmware.com     Reconcile succeeded  tkg-system
  metrics-server                     metrics-server.tanzu.vmware.com                        Reconcile succeeded  tkg-system
  pinniped                           pinniped.tanzu.vmware.com                              Reconcile succeeded  tkg-system
  vsphere-cpi                        vsphere-cpi.tanzu.vmware.com                           Reconcile succeeded  tkg-system
  vsphere-csi                        vsphere-csi.tanzu.vmware.com                           Reconcile succeeded  tkg-system

The pods in the cert-manager namespace were recreated during this process.

For the rest of the extensions, the process is a little more complicated as you likely have made customizations to the default deployment (do you remember modifying those data-values.yaml files?). You’ll need to make sure to obtain a copy of the data-values secret for each extension prior to installing the new package as you will need to use the same customizaios.

Contour

You need to remove the Contour extension so that the legacy extension framework does not try to reconcile Contour while the package framework is also reconciling it. This will have no effect on the running Contour application.

kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/ingress/contour/contour-extension.yaml

extension.clusters.tmc.cloud.vmware.com "contour" deleted

You will need to get the available version first as was done for cert-manager.

tanzu package available list contour.tanzu.vmware.com -A

- Retrieving package versions for contour.tanzu.vmware.com...
  NAME                      VERSION                RELEASED-AT           NAMESPACE
  contour.tanzu.vmware.com  1.17.1+vmware.1-tkg.1  2021-07-23T18:00:00Z  tanzu-package-repo-global

You can run the following command to get the data-values secret for Contour:

kubectl get secret contour-data-values -n tanzu-system-ingress -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-contour-data-values.yaml

And in my case, this returned the following:

#@data/values
#@overlay/match-child-defaults missing_ok=True
---
infrastructure_provider: "vsphere"
contour:
  image:
    repository: projects.registry.vmware.com/tkg
envoy:
  image:
    repository: projects.registry.vmware.com/tkg
  service:
    type: "LoadBalancer"

The one change from the default that I made here was to set the service type to LoadBalancer instead of NodePort. With this information, I can create the needed configuration file for deploying the Contour package.

From the documentation, the default configuration file for deploying Contour with NSX Advanced Load Balancer in the mix is the following:

---
infrastructure_provider: vsphere
namespace: tanzu-system-ingress
contour:
 configFileContents: {}
 useProxyProtocol: false
 replicas: 2
 pspNames: "vmware-system-restricted"
 logLevel: info
envoy:
 service:
   type: LoadBalancer
   annotations: {}
   nodePorts:
     http: null
     https: null
   externalTrafficPolicy: Cluster
   disableWait: false
 hostPorts:
   enable: true
   http: 80
   https: 443
 hostNetwork: false
 terminationGracePeriodSeconds: 300
 logLevel: info
 pspNames: null
certificates:
 duration: 8760h
 renewBefore: 360h

You can see that the service type is already set to LoadBalancer so there is no need to make any changes. If you have installed the 1.3 extension to a different namespace than the default, you’ll want to update that in this manifest.

There is a slick command noted in the Harbor docs for pulling down the Harbor package files so that you can get a clean copy of the Harbor data-values file for editing. You can use a similar process for all packages if you prefer. For example, the following will pull down the Contour package files:

imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages contour.tanzu.vmware.com.1.17.1+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/contour-package-1.17.1

Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/contour@sha256:73dc13131e6c1cfa8d3b56aeacd97734447acdf1ab8c0862e936623ca744e7c4'
  Extracting layer 'sha256:93c1f3e88f0e0181e11a38a4e04ac16c21c5949622917b6c72682cc497ab3e44' (1/1)

Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update

Succeeded

You can then navigate to the /tmp/contour-package-1.17.1/config folder and make a copy of the values.yaml (contour-data-values.yaml) and edit it:

#@data/values

---
infrastructure_provider: vsphere
namespace: tanzu-system-ingress
contour:
  configFileContents: {}
  useProxyProtocol: false
  replicas: 2
  pspNames: "vmware-system-restricted"
  logLevel: info
envoy:
  service:
    type: null
    annotations: {}
    nodePorts:
      http: null
      https: null
    externalTrafficPolicy: Cluster
    aws:
      LBType: classic
    disableWait: false
  hostPorts:
    enable: true
    http: 80
    https: 443
  hostNetwork: false
  terminationGracePeriodSeconds: 300
  logLevel: info
  pspNames: null
certificates:
  duration: 8760h
  renewBefore: 360h

In my case, only a few changes are needed:

  • envoy.service.type: null needs to be envoy.service.type: LoadBalancer
  • envoy.service.aws and envoy.service.aws.LBType: classic need to be removed

The last thing to do with the data-values file is to remove any comments:

yq -i eval '... comments=""' contour-data-values.yaml

After creating a file with the needed contents, you can install the package:

tanzu package install contour -p contour.tanzu.vmware.com -v 1.17.1+vmware.1-tkg.1 -f contour-data-values.yaml -n tanzu-system-ingress

- Installing package 'contour.tanzu.vmware.com'
| Getting namespace 'tanzu-system-ingress'
\ Getting package metadata for 'contour.tanzu.vmware.com'
/ Creating service account 'contour-tanzu-system-ingress-sa'
| Creating cluster admin role 'contour-tanzu-system-ingress-cluster-role'
| Creating cluster role binding 'contour-tanzu-system-ingress-cluster-rolebinding'
| Creating secret 'contour-tanzu-system-ingress-values'
- Creating package resource
| Package install status: Reconciling

Added installed package 'contour' in namespace 'tanzu-system-ingress'

As with the cert-manager package, you’ll see all of the Contour pods getting deleted and recreated but Contour’s configuration and functionality should be unchanged after the upgrade.

Harbor

Harbor turned out to be one of the more involved items to upgrade to a package given then number of configuration changes that were made when I initially configured it.

As with Contour, you need to remove the Harbor extension so that the legacy extension framework does not try to reconcile Harbor while the package framework is also reconciling it.

kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/registry/harbor/harbor-extension.yaml

extension.clusters.tmc.cloud.vmware.com "harbor" deleted

Get the available version(s) of the Harbor package:

tanzu package available list harbor.tanzu.vmware.com -A

\ Retrieving package versions for harbor.tanzu.vmware.com...
  NAME                     VERSION               RELEASED-AT           NAMESPACE
  harbor.tanzu.vmware.com  2.2.3+vmware.1-tkg.1  2021-07-07T18:00:00Z  tanzu-package-repo-global

As with the other two extensions that have been upgraded to packages already, you need get a copy of the data-values secret so that it can be used when configuring the new package.

kubectl get secret harbor-data-values -n tanzu-system-registry -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-harbor-data-values.yaml

Per my earlier comment, you can see that there is a lot in this secret and a several items that have custom values:

tkg-1.3-harbor-data-values.yaml
#@data/values
#@overlay/match-child-defaults missing_ok=True

---
# Docker images setting
image:
  repository: projects.registry.vmware.com/tkg/harbor
  tag: v2.1.3_vmware.1
  pullPolicy: IfNotPresent

# The namespace to install Harbor
namespace: tanzu-system-registry

# The FQDN for accessing Harbor admin UI and Registry service.
hostname: harbor.corp.tanzu
# The network port of the Envoy service in Contour or other Ingress Controller.
port:
  https: 443

# [Optional] The certificate for the ingress if you want to use your own TLS certificate.
# We will issue the certificate by cert-manager when it's empty.
tlsCertificate:
  # [Required] the certificate
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
    ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
    MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
    MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
    EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
    S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
    BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
    z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
    lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
    FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
    H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
    uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
    F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
    H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
    JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
    Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
    +LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
    AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
    AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
    ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
    VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
    LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
    Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
    dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
    RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
    AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
    JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
    Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
    ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
    AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
    CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
    8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
    RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
    aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
    s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
    n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
    hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
    5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
    hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
    LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
    R22CcKK+kduUjv0X
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
    1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
    ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
    UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
    dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
    8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
    0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
    f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
    7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
    J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
    ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
    BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
    OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
    RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
    IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
    CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
    lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
    jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
    oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
    ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
    TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
    +QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
    aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
    ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
    XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
    +CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
    62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
    R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
    Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
    fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
    O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
    73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
    3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
    8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
    pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
    Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
    GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
    ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
    L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
    csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
    Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
    6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
    NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
    a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
    uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
    eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
    cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
    l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
    zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
    I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
    -----END PRIVATE KEY-----
  # [Optional] the certificate of CA, this enables the download
  # link on portal to download the certificate of CA
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
    MRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3JwMRkw
    FwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIwMDgxOTE3MjA0NFoXDTMwMDgx
    OTE3MzAzNVowSDEVMBMGCgmSJomT8ixkARkWBXRhbnp1MRQwEgYKCZImiZPyLGQB
    GRYEY29ycDEZMBcGA1UEAxMQQ09OVFJPTENFTlRFUi1DQTCCAiIwDQYJKoZIhvcN
    AQEBBQADggIPADCCAgoCggIBALKIdX7643PzvtVXlqNIwDuNq+rhcHF0fjR414j+
    1IGQUuXrykjhSDthPP+8BGN7mBgHT8AjAS1b95xc8B0S2Fhln3AoRE9z03GtfsBu
    FSBRUVwAifX6oXu97WzffhqPtxZfLJXbhOomjlkX6iffAs2TOLUx2Oj4w2vybhzj
    lcA70ai+0Sl6axSo3lMZ4KkuZ2WgfEcaDjjj33/pV3/bnFK+7ydPttc2Tek5xsI8
    XNMirIVxUiUT4YLy4WLiS200JUfbp1ZnMvnbQ8Jv1QnZl9W7WmBPcgxR4AAub0K4
    vZLXu6MXiboTlzkMB/YthCkTNlJcKkhHf60YR/T6Sx1T2nupyBa4deo5UGPzhRiJ
    pN37uqqAdK1qMDpCjARjS6U7Lf9JKjfiriLzLeyAjP8kaN4TdHSZd0pcQoZSxexQ
    9n+4E4MQm4EJ4DrVZCilsyL2BdETcHXKPc7q+Db4XM7jPKNG5GP1EMV4Xohv58yZ
    /rRfmK64gar8AMnOKT2AP681qdZs7lljONcXUALzlX5TqIchYT0DVQmFLYoMBeZz
    0l21QjbK0YWnPza6Yi/N4m6rFbEB4WXiqhYSkxzrMvocVUgd4AAP1vfHNnFEsnUR
    nSsiglFH/xlyO3cBFrmoZAxbA2091XHWhB4c0mQEI3hOqAB8UoFGBrQpmQ+LesoC
    1LZ9AgMBAAGjUTBPMAsGA1UdDwQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
    DgQWBBTFAxSvY64Q5adhm8IYecHBAUuobzAQBgkrBgEEAYI3FQEEAwIBADANBgkq
    hkiG9w0BAQsFAAOCAgEAjg/v4mIP7gBVCw4pemtGn3PStDh/aB9vbWyjAyxSNaaH
    H0nID5q5wow9ueBiDfjTPnhbf3P768HG8oL/+9C+Vm/0liFBd+0/DaayKpANFMLB
    BV+s2adWRhQucLQfXPwum8RybWv82wkRkWCCdOBaAvAMuTgk08SwJIyQfVgpk3nY
    0OwjFwSAadvevf+LoD/9L8R9NEt/n4WJe+LtEamo9EVb+l+cYqyxyubAVY0Y6BM2
    GXqAh3FEW2aQMpwouh/5S7w5oSMYN6miY1ojki8gPm0+4+CILPWh/fr2q0O/bPtb
    Tr++nPMmZ8ov9epNGIuqhtk5ja2/JuY+RW46IRc8QpF1EyUae02E6U2Vacs7Gge2
    CeSINkoLFFmiKBfIn/HAchlme9aL6DlJ9wAreBDH3E8kH7gRDWbSK2/QD0Hqac+E
    geGHwpg/8OtBOHUMnM7eLOXBJFcJosWf0XnEgS4ubgaHgqDEu8p8PE7rpCxtUNur
    t+x2xONI/rBWgdbp51lPr7o819zPJCvYZq1Pp1st8fb3RlUSWvbQMPFtGAyaBy+G
    0RgZ9WPtyEYgnHAb5/Dq46sne9/QnPwwGpjv1s1oE3ZFQjhvnGis8+dqRxk3YZAk
    yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
    -----END CERTIFICATE-----
# Use contour http proxy instead of the ingress when it's true
enableContourHttpProxy: true

# [Required] The initial password of Harbor admin.
harborAdminPassword: VMware1!

# [Required] The secret key used for encryption. Must be a string of 16 chars.
secretKey: VMware1!VMware1!

database:
  # [Required] The initial password of the postgres database.
  password: VMware1!VMware1!

core:
  replicas: 1
  # [Required] Secret is used when core server communicates with other components.
  secret: VMware1!VMware1!
  # [Required] The XSRF key. Must be a string of 32 chars.
  xsrfKey: VMware1!VMware1!VMware1!VMware1!
jobservice:
  replicas: 1
  # [Required] Secret is used when job service communicates with other components.
  secret: VMware1!VMware1!
registry:
  replicas: 1
  # [Required] Secret is used to secure the upload state from client
  # and registry storage backend.
  # See: https://github.com/docker/distribution/blob/master/docs/configuration.md#http
  secret: VMware1!
notary:
  # Whether to install Notary
  enabled: true
clair:
  # Whether to install Clair scanner
  enabled: true
  replicas: 1
  # The interval of clair updaters, the unit is hour, set to 0 to
  # disable the updaters
  updatersInterval: 12
trivy:
  # enabled the flag to enable Trivy scanner
  enabled: true
  replicas: 1
  # gitHubToken the GitHub access token to download Trivy DB
  gitHubToken: ""
  # skipUpdate the flag to disable Trivy DB downloads from GitHub
  #
  # You might want to set the value of this flag to `true` in test or CI/CD environments to avoid GitHub rate limiting issues.
  # If the value is set to `true` you have to manually download the `trivy.db` file and mount it in the
  # `/home/scanner/.cache/trivy/db/trivy.db` path.
  skipUpdate: false

# The persistence is always enabled and a default StorageClass
# is needed in the k8s cluster to provision volumes dynamicly.
# Specify another StorageClass in the "storageClass" or set "existingClaim"
# if you have already existing persistent volumes to use
#
# For storing images and charts, you can also use "azure", "gcs", "s3",
# "swift" or "oss". Set it in the "imageChartStorage" section
persistence:
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound,
      # and specify the "subPath" if the PVC is shared with other components
      existingClaim: ""
      # Specify the "storageClass" used to provision the volume. Or the default
      # StorageClass will be used(the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi
    jobservice:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    database:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    redis:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    trivy:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
  # Define which storage backend is used for registry and chartmuseum to store
  # images and charts. Refer to
  # https://github.com/docker/distribution/blob/master/docs/configuration.md#storage
  # for the detail.
  imageChartStorage:
    # Specify whether to disable `redirect` for images and chart storage, for
    # backends which not supported it (such as using minio for `s3` storage type), please disable
    # it. To disable redirects, simply set `disableredirect` to `true` instead.
    # Refer to
    # https://github.com/docker/distribution/blob/master/docs/configuration.md#redirect
    # for the detail.
    disableredirect: false
    # Specify the "caBundleSecretName" if the storage service uses a self-signed certificate.
    # The secret must contain keys named "ca.crt" which will be injected into the trust store
    # of registry's and chartmuseum's containers.
    # caBundleSecretName:

    # Specify the type of storage: "filesystem", "azure", "gcs", "s3", "swift",
    # "oss" and fill the information needed in the corresponding section. The type
    # must be "filesystem" if you want to use persistent volumes for registry
    # and chartmuseum
    type: filesystem
    filesystem:
      rootdirectory: /storage
      #maxthreads: 100
    azure:
      accountname: accountname # required
      accountkey: base64encodedaccountkey # required
      container: containername # required
      realm: core.windows.net # optional
    gcs:
      bucket: bucketname # required
      # The base64 encoded json file which contains the key
      encodedkey: base64-encoded-json-key-file # optional
      rootdirectory: null # optional
      chunksize: 5242880 # optional
    s3:
      region: us-west-1 # required
      bucket: bucketname # required
      accesskey: null # eg, awsaccesskey
      secretkey: null # eg, awssecretkey
      regionendpoint: null # optional, eg, http://myobjects.local
      encrypt: false # optional
      keyid: null # eg, mykeyid
      secure: true # optional
      v4auth: true # optional
      chunksize: null # optional
      rootdirectory: null # optional
      storageclass: STANDARD # optional
    swift:
      authurl: https://storage.myprovider.com/v3/auth
      username: username
      password: password
      container: containername
      region: null # eg, fr
      tenant: null # eg, tenantname
      tenantid: null # eg, tenantid
      domain: null # eg, domainname
      domainid: null # eg, domainid
      trustid: null # eg, trustid
      insecureskipverify: null # bool eg, false
      chunksize: null # eg, 5M
      prefix: null # eg
      secretkey: null # eg, secretkey
      accesskey: null # eg, accesskey
      authversion: null # eg, 3
      endpointtype: null # eg, public
      tempurlcontainerkey: null # eg, false
      tempurlmethods: null # eg
    oss:
      accesskeyid: accesskeyid
      accesskeysecret: accesskeysecret
      region: regionname
      bucket: bucketname
      endpoint: null # eg, endpoint
      internal: null # eg, false
      encrypt: null # eg, false
      secure: null # eg, true
      chunksize: null # eg, 10M
      rootdirectory: null # eg, rootdirectory

# The http/https network proxy for clair, core, jobservice, trivy
proxy:
  httpProxy:
  httpsProxy:
  noProxy: 127.0.0.1,localhost,.local,.internal

You can use the imgpkg command to pull down the default configuration information needed for deploying the Harbor package. In this example, I’m using the version information obtained earlier to pull the correct version and to create a directory under /tmp to store it.

imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages harbor.tanzu.vmware.com.2.2.3+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/harbor-package-2.2.3

Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/harbor@sha256:c4fe69a762c3081e2f181dead7d79224b7012db8b74e3ca847798f030f1749ee'
  Extracting layer 'sha256:daca49ca6a24c1660418741b7c5dbb786cfadca9b25a6161e85ac7713d931fcc' (1/1)

Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update

Succeeded

Now you need to make a copy of the /tmp/harbor-package-2.2.3/config/values.yaml file (harbor-data-values.yaml) and update it with the information from the tkg-1.3-harbor-data-values.yaml file that was created from the Harbor data-values secret earlier. Your installation will likely differ from mine but the following are the items I had to set:

  • hostname
  • tlsCertificate.tls.crt
  • tlsCertificate.tls.key
  • tlsCertificate.ca.crt
  • harborAdminPassword
  • secretKey
  • database.password
  • core.secret
  • core.xsrfKey
  • jobservice.secret
  • registry.secret
  • persistence.persistentVolumeClaim.registry.size

There are numerous other items that can be configured so please be sure to read the Harbor package documentation to make sure you’re not missing anything.

The last thing to do with the data-values file is to remove any comments:

yq -i eval '... comments=""' harbor-data-values.yaml

Once you’re satisfied with your data-values file, you can proceed with deploying the Harbor package:

tanzu package install harbor -p harbor.tanzu.vmware.com -v 2.2.3+vmware.1-tkg.1 -f harbor-data-values.yaml -n tanzu-system-registry

- Installing package 'harbor.tanzu.vmware.com'
| Getting namespace 'tanzu-system-registry'
| Getting package metadata for 'harbor.tanzu.vmware.com'
| Creating service account 'harbor-tanzu-system-registry-sa'
| Creating cluster admin role 'harbor-tanzu-system-registry-cluster-role'
| Creating cluster role binding 'harbor-tanzu-system-registry-cluster-rolebinding'
| Creating secret 'harbor-tanzu-system-registry-values'
- Creating package resource
| Package install status: Reconciling

Please consider using 'tanzu package installed update' to update the installed package with correct settings


Error: timed out waiting for the condition

It took a bit longer than the tanzu package install command was willing to wait the first time around and the command failed, even though the reconciliation was still going on in the background. I was able to check this by running the tanzu package installed get command:

tanzu package installed get harbor -n tanzu-system-registry

| Retrieving installation details for harbor...
NAME:                    harbor
PACKAGE-NAME:            harbor.tanzu.vmware.com
PACKAGE-VERSION:         2.2.3+vmware.1-tkg.1
STATUS:                  Reconciling
CONDITIONS:              [{Reconciling True  }]

And by looking at the events in the tanzu-system-registry namespace (kubectl -n tanzu-system-registry get events -w) I was able to see that there was still a lot of activity with creating pods and attaching persistent volumes. Once this was all done, the reconciliation of the Harbor package completed successfully.

Another thing to look out for since I saw it as well…as of the time of this writing, I could not get Harbor to completely reconcile with Notary enabled. The status of the app and package were both “reconciling” and harbor-notary-signer pod was in a CrashLoopBackOff state. If you take a look at the logs for this pod, you’ll see entries similar to:

{"level":"fatal","msg":"Could not read config at :/etc/notary/server-config.postgres.json, viper error: open : no such file or directory","time":"2021-09-11T16:14:43Z"}

If you encounter the same thing and can live without Notary, you can update the Harbor data-values file and set notary.enabled to false and then update the package via the following command:

tanzu package installed update harbor -p harbor.tanzu.vmware.com -v 2.2.3+vmware.1-tkg.1 -f harbor-data-values.yaml -n tanzu-system-registry

And lastly regarding Notary, one of my coworkers came up with this overlay that will allow you to run Notary successfully in TKG 1.4 (until this issue is permanently fixed):

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.and_op(overlay.subset({"kind": "Deployment"}), overlay.subset({"metadata": {"name": "harbor-notary-signer"}}))
---
spec:
  template:
    spec:
      containers:
        #@overlay/match by="name",expects="0+"
        - name: notary-signer
          image: projects.registry.vmware.com/tkg/harbor/notary-signer-photon@sha256:4dfbf3777c26c615acfb466b98033c0406766692e9c32f3bb08873a0295e24d1

You can save this overlay as overlay-notary-signer-image-fix.yaml. You’ll need to create a secret with this overlay and then update the installed package, per the following two commands:

kubectl -n tanzu-system-registry create secret generic harbor-notary-singer-image-overlay -o yaml --dry-run=client --from-file=overlay-notary-signer-image-fix.yaml | kubectl apply -f -
kubectl -n tanzu-system-registry annotate packageinstalls harbor ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=harbor-notary-singer-image-overlay

You’ll see that the Notary pod is running fine with this workaround in place:

kubectl -n tanzu-system-registry get po

NAME                                    READY   STATUS    RESTARTS   AGE
harbor-core-6c66bc897b-9srkh            1/1     Running   0          3m12s
harbor-database-0                       1/1     Running   0          17h
harbor-jobservice-599f69b686-wdvqx      1/1     Running   0          17h
harbor-notary-server-cc8b5f5c9-j9tkv    1/1     Running   0          3m12s
harbor-notary-signer-7457dd5946-679m6   1/1     Running   0          3m12s
harbor-portal-5c9bf895d7-fvd4r          1/1     Running   0          17h
harbor-redis-0                          1/1     Running   0          17h
harbor-registry-697cb78c56-9mrw7        2/2     Running   0          17h
harbor-trivy-0                          1/1     Running   0          17h

Fluent-Bit

As with the previous extensions, you need to remove the Fluent-Bit extension so that the legacy extension framework does not try to reconcile Fluent-Bit while the package framework is also reconciling it. Since Fluent-Bit has no stateful data, you can actually remove it entirely and then reinstall it in the package format. You will need to get the contents of the current data-values secret first though.

kubectl get secret fluent-bit-data-values -n tanzu-system-logging -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-fluent-bit-data-values.yaml

And in my case, this returned the following (syslog configuration, sending data to vRealize Log Insight):

#@data/values
#@overlay/match-child-defaults missing_ok=True
---
logging:
  image:
    repository: projects.registry.vmware.com/tkg
tkg:
  instance_name: "tkg-mgmt"
  cluster_name: "tkg-wld"
fluent_bit:
  output_plugin: "syslog"
  syslog:
    host: "vrli-01a.corp.tanzu"
    port: "514"
    mode: "udp"
    format: "rfc5424"

Removing the extension is fairly simple:

kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/logging/fluent-bit/fluent-bit-extension.yaml

extension.clusters.tmc.cloud.vmware.com "fluent-bit" deleted

kubectl delete app fluent-bit -n tanzu-system-logging

app.kappctrl.k14s.io "fluent-bit" deleted

kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/logging/fluent-bit/namespace-role.yaml

namespace "tanzu-system-logging" deleted
serviceaccount "fluent-bit-extension-sa" deleted
role.rbac.authorization.k8s.io "fluent-bit-extension-role" deleted
rolebinding.rbac.authorization.k8s.io "fluent-bit-extension-rolebinding" deleted
clusterrole.rbac.authorization.k8s.io "fluent-bit-extension-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "fluent-bit-extension-cluster-rolebinding" deleted

You will need to get the available Fluent-Bit version as was done for the previous packages.

tanzu package available list fluent-bit.tanzu.vmware.com -A

- Retrieving package versions for fluent-bit.tanzu.vmware.com...
  NAME                         VERSION               RELEASED-AT           NAMESPACE
  fluent-bit.tanzu.vmware.com  1.7.5+vmware.1-tkg.1  2021-05-13T18:00:00Z  tanzu-package-repo-global

You can use the imgpkg command to pull down the default configuration information needed for deploying the Fluent-Bit package. In this example, I’m using the version information obtained earlier to pull the correct version and to create a directory under /tmp to store it.

imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages fluent-bit.tanzu.vmware.com.1.7.5+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/fluent-bit-package-1.7.5

Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/fluent-bit@sha256:c83d038c57f244aae2819dd77dc5184bb3e1ec96524d3f6a09fe8a244b7bc9e4'
  Extracting layer 'sha256:48456e4452a35786ed148a3f1a9ede1427f8fccb0079a0eb0904820211b6cebd' (1/1)

Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update

Succeeded

You can then navigate to the /tmp/fluent-bit-package-1.7.5/config folder and make a copy of the values.yaml file (fluent-bit-data-values.yaml) and edit it. The following is what this file looks like by default:

fluent-bit-data-values.yaml
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
namespace: "tanzu-system-logging"
#! Required params for supported output plugins
fluent_bit:
  config:
    #! https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/variables
    service: |
      [Service]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    outputs: |
      [OUTPUT]
        Name              stdout
        Match             *
    inputs: |
      [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser docker
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

      [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On
    filters: |
      [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

      [FILTER]
        Name                modify
        Match               *
        Rename message text
    parsers: |
      [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    streams: ""
    plugins: ""
  #! optional configuration for the daemonset
  daemonset:
    resources: { }
    #! limits:
    #!   cpu: 100m
    #!   memory: 128Mi
    #! requests:
    #!   cpu: 100m
    #!   memory: 128Mi
    podAnnotations: { }
    podLabels: { }

Unfortunately, this default file will result in a fluent-bit implementation that doesn’t do very much. I did a comparison between what gets deployed in the TKG 1.3 Fluent-Bit pods and the TKG 1.4 Fluent-Bit pods and came up with the following changes to the default config file:

  • Add the following to the outputs: section (this was very specific to my configuration, you will need to update this based on your specific output type):
      [OUTPUT]
        Name   syslog
        Match  kube.*
        Host   vrli-01a.corp.tanzu
        Port   514
        Mode   udp
        Syslog_Format        rfc5424
        Syslog_Hostname_key  tkg_cluster
        Syslog_Appname_key   pod_name
        Syslog_Procid_key    container_name
        Syslog_Message_key   message
        Syslog_SD_key        k8s
        Syslog_SD_key        labels
        Syslog_SD_key        annotations
        Syslog_SD_key        tkg
    
      [OUTPUT]
        Name   syslog
        Match  kube_systemd.*
        Host   vrli-01a.corp.tanzu
        Port   514
        Mode   udp
        Syslog_Format        rfc5424
        Syslog_Hostname_key  tkg_cluster
        Syslog_Appname_key   tkg_instance
        Syslog_Message_key   MESSAGE
        Syslog_SD_key        systemd
  • Remove the following from the filters: section:
      [FILTER]
        Name                modify
        Match               *
        Rename message text
  • Add the following to the filters: section:
      [FILTER]
        Name                record_modifier
        Match               *
        Record tkg_cluster tkg-wld
        Record tkg_instance tkg-mgmt
    	
      [FILTER]
        Name                  nest
        Match                 kube.*
        Operation             nest
        Wildcard              tkg_instance*
        Nest_Under            tkg
    
      [FILTER]
        Name                  nest
        Match                 kube_systemd.*
        Operation             nest
        Wildcard              SYSTEMD*
        Nest_Under            systemd
    
      [FILTER]
        Name                  modify
        Match                 kube.*
        Copy                  kubernetes k8s
    
      [FILTER]
        Name                  nest
        Match                 kube.*
        Operation             lift
        Nested_Under          kubernetes
  • Remove the following from the inputs: section:
      [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On
  • Add the following to the inputs: section:
      [INPUT]
        Name              tail
        Tag               audit.*
        Path              /var/log/audit/audit.log
        Parser            logfmt
        DB                /var/log/flb_system_audit.db
        Mem_Buf_Limit     50MB
        Refresh_Interval  10
        Skip_Long_Lines   On
    
      [INPUT]
        Name                systemd
        Tag                 kube_systemd.*
        Path                /var/log/journal
        DB                  /var/log/flb_kube_systemd.db
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        Systemd_Filter      _SYSTEMD_UNIT=containerd.service
        Read_From_Tail      On
        Strip_Underscores   On
    
      [INPUT]
        Name              tail
        Tag               apiserver_audit.*
        Path              /var/log/kubernetes/audit.log
        Parser            json
        DB                /var/log/flb_kube_audit.db
        Mem_Buf_Limit     50MB
        Refresh_Interval  10
        Skip_Long_Lines   On	
  • Add the following to the parsers: section:
      [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    
      [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    
      [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
    
      [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    
      [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
    
      [PARSER]
        Name        docker-daemon
        Format      regex
        Regex       time="(?<time>[^ ]*)" level=(?<level>[^ ]*) msg="(?<msg>[^ ].*)"
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
    
      [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    
      [PARSER]
        Name        logfmt
        Format      logfmt
    
      [PARSER]
        Name        syslog-rfc5424
        Format      regex
        Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
    
      [PARSER]
        Name        syslog-rfc3164-local
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
        Time_Keep   On
    
      [PARSER]
        Name        syslog-rfc3164
        Format      regex
        Regex       /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
        Time_Key    time
        Time_Format %b %d %H:%M:%S
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
    
      [PARSER]
        Name    kube-custom
        Format  regex
        Regex   (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$

The last thing to do with the data-values file is to remove any comments:

yq -i eval '... comments=""' fluent-bit-data-values.yaml

My completed fluent-bit-data-values.yaml file looks like the following:

fluent-bit-data-values.yaml
namespace: "tanzu-system-logging"
fluent_bit:
  config:
    service: |
      [Service]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    outputs: |
      [OUTPUT]
        Name              stdout
        Match             *

      [OUTPUT]
        Name   syslog
        Match  kube.*
        Host   vrli-01a.corp.tanzu
        Port   514
        Mode   udp
        Syslog_Format        rfc5424
        Syslog_Hostname_key  tkg_cluster
        Syslog_Appname_key   pod_name
        Syslog_Procid_key    container_name
        Syslog_Message_key   message
        Syslog_SD_key        k8s
        Syslog_SD_key        labels
        Syslog_SD_key        annotations
        Syslog_SD_key        tkg

      [OUTPUT]
        Name   syslog
        Match  kube_systemd.*
        Host   vrli-01a.corp.tanzu
        Port   514
        Mode   udp
        Syslog_Format        rfc5424
        Syslog_Hostname_key  tkg_cluster
        Syslog_Appname_key   tkg_instance
        Syslog_Message_key   MESSAGE
        Syslog_SD_key        systemd

    inputs: |
      [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser docker
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

      [INPUT]
        Name              tail
        Tag               audit.*
        Path              /var/log/audit/audit.log
        Parser            logfmt
        DB                /var/log/flb_system_audit.db
        Mem_Buf_Limit     50MB
        Refresh_Interval  10
        Skip_Long_Lines   On

      [INPUT]
        Name                systemd
        Tag                 kube_systemd.*
        Path                /var/log/journal
        DB                  /var/log/flb_kube_systemd.db
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        Systemd_Filter      _SYSTEMD_UNIT=containerd.service
        Read_From_Tail      On
        Strip_Underscores   On

      [INPUT]
        Name              tail
        Tag               apiserver_audit.*
        Path              /var/log/kubernetes/audit.log
        Parser            json
        DB                /var/log/flb_kube_audit.db
        Mem_Buf_Limit     50MB
        Refresh_Interval  10
        Skip_Long_Lines   On

    filters: |
      [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

      [FILTER]
        Name                record_modifier
        Match               *
        Record tkg_cluster tkg-wld
        Record tkg_instance tkg-mgmt

      [FILTER]
        Name                  nest
        Match                 kube.*
        Operation             nest
        Wildcard              tkg_instance*
        Nest_Under            tkg

      [FILTER]
        Name                  nest
        Match                 kube_systemd.*
        Operation             nest
        Wildcard              SYSTEMD*
        Nest_Under            systemd

      [FILTER]
        Name                  modify
        Match                 kube.*
        Copy                  kubernetes k8s

      [FILTER]
        Name                  nest
        Match                 kube.*
        Operation             lift
        Nested_Under          kubernetes

    parsers: |
      [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
        Name   apache
        Format regex
        Regex  ^(?[^ ]*) [^ ]* (?[^ ]*) \[(?


After creating a file with the needed contents, you can install the package:

tanzu package install fluent-bit -p fluent-bit.tanzu.vmware.com -n tanzu-system-logging -v 1.7.5+vmware.1-tkg.1 -f fluent-bit-data-values.yaml --create-namespace

- Installing package 'fluent-bit.tanzu.vmware.com'
| Creating namespace 'tanzu-system-logging'
| Getting package metadata for 'fluent-bit.tanzu.vmware.com'
| Creating service account 'fluent-bit-tanzu-system-logging-sa'
| Creating cluster admin role 'fluent-bit-tanzu-system-logging-cluster-role'
| Creating cluster role binding 'fluent-bit-tanzu-system-logging-cluster-rolebinding'
| Creating secret 'fluent-bit-tanzu-system-logging-values'
\ Creating package resource
| Package install status: Reconciling


 Added installed package 'fluent-bit' in namespace 'tanzu-system-logging'

External DNS

As with Fluent-Bit, External DNS is also stateless so it can be completely removed prior to installing the new package but you need to get a copy of the data-values secret first.

kubectl get secret external-dns-data-values -n tanzu-system-service-discovery -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-external-dns-data-values.yaml

The following were the contents of this secret:

#@data/values
#@overlay/match-child-defaults missing_ok=True
---
externalDns:
  image:
    repository: projects.registry.vmware.com/tkg
  deployment:
    #@overlay/replace
    args:
    - --txt-owner-id=k8s
    - --provider=rfc2136
    - --rfc2136-host=192.168.110.10
    - --rfc2136-port=53
    - --rfc2136-zone=corp.tanzu
    - --rfc2136-insecure
    - --rfc2136-tsig-axfr
    - --source=service
    - --source=contour-httpproxy
    - --domain-filter=corp.tanzu

As with Fluent-Bit, removing the External DNS extension is three very similar commands:

kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/service-discovery/external-dns/external-dns-extension.yaml

extension.clusters.tmc.cloud.vmware.com "external-dns" deleted

kubectl -n tanzu-system-service-discovery delete app external-dns

app.kappctrl.k14s.io "external-dns" deleted

kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/service-discovery/external-dns/namespace-role.yaml

namespace "tanzu-system-service-discovery" deleted
serviceaccount "external-dns-extension-sa" deleted
role.rbac.authorization.k8s.io "external-dns-extension-role" deleted
rolebinding.rbac.authorization.k8s.io "tanzu-system-service-discovery-rolebinding" deleted
clusterrole.rbac.authorization.k8s.io "external-dns-extension-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "external-dns-extension-cluster-rolebinding" deleted

Don’t worry, none of your DNS records created by External DNS will be affected by removing the extension.

Get the available version(s) of the External DNS package:

tanzu package available list external-dns.tanzu.vmware.com -A

- Retrieving package versions for external-dns.tanzu.vmware.com...
  NAME                           VERSION               RELEASED-AT           NAMESPACE
  external-dns.tanzu.vmware.com  0.8.0+vmware.1-tkg.1  2021-06-11T18:00:00Z  tanzu-package-repo-global

Use imgpkg to pull down a default data-values file:

imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages external-dns.tanzu.vmware.com.0.8.0+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/external-dns-package-0.8.0

Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/external-dns@sha256:ae8cd950bbeb94b8ff66e41eaf62bcc4470f1f1ab424032a996a0193f5245961'
  Extracting layer 'sha256:5227e0d571e7a5c02af43260bac2c783f02f1938e67070784a0e8870b0c80255' (1/1)

Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update

Succeeded

Which returns the following (unhelpful) values.yaml file:

#@data/values
---

#! The namespace in which to deploy ExternalDNS.
namespace: tanzu-system-service-discovery

#! Deployment related configuration
deployment:
  #! (REQUIRED) Args passed via command-line to external-dns
  #! For more guidance on configuration options for your desired DNS provider, consult the
  #! ExternalDNS docs at https://github.com/kubernetes-sigs/external-dns#running-externaldns
  args: []
  #! Environment variables to pass to external-dns
  env: []
  #! Security context of the external-dns container
  securityContext: {}
  #! Volume mounts of the external-dns container
  volumeMounts: []
  #! Volume of the external-dns pod
  volumes: []

In this case, you’ll be a lot better off using the various example files noted in the External DNS documentation.

As you would have seen in the original External DNS data-values secret, I’m using rfc2136 DNS, specifically Active Directory DNS. Our existing instructions for External DNS are very bind-specific when it comes to rfc2136 but there are instructions available for Active Directory DNS at Secure Updates Using RFC3645 (GSS-TSIG). I had actually tried to use these steps in the What I wanted to do section of my previous post, How to configure external-dns with Microsoft DNS in TKG 1.3 (plus Harbor and Contour). Due to a now-resolved issue, it didn’t work out but with the newer (0.8.0) version of External DNS, it did.

A configmap is needed that defines the kerberos configuration that External DNS will use…this is fairly generic and the only things I customized were the domain/realm name and the kdc/admin_server addresses:

apiVersion: v1
kind: ConfigMap
metadata:
  name: krb5.conf
  namespace: tanzu-system-service-discovery
data:
  krb5.conf: |
    [logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

    [libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
    default_ccache_name = KEYRING:persistent:%{uid}

    default_realm = CORP.TANZU

    [realms]
    CORP.TANZU = {
      kdc = controlcenter.corp.tanzu
      admin_server = controlcenter.corp.tanzu
    }

    [domain_realm]
    corp.tanzu = CORP.TANZU
    .corp.tanzu = CORP.TANZU

Now you can re-create the tanzu-system-service-discovery namespace and create the noted configmap:

kubectl create ns tanzu-system-service-discovery

namespace/tanzu-system-service-discovery created
kubectl apply -f external-dns-krb5-cm.yaml

configmap/krb5.conf created

And the External DNS data-values file looks like the following:

namespace: tanzu-system-service-discovery
deployment:
  args:
    - --provider=rfc2136
    - --rfc2136-host=controlcenter.corp.tanzu
    - --rfc2136-port=53
    - --rfc2136-zone=corp.tanzu
    - --rfc2136-gss-tsig
    - --rfc2136-kerberos-realm=corp.tanzu
    - --rfc2136-kerberos-username=administrator
    - --rfc2136-kerberos-password=VMware1!
    - --rfc2136-tsig-axfr
    - --source=service
    - --source=ingress
    - --source=contour-httpproxy
    - --domain-filter=corp.tanzu
    - --txt-owner-id=k8s
    - --txt-prefix=external-dns-
    - --registry=txt
    - --policy=upsert-only
  env: []
  securityContext: {}
  volumeMounts:
  - name: kerberos-config-volume
    mountPath: /etc/krb5.conf
    subPath: krb5.conf
  volumes:
  - name: kerberos-config-volume
    configMap:
      defaultMode: 420
      name: krb5.conf

You can see from this file that the /etc/krb5.conf file in the external-dns pod will be created from krb5.conf section of the configmap created earlier.

Now you should be ready to deploy the External DNS package:

tanzu package install external-dns -p external-dns.tanzu.vmware.com -n tanzu-system-service-discovery -v 0.8.0+vmware.1-tkg.1 -f external-dns-data-values.yaml

| Installing package 'external-dns.tanzu.vmware.com'
- Creating namespace 'tanzu-system-service-discovery'
/ Getting package metadata for 'external-dns.tanzu.vmware.com'
| Creating service account 'external-dns-tanzu-system-service-discovery-sa'
| Creating cluster admin role 'external-dns-tanzu-system-service-discovery-cluster-role'
| Creating cluster role binding 'external-dns-tanzu-system-service-discovery-cluster-rolebinding'
| Creating secret 'external-dns-tanzu-system-service-discovery-values'
- Creating package resource
- Package install status: Reconciling


 Added installed package 'external-dns' in namespace 'tanzu-system-service-discovery'

To validate that External DNS was working, I redeployed Harbor while External DNS was uninstalled. The installation was successful but no DNS record was created, as expected. After the External DNS package was installed, I was able to see the following in the logs for the External DNS pod:

time="2021-09-12T20:45:24Z" level=info msg="Instantiating new Kubernetes client"
time="2021-09-12T20:45:24Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-09-12T20:45:24Z" level=info msg="Created Kubernetes client https://100.64.0.1:443"
time="2021-09-12T20:45:26Z" level=info msg="Created Dynamic Kubernetes client https://100.64.0.1:443"
time="2021-09-12T20:45:27Z" level=info msg="Configured RFC2136 with zone 'corp.tanzu.' and nameserver 'controlcenter.corp.tanzu:53'"
time="2021-09-12T20:45:32Z" level=info msg="Adding RR: harbor.corp.tanzu 0 A 192.168.220.2"
time="2021-09-12T20:45:32Z" level=info msg="Adding RR: external-dns-harbor.corp.tanzu 0 TXT \"heritage=external-dns,external-dns/owner=k8s,external-dns/resource=HTTPProxy/tanzu-system-registry/harbor-httpproxy\""

Additionally, I was able to set the dynamic update policy in Active Directory back to Secure only and everything still worked as expected.

Prometheus

As with previous packages, you need to remove the Prometheus extension so that the legacy extension framework does not try to reconcile Prometheus while the package framework is also reconciling it.

kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/monitoring/prometheus/prometheus-extension.yaml

extension.clusters.tmc.cloud.vmware.com "prometheus" deleted

Get the available version(s) of the Prometheus package:

tanzu package available list prometheus.tanzu.vmware.com -A

- Retrieving package versions for prometheus.tanzu.vmware.com...
  NAME                         VERSION                RELEASED-AT           NAMESPACE
  prometheus.tanzu.vmware.com  2.27.0+vmware.1-tkg.1  2021-05-12T18:00:00Z  tanzu-package-repo-global

As with the other extensions that have been upgraded to packages already, you need get a copy of the data-values secret so that it can be used when configuring the new package.

kubectl get secret prometheus-data-values -n tanzu-system-monitoring -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-prometheus-data-values.yaml

The only things that have been customized in here are the hostname and certificate data (I’m using a wildcard certificate) and ingress has been enabled.

tkg-1.3-prometheus-data-values.yaml
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
monitoring:
  ingress:
    enabled: true
    virtual_host_fqdn: "prometheus.corp.tanzu"
    prometheus_prefix: "/"
    alertmanager_prefix: "/alertmanager/"
    tlsCertificate:
      tls.crt: |
        -----BEGIN CERTIFICATE-----
        MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
        ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
        MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
        MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
        EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
        S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
        BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
        z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
        lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
        FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
        H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
        uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
        F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
        H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
        JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
        Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
        +LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
        AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
        AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
        ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
        VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
        LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
        Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
        dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
        RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
        AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
        JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
        Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
        ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
        AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
        CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
        8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
        RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
        aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
        s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
        n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
        hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
        5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
        hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
        LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
        R22CcKK+kduUjv0X
        -----END CERTIFICATE-----
      tls.key: |
        -----BEGIN PRIVATE KEY-----
        MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
        1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
        ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
        UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
        dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
        8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
        0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
        f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
        7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
        J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
        ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
        BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
        OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
        RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
        IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
        CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
        lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
        jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
        oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
        ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
        TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
        +QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
        aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
        ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
        XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
        +CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
        62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
        R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
        Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
        fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
        O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
        73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
        3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
        8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
        pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
        Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
        GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
        ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
        L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
        csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
        Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
        6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
        NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
        a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
        uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
        eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
        cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
        l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
        zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
        I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
        -----END PRIVATE KEY-----
  prometheus_server:
    image:
      repository: projects.registry.vmware.com/tkg/prometheus
  alertmanager:
    image:
      repository: projects.registry.vmware.com/tkg/prometheus
  kube_state_metrics:
    image:
      repository: projects.registry.vmware.com/tkg/prometheus
  node_exporter:
    image:
      repository: projects.registry.vmware.com/tkg/prometheus
  pushgateway:
    image:
      repository: projects.registry.vmware.com/tkg/prometheus
  cadvisor:
    image:
      repository: projects.registry.vmware.com/tkg/prometheus
  prometheus_server_configmap_reload:
    image:
      repository: projects.registry.vmware.com/tkg/prometheus
  prometheus_server_init_container:
    image:
      repository: projects.registry.vmware.com/tkg/prometheus

You can use the imgpkg command to pull down the default configuration information needed for deploying the Prometheus package. In this example, I’m using the version information obtained earlier to pull the correct version and to create a directory under /tmp to store it.

imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages prometheus.tanzu.vmware.com.2.27.0+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/prometheus-package-2.27.0

Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/prometheus@sha256:27af034c1c77bcae4e1f7f6d3883286e34419ea2e88222642af17393cd34e46a'
  Extracting layer 'sha256:44798ebd112b55ea792f0198cf220a7eaed37c2abc531d6cf8efe89aadc8bff2' (1/1)

Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update

Succeeded

Now you need to make a copy of the /tmp/prometheus-package-2.27.0/config/values.yaml file (prometheus-data-values.yaml) and update it with the information from the tkg-1.3-prometheus-data-values.yaml file that was created from the Prometheus data-values secret earlier.

The last thing to do with the data-values file is to remove any comments:

yq -i eval '... comments=""' prometheus-data-values.yaml

My final prometheus-data-values.yaml file looked like the following (truncated to just the part I had to modify since it was really long):

prometheus-data-values.yaml
ingress:
  enabled: true
  virtual_host_fqdn: "prometheus.corp.tanzu"
  prometheus_prefix: "/"
  alertmanager_prefix: "/alertmanager/"
  prometheusServicePort: 80
  alertmanagerServicePort: 80
  tlsCertificate:
    tls.crt: |
      -----BEGIN CERTIFICATE-----
      MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
      ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
      MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
      MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
      EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
      S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
      BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
      z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
      lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
      FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
      H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
      uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
      F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
      H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
      JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
      Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
      +LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
      AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
      AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
      ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
      VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
      LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
      Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
      dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
      RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
      AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
      JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
      Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
      ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
      AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
      CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
      8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
      RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
      aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
      s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
      n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
      hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
      5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
      hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
      LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
      R22CcKK+kduUjv0X
      -----END CERTIFICATE-----
    tls.key: |
      -----BEGIN PRIVATE KEY-----
      MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
      1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
      ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
      UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
      dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
      8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
      0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
      f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
      7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
      J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
      ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
      BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
      OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
      RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
      IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
      CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
      lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
      jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
      oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
      ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
      TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
      +QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
      aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
      ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
      XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
      +CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
      62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
      R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
      Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
      fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
      O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
      73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
      3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
      8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
      pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
      Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
      GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
      ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
      L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
      csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
      Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
      6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
      NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
      a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
      uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
      eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
      cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
      l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
      zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
      I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
      -----END PRIVATE KEY-----
    ca.crt:

I thought I was ready to install the package but realized that there was a difference between how the Prometheus package would be deployed and how the Prometheus extension was installed. The extension provisioned a PVC of 8GB in size but the package provisions a PVC of 150 GB in size. In order to keep using the same PVC, it would need to be resized. Unfortunately, the implementation of vSphere CSI included with TKG 1.4 does not include the resizer sidecar to the vsphere-csi-controller pod. Luckily, we have documentation for enabling the resizer sidecar at Enable Offline Volume Expansion for vSphere CSI (vSphere 7). And don’t let the title mislead you…if you are running vSphere 7.0 U2 or later, you can actually do online volume expansion as well.

With this in mind, there are a couple of options here. The Prometheus upgrade documentation calls for completely removing the Prometheus extension prior to installing the package…this will certainly avoid the issue of volume expansion but results in all historical Prometheus data being lost. Another option would be to edit the storage: "150Gi" line in the prometheus-data-values.yaml file to be storage: "8Gi". I actually chose to go the route of installing the resizer sidecar and let the PVC be increased to 150GB during the package installation.

At this point, you should be ready to install the package:

tanzu package install prometheus -p prometheus.tanzu.vmware.com -v 2.27.0+vmware.1-tkg.1 -f prometheus-data-values.yaml -n tanzu-system-monitoring

- Installing package 'prometheus.tanzu.vmware.com'
| Getting namespace 'tanzu-system-monitoring'
| Getting package metadata for 'prometheus.tanzu.vmware.com'
| Creating service account 'prometheus-tanzu-system-monitoring-sa'
| Creating cluster admin role 'prometheus-tanzu-system-monitoring-cluster-role'
| Creating cluster role binding 'prometheus-tanzu-system-monitoring-cluster-rolebinding'
| Creating secret 'prometheus-tanzu-system-monitoring-values'
- Creating package resource
\ Package install status: Reconciling


 Added installed package 'prometheus' in namespace 'tanzu-system-monitoring'

If you choose to go the same route that I did, you can validate that the PVC was resized:

kubectl -n tanzu-system-monitoring get events |grep -i volume

15m         Normal    Resizing                     persistentvolumeclaim/prometheus-server   External resizer is resizing volume pvc-eac3c942-170b-4ce9-ab9b-36db528dcf71
14m         Normal    FileSystemResizeRequired     persistentvolumeclaim/prometheus-server   Require file system resize of volume on node
13m         Normal    FileSystemResizeSuccessful   persistentvolumeclaim/prometheus-server   MountVolume.NodeExpandVolume succeeded for volume "pvc-eac3c942-170b-4ce9-ab9b-36db528dcf71"
kubectl -n tanzu-system-monitoring get pvc prometheus-server

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prometheus-server   Bound    pvc-eac3c942-170b-4ce9-ab9b-36db528dcf71   150Gi      RWO            default        163d

Grafana

I had a bit of a hiccup with Grafana while learning that the package/app framework does not work with multiple items installed to the same namespace. The namespace is effectively “owned” by the first app deployed there and subsequent apps will run afoul of ownership errors, similar to the following:

    stderr: |-
      kapp: Error: Ownership errors:
      - Resource 'namespace/tanzu-system-monitoring (v1) cluster' is already associated with a different app 'prometheus-ctrl' namespace: tanzu-system-monitoring (label 'kapp.k14s.io/app=1617374954129473715')

The documentation for upgrading Grafana calls out uninstalling Grafana completely before installing the Grafana package (as was noted for Prometheus). The one big difference between the extension and the package is that the package deploys its app to a different namespace….tanzu-system-dashboards. This method would certainly avoid the issue of multiple apps in the same namespace (apparently only a new concern for 1.4). You could follow the instructions and deploy Prometheus and Grafana as documented but lose all historical data or pick one to keep in tanzu-system-monitoring and put the other in a new namespace. I chose to go with leaving Prometheus in the tanzu-system-monitoring namespace and put Grafana in the tanzu-system-dashboards namespace.

As with the other extensions that have been upgraded to packages already, you need get a copy of the data-values secret so that it can be used when configuring the new package.

kubectl get secret grafana-data-values -n tanzu-system-monitoring -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-grafana-data-values.yaml

As with Prometheus, the items that have been customized in here are the hostname, certificate data (I’m using a wildcard certificate), admin_password and ingress has been enabled.

tkg-1.3-grafana-data-values.yaml
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
monitoring:
  grafana:
    image:
      repository: "projects.registry.vmware.com/tkg/grafana"
    ingress:
      enabled: true
      virtual_host_fqdn: "grafana.corp.tanzu"
      prefix: "/"
      tlsCertificate:
        tls.crt: |
          -----BEGIN CERTIFICATE-----
          MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
          ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
          MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
          MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
          EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
          S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
          BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
          z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
          lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
          FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
          H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
          uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
          F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
          H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
          JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
          Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
          +LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
          AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
          AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
          ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
          VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
          LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
          Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
          dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
          RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
          AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
          JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
          Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
          ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
          AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
          CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
          8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
          RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
          aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
          s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
          n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
          hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
          5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
          hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
          LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
          R22CcKK+kduUjv0X
          -----END CERTIFICATE-----
        tls.key: |
          -----BEGIN PRIVATE KEY-----
          MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
          1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
          ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
          UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
          dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
          8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
          0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
          f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
          7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
          J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
          ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
          BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
          OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
          RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
          IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
          CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
          lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
          jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
          oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
          ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
          TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
          +QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
          aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
          ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
          XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
          +CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
          62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
          R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
          Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
          fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
          O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
          73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
          3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
          8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
          pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
          Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
          GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
          ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
          L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
          csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
          Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
          6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
          NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
          a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
          uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
          eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
          cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
          l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
          zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
          I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
          -----END PRIVATE KEY-----
    secret:
      admin_password: "Vk13YXJlMSE="
  grafana_init_container:
    image:
      repository: "projects.registry.vmware.com/tkg/grafana"
  grafana_sc_dashboard:
    image:
      repository: "projects.registry.vmware.com/tkg/grafana"

As with the external-dns package, you need to remove the Grafana extension completely.

kubectl -n tanzu-system-monitoring delete app grafana

app.kappctrl.k14s.io "grafana" deleted

kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/monitoring/grafana/namespace-role.yaml

kubectl -n tanzu-system-monitoring delete sa grafana-extension-sa
serviceaccount "grafana-extension-sa" deleted

kubectl -n tanzu-system-monitoring delete role grafana-extension-role
role.rbac.authorization.k8s.io "grafana-extension-role" deleted

kubectl -n tanzu-system-monitoring delete rolebinding grafana-extension-rolebinding
rolebinding.rbac.authorization.k8s.io "grafana-extension-rolebinding" deleted

kubectl delete clusterrole grafana-extension-cluster-role
clusterrole.rbac.authorization.k8s.io "grafana-extension-cluster-role" deleted

kubectl delete clusterrolebinding grafana-extension-cluster-rolebinding
clusterrolebinding.rbac.authorization.k8s.io "grafana-extension-cluster-rolebinding" deletedleted

The documentation calls for running kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/monitoring/grafana/grafana-extension.yaml which will delete all RBAC objects but will also delete the namespace. This would cause problems for Prometheus so I opted to manually delete each RBAC object.

Get the available version(s) of the Grafana package:

tanzu package available list grafana.tanzu.vmware.com -A

- Retrieving package versions for grafana.tanzu.vmware.com...
  NAME                      VERSION               RELEASED-AT           NAMESPACE
  grafana.tanzu.vmware.com  7.5.7+vmware.1-tkg.1  2021-05-19T18:00:00Z  tanzu-package-repo-global

You can use the imgpkg command to pull down the default configuration information needed for deploying the Grafana package. In this example, I’m using the version information obtained earlier to pull the correct version and to create a directory under /tmp to store it.

imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages grafana.tanzu.vmware.com.7.5.7+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/grafana-package-7.5.7

Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/grafana@sha256:561e6afe31bf627f89dbb92a7dfded25128037fc407efc7d0190f2c6bc6deec7'
  Extracting layer 'sha256:a6deb6b8b1e485dce2e45bea45a6055d502b945dc0abd22c47e54b9763c0486b' (1/1)

Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update

Succeeded

Now you need to make a copy of the /tmp/grafana-package-7.5.7/config/values.yaml file (grafana-data-values.yaml) and update it with the information from the tkg-1.3-prometheus-data-values.yaml file that was created from the Prometheus data-values secret earlier. Some additional notes: the admin_password value has to be base64 encoded before being entered…echo -n "<password>" | base64, the namespace: value needs to be changed from tanzu-system-monitoring to tanzu-system-dashboards and the service.type: value needs to be changed from LoadBalancer to NodePort.

The last thing to do with the data-values file is to remove any comments:

yq -i eval '... comments=""' grafana-data-values.yaml

My final grafana-data-values.yaml file looked like the following:

grafana-data-values.yaml
namespace: tanzu-system-dashboards
grafana:
  deployment:
    replicas: 1
    containers:
      resources: {}
    podAnnotations: {}
    podLabels: {}
    k8sSidecar:
      containers:
        resources: {}
  service:
    type: NodePort
    port: 80
    targetPort: 3000
    labels: {}
    annotations: {}
  config:
    grafana_ini: |
      [analytics]
      check_for_updates = false
      [grafana_net]
      url = https://grafana.com
      [log]
      mode = console
      [paths]
      data = /var/lib/grafana/data
      logs = /var/log/grafana
      plugins = /var/lib/grafana/plugins
      provisioning = /etc/grafana/provisioning
    datasource_yaml: |-
      apiVersion: 1
      datasources:
        - name: Prometheus
          type: prometheus
          url: prometheus-server.tanzu-system-monitoring.svc.cluster.local
          access: proxy
          isDefault: true
    dashboardProvider_yaml: |-
      apiVersion: 1
      providers:
        - name: 'sidecarDashboardProvider'
          orgId: 1
          folder: ''
          folderUid: ''
          type: file
          disableDeletion: false
          updateIntervalSeconds: 10
          allowUiUpdates: false
          options:
            path: /tmp/dashboards
            foldersFromFilesStructure: true
  pvc:
    annotations: {}
    storageClassName: null
    accessMode: ReadWriteOnce
    storage: "2Gi"
  secret:
    type: "Opaque"
    admin_user: "YWRtaW4="
    admin_password: "Vk13YXJlMSE="
ingress:
  enabled: true
  virtual_host_fqdn: "grafana.corp.tanzu"
  prefix: "/"
  servicePort: 80
  tlsCertificate:
    tls.crt: |
      -----BEGIN CERTIFICATE-----
      MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
      ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
      MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
      MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
      EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
      S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
      BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
      z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
      lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
      FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
      H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
      uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
      F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
      H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
      JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
      Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
      +LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
      AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
      AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
      ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
      VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
      LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
      Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
      dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
      RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
      AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
      JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
      Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
      ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
      AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
      CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
      8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
      RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
      aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
      s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
      n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
      hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
      5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
      hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
      LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
      R22CcKK+kduUjv0X
      -----END CERTIFICATE-----
    tls.key: |
      -----BEGIN PRIVATE KEY-----
      MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
      1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
      ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
      UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
      dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
      8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
      0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
      f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
      7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
      J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
      ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
      BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
      OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
      RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
      IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
      CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
      lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
      jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
      oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
      ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
      TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
      +QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
      aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
      ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
      XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
      +CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
      62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
      R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
      Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
      fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
      O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
      73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
      3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
      8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
      pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
      Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
      GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
      ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
      L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
      csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
      Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
      6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
      NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
      a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
      uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
      eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
      cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
      l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
      zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
      I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
      -----END PRIVATE KEY-----
    ca.crt:

At this point, you should be ready to install the package:

tanzu package install grafana -p grafana.tanzu.vmware.com -v 7.5.7+vmware.1-tkg.1 -f grafana-data-values.yaml -n tanzu-system-dashboards --create-namespace

- Installing package 'grafana.tanzu.vmware.com'
| Getting namespace 'tanzu-system-monitoring'
| Getting package metadata for 'grafana.tanzu.vmware.com'
| Creating service account 'grafana-tanzu-system-monitoring-sa'
| Creating cluster admin role 'grafana-tanzu-system-monitoring-cluster-role'
| Creating cluster role binding 'grafana-tanzu-system-monitoring-cluster-rolebinding'
| Creating secret 'grafana-tanzu-system-monitoring-values'
- Creating package resource


 Added installed package 'grafana' in namespace 'tanzu-system-monitoring'

Troubleshooting package installation issues

As you can probably guess, I had to troubleshoot a number of issues during the package installation/upgrade process. The following are a few of the commands that I would run to help determine what was going on:

  • tanzu package installed list -A – this will list the status of all packages and quickly show any that were not health:
tanzu package installed list -A

- Retrieving installed packages...
  NAME                               PACKAGE-NAME                                        PACKAGE-VERSION        STATUS                                                                NAMESPACE
  cert-manager                       cert-manager.tanzu.vmware.com                       1.1.0+vmware.1-tkg.2   Reconcile succeeded                                                   cert-manager
  grafana                            grafana.tanzu.vmware.com                            7.5.7+vmware.1-tkg.1   Reconcile failed: Error (see .status.usefulErrorMessage for details)  tanzu-system-dashboards
  contour                            contour.tanzu.vmware.com                            1.17.1+vmware.1-tkg.1  Reconcile succeeded                                                   tanzu-system-ingress
  fluent-bit                         fluent-bit.tanzu.vmware.com                         1.7.5+vmware.1-tkg.1   Reconcile succeeded                                                   tanzu-system-logging
  prometheus                         prometheus.tanzu.vmware.com                         2.27.0+vmware.1-tkg.1  Reconcile succeeded                                                   tanzu-system-monitoring
  harbor                             harbor.tanzu.vmware.com                             2.2.3+vmware.1-tkg.1   Reconcile succeeded                                                   tanzu-system-registry
  external-dns                       external-dns.tanzu.vmware.com                       0.8.0+vmware.1-tkg.1   Reconcile succeeded                                                   tanzu-system-service-discovery
  antrea                             antrea.tanzu.vmware.com                                                    Reconcile succeeded                                                   tkg-system
  load-balancer-and-ingress-service  load-balancer-and-ingress-service.tanzu.vmware.com                         Reconcile succeeded                                                   tkg-system
  metrics-server                     metrics-server.tanzu.vmware.com                                            Reconcile succeeded                                                   tkg-system
  pinniped                           pinniped.tanzu.vmware.com                                                  Reconcile succeeded                                                   tkg-system
  vsphere-cpi                        vsphere-cpi.tanzu.vmware.com                                               Reconcile succeeded                                                   tkg-system
  vsphere-csi                        vsphere-csi.tanzu.vmware.com                                               Reconcile succeeded                                                   tkg-system
  • tanzu package installed get <package name> -n <namespace> – this will give more details on any installed package and bubble up any relevant errors:
tanzu package installed get grafana -n tanzu-system-monitoring

/ Retrieving installation details for grafana...
NAME:                    grafana
PACKAGE-NAME:            grafana.tanzu.vmware.com
PACKAGE-VERSION:         7.5.7+vmware.1-tkg.1
STATUS:                  Reconcile failed: Error (see .status.usefulErrorMessage for details)
CONDITIONS:              [{ReconcileFailed True  Error (see .status.usefulErrorMessage for details)}]
USEFUL-ERROR-MESSAGE:    kapp: Error: Ownership errors:
- Resource 'namespace/tanzu-system-monitoring (v1) cluster' is already associated with a different app 'prometheus-ctrl' namespace: tanzu-system-monitoring (label 'kapp.k14s.io/app=1617374954129473715')
  • kubectl package installed updated <installed package name> -p <public package name> -v <version> -f <configuration file> -n <namespace> – this will allow you to make changes to an already installed package:
tanzu package installed update grafana -p grafana.tanzu.vmware.com -v 7.5.7+vmware.1-tkg.1 -f grafana-data-values.yaml -n tanzu-system-dashboards

| Updating package 'grafana'
- Getting package install for 'grafana'
| Updating secret 'grafana-tanzu-system-dashboards-values'
| Updating package install for 'grafana'

 Updated package install 'grafana' in namespace 'tanzu-system-dashboards'
  • kubectl get app -A – shows the status of the apps deployed via the package framework, should mimic the tanzu package installed list -A output:
kubectl get app -A

NAMESPACE                        NAME                                DESCRIPTION                                                                       SINCE-DEPLOY   AGE
cert-manager                     cert-manager                        Reconcile succeeded                                                               32s            2d2h
tanzu-system-ingress             contour                             Reconcile succeeded                                                               41s            28h
tanzu-system-logging             fluent-bit                          Reconcile succeeded                                                               4h49m          11h
tanzu-system-monitoring          grafana                             Reconcile failed: Deploying: Error (see .status.usefulErrorMessage for details)   58s            19m
tanzu-system-monitoring          prometheus                          Reconcile succeeded                                                               71s            163d
tanzu-system-registry            harbor                              Reconcile succeeded                                                               74s            120m
tanzu-system-service-discovery   external-dns                        Reconcile succeeded                                                               63s            5h16m
tkg-system                       antrea                              Reconcile succeeded                                                               95s            163d
tkg-system                       load-balancer-and-ingress-service   Reconcile succeeded                                                               5m17s          73m
tkg-system                       metrics-server                      Reconcile succeeded                                                               5m32s          163d
tkg-system                       pinniped                            Reconcile succeeded                                                               2m5s           163d
tkg-system                       vsphere-cpi                         Reconcile succeeded                                                               82s            163d
tkg-system                       vsphere-csi                         Reconcile succeeded                                                               106s           163d    
  • kubectl get packageinstalls -A – yet another way of looking at installed packages:
kubectl get packageinstalls -A

NAMESPACE                        NAME                                PACKAGE NAME                                         PACKAGE VERSION         DESCRIPTION                                                            AGE
cert-manager                     cert-manager                        cert-manager.tanzu.vmware.com                        1.1.0+vmware.1-tkg.2    Reconcile succeeded   31h
tanzu-system-dashboards          grafana                             grafana.tanzu.vmware.com                             7.5.7+vmware.1-tkg.1    Reconcile succeeded   26m
tanzu-system-ingress             contour                             contour.tanzu.vmware.com                             1.17.1+vmware.1-tkg.1   Reconcile succeeded   30h
tanzu-system-logging             fluent-bit                          fluent-bit.tanzu.vmware.com                          1.7.5+vmware.1-tkg.1    Reconcile succeeded   11h
tanzu-system-monitoring          prometheus                          prometheus.tanzu.vmware.com                          2.27.0+vmware.1-tkg.1   Reconcile succeeded   3h34m
tanzu-system-registry            harbor                              harbor.tanzu.vmware.com                              2.2.3+vmware.1-tkg.1    Reconcile succeeded   30h
tanzu-system-service-discovery   external-dns                        external-dns.tanzu.vmware.com                        0.8.0+vmware.1-tkg.1    Reconcile succeeded   5h21m
tkg-system                       antrea                              antrea.tanzu.vmware.com                              0.13.3+vmware.1-tkg.1   Reconcile succeeded   31h
tkg-system                       load-balancer-and-ingress-service   load-balancer-and-ingress-service.tanzu.vmware.com   1.4.3+vmware.1-tkg.1    Reconcile succeeded   31h
tkg-system                       metrics-server                      metrics-server.tanzu.vmware.com                      0.4.0+vmware.1-tkg.1    Reconcile succeeded   31h
tkg-system                       pinniped                            pinniped.tanzu.vmware.com                            0.4.4+vmware.1-tkg.1    Reconcile succeeded   31h
tkg-system                       vsphere-cpi                         vsphere-cpi.tanzu.vmware.com                         1.21.0+vmware.1-tkg.1   Reconcile succeeded   31h
tkg-system                       vsphere-csi                         vsphere-csi.tanzu.vmware.com                         2.3.0+vmware.1-tkg.2    Reconcile succeeded   31h
  • kubectl get events -n <namespace> – useful for seeing if a particular pod (or other resource) was having some issue while coming up.
  • kubectl get po -n <namespace> – also useful for seeing what pod(s) might not be fully functional.

One last thing that you might notice if you take a closer look at any of the apps that were not newly deployed but upgraded via the package framework. There will be lots of annotations and labels on them related to the older extension framework, per the following example:

apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kappctrl.k14s.io/v1alpha1","kind":"App","metadata":{"annotations":{"tmc.cloud.vmware.com/extension-version":"v2.17.1_vmware.1","tmc.cloud.vmware.com/orphan-resource":"true"},"labels":{"tmc-extension-name":"prometheus","tmc-extension-version":"v2.17.1_vmware.1"},"name":"prometheus","namespace":"tanzu-system-monitoring"},"spec":{"deploy":[{"kapp":{"rawOptions":["--wait-timeout=5m"]}}],"fetch":[{"image":{"url":"projects.registry.vmware.com/tkg/tkg-extensions-templates:v1.3.0_vmware.1"}}],"serviceAccountName":"prometheus-extension-sa","syncPeriod":"5m","template":[{"ytt":{"ignoreUnknownComments":true,"inline":{"pathsFrom":[{"secretRef":{"name":"prometheus-data-values"}}]},"paths":["tkg-extensions/common","tkg-extensions/monitoring/prometheus"]}}]}}
    tmc.cloud.vmware.com/extension-version: v2.17.1_vmware.1
    tmc.cloud.vmware.com/orphan-resource: "true"
  creationTimestamp: "2021-04-02T14:49:10Z"
  finalizers:
  - finalizers.kapp-ctrl.k14s.io/delete
  generation: 4
  labels:
    tmc-extension-name: prometheus
    tmc-extension-version: v2.17.1_vmware.1
  name: prometheus

The annotations and finalizers like these can be removed if desired.

4 thoughts on “Upgrading from TKG 1.3 to 1.4 (including extensions) on vSphere”

  1. Pingback: How to configure external-dns with Microsoft DNS in TKG 1.3 (plus Harbor and Contour) – Little Stuff

  2. Does this guide also apply to Supervisor clusters? “tanzu management-cluster get” does not work for me as there seems to be no management cluster? My test environment works fine, yet im unable to find my management cluster with tanzu cli.

  3. Pingback: Migrating a TKG cluster control-plane endpoint from kube-vip to NSX-ALB – Little Stuff

Leave a Reply to Chris Little Cancel Reply

Your email address will not be published.