Tanzu Kuberentes Grid (TKG) 1.4 was just released on September 9th, 2021 and I couldn’t wait to take it for a spin. You can read all about the new features in the Release Notes and be sure to check out the updated documentation as there are several changes to how things work from 1.3.
My first task with exploring TKG 1.4 was to upgrade my existing 1.3.0 cluster and deployed extensions. This was a fairly painless process as you’ll see in the following post. For details on how my 1.3.0 environment was deployed, please see my previous posts, Installing Tanzu Kubernetes Grid 1.3 on vSphere with NSX Advanced Load Balancer, How to configure external-dns with Microsoft DNS in TKG 1.3 (plus Harbor and Contour) and Working with TKG Extensions and Shared Services in TKG 1.2.
Table of Contents
Download updated components
You can download updated tanzu CLI binaries, node OS OVA files, signed kubectl, crashd and velero binaries from customerconnect.vmware.com.
Install the 1.4 tanzu CLI binary and additional executables
You can first verify your current tanzu CLI binary version via the tanzu version command:
tanzu version
version: v1.3.0
buildDate: 2021-03-16
sha: 06ddc9aAfter downloading the updated tanzu CLI binary (tar/gz format), you will need to extract and install it on your local system:
tar -xvf tanzu-cli-bundle-linux-amd64.tar
cli/
cli/core/
cli/core/v1.4.0/
cli/core/v1.4.0/tanzu-core-linux_amd64
cli/core/plugin.yaml
cli/login/
cli/login/v1.4.0/
cli/login/v1.4.0/tanzu-login-linux_amd64
cli/login/plugin.yaml
cli/cluster/
cli/cluster/v1.4.0/
cli/cluster/v1.4.0/tanzu-cluster-linux_amd64
cli/cluster/plugin.yaml
cli/package/
cli/package/v1.4.0/
cli/package/v1.4.0/tanzu-package-linux_amd64
cli/package/plugin.yaml
cli/management-cluster/
cli/management-cluster/v1.4.0/
cli/management-cluster/v1.4.0/tanzu-management-cluster-linux_amd64
cli/management-cluster/plugin.yaml
cli/pinniped-auth/
cli/pinniped-auth/v1.4.0/
cli/pinniped-auth/v1.4.0/tanzu-pinniped-auth-linux_amd64
cli/pinniped-auth/plugin.yaml
cli/kubernetes-release/
cli/kubernetes-release/v1.4.0/
cli/kubernetes-release/v1.4.0/tanzu-kubernetes-release-linux_amd64
cli/kubernetes-release/plugin.yaml
cli/manifest.yaml
cli/ytt-linux-amd64-v0.34.0+vmware.1.gz
cli/kapp-linux-amd64-v0.37.0+vmware.1.gz
cli/imgpkg-linux-amd64-v0.10.0+vmware.1.gz
cli/kbld-linux-amd64-v0.30.0+vmware.1.gz
cli/vendir-linux-amd64-v0.21.1+vmware.1.gzcd cli/
ubuntu@cli-vm:~/cli$ ls
cluster kubernetes-release pinniped-auth
core login vendir-linux-amd64-v0.21.1+vmware.1.gz
imgpkg-linux-amd64-v0.10.0+vmware.1.gz management-cluster ytt-linux-amd64-v0.34.0+vmware.1.gz
kapp-linux-amd64-v0.37.0+vmware.1.gz manifest.yaml
kbld-linux-amd64-v0.30.0+vmware.1.gz packagesudo install core/v1.4.0/tanzu-core-linux_amd64 /usr/local/bin/tanzuYou can now check to see that the tanzu CLI binary version has been updated:
tanzu version
! Configuration is now stored in /home/ubuntu/.config/tanzu. Legacy configuration directory /home/ubuntu/.tanzu is deprecated and will be removed in a future release.
! To complete migration, please remove legacy configuration directory /home/ubuntu/.tanzu and adjust your script(s), if any, to point to the new location.
version: v1.4.0
buildDate: 2021-08-30
sha: c9929b8fBe sure to make note of the new configuration directory as noted in the tanzu version output.
You can follow a similar process to gunzip and install the other utilities located in the cli folder (imgpkg, kapp, kbld, vendir, ytt):
for file in $(ls *.gz)
do
gunzip $file
sudo install ${file::-3} /usr/local/bin/$(echo $file | awk -F - '{print $1}')
doneLastly, you can follow a similar process to update the kubectl binary to the latest version:
gunzip kubectl-linux-v1.21.2+vmware.1.gz
sudo install kubectl-linux-v1.21.2+vmware.1 /usr/local/bin/kubectl
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2+vmware.1", GitCommit:"54e7e68e30dd3f9f7bb4f814c9d112f54f0fb273", GitTreeState:"clean", BuildDate:"2021-06-28T22:17:36Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4+vmware.1", GitCommit:"d475bbd9e7cd66c6db7069cb447766daada65e3b", GitTreeState:"clean", BuildDate:"2021-02-22T22:15:46Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}Update the tanzu CLI plugins
The tanzu CLI plugins that were installed in a previous version also need to be updated but this a simple operation:
tanzu plugin list
NAME LATEST VERSION DESCRIPTION REPOSITORY VERSION STATUS
alpha v1.3.1 Alpha CLI commands core not installed
cluster v1.3.1 Kubernetes cluster operations core v1.3.0 upgrade available
kubernetes-release v1.3.1 Kubernetes release operations core v1.3.0 upgrade available
login v1.3.1 Login to the platform core v1.3.0 upgrade available
management-cluster v1.3.1 Kubernetes management cluster operations core v1.3.0 upgrade available
pinniped-auth v1.3.1 Pinniped authentication operations (usually not directly invoked) core v1.3.0 upgrade availabletanzu plugin cleantanzu plugin list
NAME LATEST VERSION DESCRIPTION REPOSITORY VERSION STATUS
alpha v1.3.1 Alpha CLI commands core not installed
cluster v1.3.1 Kubernetes cluster operations core not installed
kubernetes-release v1.3.1 Kubernetes release operations core not installed
login v1.3.1 Login to the platform core not installed
management-cluster v1.3.1 Kubernetes management cluster operations core not installed
pinniped-auth v1.3.1 Pinniped authentication operations (usually not directly invoked) core not installedNote: This next command needs to be run from the cli directory noted previously.
tanzu plugin install --local . alltanzu plugin list
NAME LATEST VERSION DESCRIPTION REPOSITORY VERSION STATUS
alpha v1.3.1 Alpha CLI commands core not installed
cluster v1.3.1 Kubernetes cluster operations core v1.4.0 installed
kubernetes-release v1.3.1 Kubernetes release operations core v1.4.0 installed
login v1.3.1 Login to the platform core v1.4.0 installed
management-cluster v1.3.1 Kubernetes management cluster operations core v1.4.0 installed
package Tanzu package management v1.4.0 installed
pinniped-auth v1.3.1 Pinniped authentication operations (usually not directly invoked) core v1.4.0 installedDeploy a new node OS template to vSphere
Since the upgrade includes support for newer Kubernetes versions, you’ll want to deploy the OVA files corresponding to the versions you’ll want/need (you’ll need at least the 1.21.2 OVA as it is required for the management cluster).
I like to take a snapshot of the VM before converting it to a template so that I can deploy my nodes as linked-clones and save time and disk space. Keep in mind that if you do this, you will not be able to resize the node disk size during cluster creation.
You’ll want to repeat this for as many different Kubernetes versions (or nodes OS types) that you might want to deploy.
Validate current management and workload cluster
With the prerequisites out of the way, you can move on to inspecting your current installation prior to upgrading.
If you’re not already accessing your management cluster, you’ll need to use the tanzu login command to get started:
tanzu login
? Select a server tkg-mgmt ()
✔ successfully logged in to management cluster using the kubeconfig tkg-mgmttanzu management-cluster get
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
tkg-mgmt tkg-system running 1/1 1/1 v1.21.2+vmware.1 management
Details:
NAME READY SEVERITY REASON SINCE MESSAGE
/tkg-mgmt True 26m
├─ClusterInfrastructure - VSphereCluster/tkg-mgmt True 160d
├─ControlPlane - KubeadmControlPlane/tkg-mgmt-control-plane True 26m
│ └─Machine/tkg-mgmt-control-plane-cqrs4 True 33m
└─Workers
└─MachineDeployment/tkg-mgmt-md-0
└─Machine/tkg-mgmt-md-0-5895d84d4c-ljmr9 True 22m
Providers:
NAMESPACE NAME TYPE PROVIDERNAME VERSION WATCHNAMESPACE
capi-kubeadm-bootstrap-system bootstrap-kubeadm BootstrapProvider kubeadm v0.3.23
capi-kubeadm-control-plane-system control-plane-kubeadm ControlPlaneProvider kubeadm v0.3.23
capi-system cluster-api CoreProvider cluster-api v0.3.23
capv-system infrastructure-vsphere InfrastructureProvider vsphere v0.7.10tanzu cluster list --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN
tkg-wld default running 1/1 2/2 v1.20.4+vmware.1 <none> dev
tkg-mgmt tkg-system running 1/1 1/1 v1.20.4+vmware.1 management dev You can see from the previous output that I have single control plane node management cluster and workload clusters present, both running Kubernetes version 1.20.4.
Upgrade the management cluster
Note: I you’re using NSX Advanced Load Balancer (NSX ALB) as I am, be sure that you are on a supported version. The supported versions for TKG 1.4 are 20.1.3 and 20.1.6. If you need to upgraded, check out my previous post, Upgrading NSX Advanced Load Balancer.
There are a few parameters that you can pass to the tanzu management-cluster upgrade command but I didn’t need any of them since my installation is fairly simple:
tanzu management-cluster upgrade
Upgrading management cluster 'tkg-mgmt' to TKG version 'v1.4.0' with Kubernetes version 'v1.21.2+vmware.1'. Are you sure? [y/N]: yA new control plane VM will be created from the 1.21 node OS image that was uploaded previously.
Once it’s powered on, it will be joined to the existing cluster (using kubeadm) and then additional control plane and worker nodes will be created and joined to the cluster. The original nodes will be powered off and deleted.
You can see that the new and old control plane nodes are both present part way through the upgrade:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
tkg-mgmt-control-plane-6vd6z Ready control-plane,master 160d v1.20.4+vmware.1
tkg-mgmt-control-plane-cqrs4 Ready control-plane,master 3m12s v1.21.2+vmware.1
tkg-mgmt-md-0-766c78c69c-7r66n Ready <none> 160d v1.20.4+vmware.1 And once the new control plane node is fully joined to the cluster, the old control plane node is deactivated and deleted:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
tkg-mgmt-control-plane-6vd6z Ready,SchedulingDisabled control-plane,master 160d v1.20.4+vmware.1
tkg-mgmt-control-plane-cqrs4 Ready control-plane,master 5m9s v1.21.2+vmware.1
tkg-mgmt-md-0-766c78c69c-7r66n Ready <none> 160d v1.20.4+vmware.1 
kubectl get nodes
NAME STATUS ROLES AGE VERSION
tkg-mgmt-control-plane-cqrs4 Ready control-plane,master 7m17s v1.21.2+vmware.1
tkg-mgmt-md-0-766c78c69c-7r66n Ready <none> 160d v1.20.4+vmware.1 This same process is repeated for the worker nodes in the management cluster. When the upgrade is complete, you should see all control plane and worker nodes running at the new version:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
tkg-mgmt-control-plane-cqrs4 Ready control-plane,master 14m v1.21.2+vmware.1
tkg-mgmt-md-0-5895d84d4c-ljmr9 Ready <none> 3m45s v1.21.2+vmware.1 And the same tanzu commands that were run earlier to inspect the clusters will show that the management cluster is now running on a newer version:
tanzu management-cluster get
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
tkg-mgmt tkg-system running 1/1 1/1 v1.21.2+vmware.1 management
Details:
NAME READY SEVERITY REASON SINCE MESSAGE
/tkg-mgmt True 9m49s
├─ClusterInfrastructure - VSphereCluster/tkg-mgmt True 160d
├─ControlPlane - KubeadmControlPlane/tkg-mgmt-control-plane True 9m49s
│ └─Machine/tkg-mgmt-control-plane-cqrs4 True 17m
└─Workers
└─MachineDeployment/tkg-mgmt-md-0
└─Machine/tkg-mgmt-md-0-5895d84d4c-ljmr9 True 5m49s
Providers:
NAMESPACE NAME TYPE PROVIDERNAME VERSION WATCHNAMESPACE
capi-kubeadm-bootstrap-system bootstrap-kubeadm BootstrapProvider kubeadm v0.3.23
capi-kubeadm-control-plane-system control-plane-kubeadm ControlPlaneProvider kubeadm v0.3.23
capi-system cluster-api CoreProvider cluster-api v0.3.23
capv-system infrastructure-vsphere InfrastructureProvider vsphere v0.7.10
tanzu cluster list --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN
tkg-wld default running 1/1 2/2 v1.20.4+vmware.1 <none> dev
tkg-mgmt tkg-system running 1/1 1/1 v1.21.2+vmware.1 management dev You can expand the link below to see an example of the default log output during a management cluster upgrade:
tanzu management-cluster upgrade outputcluster specific secret is not present, fallback on bootstrap credential secret
Upgrading management cluster providers...
Checking cert-manager version...
Deleting cert-manager Version="v0.16.1"
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Performing upgrade...
Deleting Provider="cluster-api" Version="" TargetNamespace="capi-system"
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Deleting Provider="bootstrap-kubeadm" Version="" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Deleting Provider="control-plane-kubeadm" Version="" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Deleting Provider="infrastructure-vsphere" Version="" TargetNamespace="capv-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Management cluster providers upgraded successfully...
Upgrading management cluster kubernetes version...
Verifying kubernetes version...
Retrieving configuration for upgrade cluster...
Create InfrastructureTemplate for upgrade...
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1...
Waiting for kubernetes version to be updated for control plane nodes
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1...
Waiting for kubernetes version to be updated for worker nodes...
updating additional components: 'metadata/tkg' ...
updating additional components: 'addons-management/kapp-controller' ...
updating additional components: 'addons-management/standard-package-repo' ...
updating additional components: 'addons-management/tanzu-addons-manager' ...
updating additional components: 'tkr/tkr-controller' ...
updating additional components: 'addons-management/core-package-repo' ...
Waiting for additional components to be up and running...
Waiting for packages to be up and running...
Management cluster 'tkg-mgmt' successfully upgraded to TKG version 'v1.4.0' with kubernetes version 'v1.21.2+vmware.1'Note: I went through this upgrade process a few times and once saw the following message at the end of the management cluster upgrade process:
Warning: Management cluster is upgraded successfully, but some packages are failing. Failure while waiting for packages to be installed: package reconciliation failed: kapp: Error: Timed out waiting after 30s
Everything was fine in my management cluster and it just appeared to take a little longer than expected for all pods to be up and running again. You can quickly check the status via the kubectl get po -A command.
Update the Pinniped settings
This wasn’t a concern for me as I am using LDAP (AD) authentication in my TKG clusters but if you had configured OIDC authentication prior to upgrading TKG to 1.4, you will need to follow the instructions noted at Update Pinniped Settings for Management Clusters with OIDC Authentication. The is because both Dex and Pinniped were needed for OIDC and LDAP authentication in TKG 1.3 but only Pinniped is needed for OIDC authentication in TKG 1.4 (LDAP still needs Dex). If you’re using LDAP authentication like I am, you can skip this step.
Upgrade the workload cluster(s)
You’ll want to get a list of your clusters first:
tanzu cluster list
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN
tkg-wld default running 1/1 2/2 v1.20.4+vmware.1 <none> dev And you’ll also want to see what available upgrades you have.
tanzu kubernetes-release get
NAME VERSION COMPATIBLE ACTIVE UPDATES AVAILABLE
v1.19.12---vmware.1-tkg.1 v1.19.12+vmware.1-tkg.1 True True True
v1.20.8---vmware.1-tkg.2 v1.20.8+vmware.1-tkg.2 True True True
v1.21.2---vmware.1-tkg.1 v1.21.2+vmware.1-tkg.1 True True FalseThere is a new feature with the kubernetes-release option to tanzu that allows you to activate or deactivate specific tkr versions. For example, if I don’t want to see the 1.19.12 version any longer, I can run tanzu kubernetes-release deactivate v1.19.12---vmware.1-tkg.1 to remove it from the list of available upgrades. After making this change, tanzu kuberentes-release get will no longer show it:
tanzu kubernetes-release get
NAME VERSION COMPATIBLE ACTIVE UPDATES AVAILABLE
v1.20.8---vmware.1-tkg.2 v1.20.8+vmware.1-tkg.2 True True True
v1.21.2---vmware.1-tkg.1 v1.21.2+vmware.1-tkg.1 True True FalseIf you want to see all tkr versions, you can pass the -a switch to tanzu kubernetes-release get since by default, it only returns compatible, active releases:
tanzu kubernetes-release get -a
NAME VERSION COMPATIBLE ACTIVE UPDATES AVAILABLE
v1.17.16---vmware.2-tkg.1 v1.17.16+vmware.2-tkg.1 False True True
v1.17.16---vmware.2-tkg.2 v1.17.16+vmware.2-tkg.2 False True True
v1.17.16---vmware.3-tkg.1 v1.17.16+vmware.3-tkg.1 False True True
v1.18.16---vmware.1-tkg.1 v1.18.16+vmware.1-tkg.1 False True True
v1.18.16---vmware.1-tkg.2 v1.18.16+vmware.1-tkg.2 False True True
v1.18.16---vmware.3-tkg.1 v1.18.16+vmware.3-tkg.1 False True True
v1.18.17---vmware.1-tkg.1 v1.18.17+vmware.1-tkg.1 False True True
v1.18.17---vmware.2-tkg.1 v1.18.17+vmware.2-tkg.1 False True True
v1.19.12---vmware.1-tkg.1 v1.19.12+vmware.1-tkg.1 True False True
v1.19.8---vmware.1-tkg.1 v1.19.8+vmware.1-tkg.1 False True True
v1.19.8---vmware.1-tkg.2 v1.19.8+vmware.1-tkg.2 False True True
v1.19.8---vmware.3-tkg.1 v1.19.8+vmware.3-tkg.1 False True True
v1.19.9---vmware.1-tkg.1 v1.19.9+vmware.1-tkg.1 False True True
v1.19.9---vmware.2-tkg.1 v1.19.9+vmware.2-tkg.1 False True True
v1.20.4---vmware.1-tkg.1 v1.20.4+vmware.1-tkg.1 False True True
v1.20.4---vmware.1-tkg.2 v1.20.4+vmware.1-tkg.2 False True True
v1.20.4---vmware.3-tkg.1 v1.20.4+vmware.3-tkg.1 False True True
v1.20.5---vmware.1-tkg.1 v1.20.5+vmware.1-tkg.1 False True True
v1.20.5---vmware.2-fips.1-tkg.1 v1.20.5+vmware.2-fips.1-tkg.1 False True True
v1.20.5---vmware.2-tkg.1 v1.20.5+vmware.2-tkg.1 False True True
v1.20.8---vmware.1-tkg.2 v1.20.8+vmware.1-tkg.2 True True True
v1.21.2---vmware.1-tkg.1 v1.21.2+vmware.1-tkg.1 True True FalseAs noted previously, my cluster is running on 1.20.4 so I will check to see what upgrades are available:
tanzu kubernetes-release available-upgrades get v1.20.4---vmware.1
There are no available upgrades for this TanzuKubernetesRelease
NAME VERSIONHmmm, that’s certainly not what I expected. I tried it a little differently (passed in the cluster name) to see if I could get the results I was looking for:
tanzu kubernetes-release available-upgrades get tkg-wld
There are no available upgrades for this TanzuKubernetesRelease
NAME VERSIONSame thing. It turns out that this is a known issue as documented in the Known Issues section of the TKG 1.4 Release Notes (look for tanzu kubernetes-release available-upgrades does not show available upgrades for a given Kubernetes version). The workaround is to validate that there is an available version to upgrade to in the tanzu kubernetes-release get output and then pass the version explicitly when performing the upgrade.
One thing I did notice was that I can use a tanzu cluster command to list what upgrades are available:
tanzu cluster available-upgrades get tkg-wld
NAME VERSION COMPATIBLE
v1.20.4---vmware.1-tkg.2 v1.20.4+vmware.1-tkg.2 False
v1.20.4---vmware.3-tkg.1 v1.20.4+vmware.3-tkg.1 False
v1.20.5---vmware.1-tkg.1 v1.20.5+vmware.1-tkg.1 False
v1.20.5---vmware.2-fips.1-tkg.1 v1.20.5+vmware.2-fips.1-tkg.1 False
v1.20.5---vmware.2-tkg.1 v1.20.5+vmware.2-tkg.1 False
v1.20.8---vmware.1-tkg.2 v1.20.8+vmware.1-tkg.2 True
v1.21.2---vmware.1-tkg.1 v1.21.2+vmware.1-tkg.1 TrueI’m going to upgrade my cluster to 1.21.2 and since I already have that version saved as a template in vSphere, I should be ready to go at this point:
tanzu cluster upgrade tkg-wld --tkr v1.21.2---vmware.1-tkg.1
Upgrading workload cluster 'tkg-wld' to kubernetes version 'v1.21.2+vmware.1'. Are you sure? [y/N]: yAs with the management cluster, you’ll see the control plane VM(s) getting created first and the old ones deleted and then the worker nodes will follow.
Before the upgrade:
kubectl --context=tkg-wld-admin@tkg-wld get nodes
NAME STATUS ROLES AGE VERSION
tkg-wld-control-plane-bfgdk Ready control-plane,master 160d v1.20.4+vmware.1
tkg-wld-md-0-84c4d7898c-ds2b2 Ready <none> 160d v1.20.4+vmware.1
tkg-wld-md-0-84c4d7898c-vv9g2 Ready <none> 160d v1.20.4+vmware.1 And after the upgrade:
kubectl --context=tkg-wld-admin@tkg-wld get nodes
NAME STATUS ROLES AGE VERSION
tkg-wld-control-plane-nnvtr Ready control-plane,master 25m v1.21.2+vmware.1
tkg-wld-md-0-5c98898f95-9x2mp Ready <none> 16m v1.21.2+vmware.1
tkg-wld-md-0-5c98898f95-k9fsx Ready <none> 4m35s v1.21.2+vmware.1 You can expand the link below to see an example of the default log output during a management cluster upgrade:
tanzu cluster upgrade outputValidating configuration...
Verifying kubernetes version...
Retrieving configuration for upgrade cluster...
Create InfrastructureTemplate for upgrade...
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1...
Waiting for kubernetes version to be updated for control plane nodes
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1...
Waiting for kubernetes version to be updated for worker nodes...
updating additional components: 'metadata/tkg' ...
updating additional components: 'addons-management/kapp-controller' ...
updating additional components: 'addons-management/standard-package-repo' ...
Waiting for packages to be up and running...
Cluster 'tkg-wld' successfully upgraded to kubernetes version 'v1.21.2+vmware.1'If you have applications deployed that use persistent volumes, you’ll see a lot of activity similar to the following in the vSphere Client when the upgrade process gets near the end:
This is expected since all of the virtual disks that back the PVCs have to be reattached to the new worker nodes.
One thing I noticed that is worth mentioning is that even though the upgrade process was finished, my deployed applications were not all available for a few more minutes. Harbor was the longest holdout at being fully functional about 10 minutes after the upgrade completed.
Update the AKO operator
The AKO operator controls the ability to work with NSX Advanced Load Balancer (NSX ALB) for creating and managing load balancer services. In the 1.4 version of TKG, this component is installed as a package. Since we’re upgrading, that package doesn’t exist and needs to be installed (preserving the existing configuration, of course). If you are not using NSX ALB you can skip this section.
Taking a look at the installed packages in the management cluster, you can see that the AKO Operator package is not installed (I have a lot more on Packages in this post):
tanzu package installed list -A
- Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system
metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system
pinniped pinniped.tanzu.vmware.com Reconcile succeeded tkg-system
tanzu-addons-manager addons-manager.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-systemYou will first need to export a few environment variables that will be used by the subsequent tanzu cluster create command.
export _TKG_CLUSTER_FORCE_ROLE="management"
export FILTER_BY_ADDON_TYPE="networking/ako-operator"
export REMOVE_CRS_FOR_ADDON_TYPE="networking/ako-operator"
export NAMESPACE="tkg-system"You need to issue a tanzu cluster create command with the --dry-run option to create an AKO operator addon secret manifest file.
tanzu cluster create tkg-mgmt --dry-run -f ~/.config/tanzu/tkg/clusterconfigs/rj3duprspd.yaml --vsphere-controlplane-endpoint 1.1.1.1 > ako-operator-addon-manifest.yamlNote: Replace tkg-mgmt with the name of your management cluster and rj3duprspd.yaml with the name of the cluster configuration file that was created when your management cluster was created. You can leave the 1.1.1.1 address as-is since it is just there as a placeholder and not really used in the generated manifest.
You should end up with a file similar to the following:
apiVersion: v1
kind: Secret
metadata:
annotations:
tkg.tanzu.vmware.com/addon-type: networking/ako-operator
labels:
clusterctl.cluster.x-k8s.io/move: ""
tkg.tanzu.vmware.com/addon-name: ako-operator
tkg.tanzu.vmware.com/cluster-name: tkg-mgmt
name: tkg-mgmt-ako-operator-addon
namespace: tkg-system
stringData:
values.yaml: |
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
akoOperator:
avi_enable: true
namespace: tkg-system-networking
cluster_name: tkg-mgmt
config:
avi_disable_ingress_class: true
avi_ingress_default_ingress_controller: false
avi_ingress_shard_vs_size: ""
avi_ingress_service_type: ""
avi_ingress_node_network_list: '""'
avi_admin_credential_name: avi-controller-credentials
avi_ca_name: avi-controller-ca
avi_controller: nsxalb-cluster.corp.tanzu
avi_username: admin
avi_password: VMware1!
avi_cloud_name: Default-Cloud
avi_service_engine_group: Default-Group
avi_data_network: K8s-Frontend
avi_data_network_cidr: 192.168.220.0/23
avi_ca_data_b64: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZhekNDQTFPZ0F3SUJBZ0lRTWZaeTA4bXV2SVZLZFpWRHo3L3JZekFOQmdrcWhraUc5dzBCQVFzRkFEQkkKTVJVd0V3WUtDWkltaVpQeUxHUUJHUllGZEdGdWVuVXhGREFTQmdvSmtpYUprL0lzWkFFWkZnUmpiM0p3TVJrdwpGd1lEVlFRREV4QkRUMDVVVWs5TVEwVk9WRVZTTFVOQk1CNFhEVEl3TURneE9URTNNakEwTkZvWERUTXdNRGd4Ck9URTNNekF6TlZvd1NERVZNQk1HQ2dtU0pvbVQ4aXhrQVJrV0JYUmhibnAxTVJRd0VnWUtDWkltaVpQeUxHUUIKR1JZRVkyOXljREVaTUJjR0ExVUVBeE1RUTA5T1ZGSlBURU5GVGxSRlVpMURRVENDQWlJd0RRWUpLb1pJaHZjTgpBUUVCQlFBRGdnSVBBRENDQWdvQ2dnSUJBTEtJZFg3NjQzUHp2dFZYbHFOSXdEdU5xK3JoY0hGMGZqUjQxNGorCjFJR1FVdVhyeWtqaFNEdGhQUCs4QkdON21CZ0hUOEFqQVMxYjk1eGM4QjBTMkZobG4zQW9SRTl6MDNHdGZzQnUKRlNCUlVWd0FpZlg2b1h1OTdXemZmaHFQdHhaZkxKWGJoT29tamxrWDZpZmZBczJUT0xVeDJPajR3MnZ5Ymh6agpsY0E3MGFpKzBTbDZheFNvM2xNWjRLa3VaMldnZkVjYURqamozMy9wVjMvYm5GSys3eWRQdHRjMlRlazV4c0k4ClhOTWlySVZ4VWlVVDRZTHk0V0xpUzIwMEpVZmJwMVpuTXZuYlE4SnYxUW5abDlXN1dtQlBjZ3hSNEFBdWIwSzQKdlpMWHU2TVhpYm9UbHprTUIvWXRoQ2tUTmxKY0traEhmNjBZUi9UNlN4MVQybnVweUJhNGRlbzVVR1B6aFJpSgpwTjM3dXFxQWRLMXFNRHBDakFSalM2VTdMZjlKS2pmaXJpTHpMZXlBalA4a2FONFRkSFNaZDBwY1FvWlN4ZXhRCjluKzRFNE1RbTRFSjREclZaQ2lsc3lMMkJkRVRjSFhLUGM3cStEYjRYTTdqUEtORzVHUDFFTVY0WG9odjU4eVoKL3JSZm1LNjRnYXI4QU1uT0tUMkFQNjgxcWRaczdsbGpPTmNYVUFMemxYNVRxSWNoWVQwRFZRbUZMWW9NQmVaegowbDIxUWpiSzBZV25QemE2WWkvTjRtNnJGYkVCNFdYaXFoWVNreHpyTXZvY1ZVZ2Q0QUFQMXZmSE5uRkVzblVSCm5Tc2lnbEZIL3hseU8zY0JGcm1vWkF4YkEyMDkxWEhXaEI0YzBtUUVJM2hPcUFCOFVvRkdCclFwbVErTGVzb0MKMUxaOUFnTUJBQUdqVVRCUE1Bc0dBMVVkRHdRRUF3SUJoakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZApEZ1FXQkJURkF4U3ZZNjRRNWFkaG04SVllY0hCQVV1b2J6QVFCZ2tyQmdFRUFZSTNGUUVFQXdJQkFEQU5CZ2txCmhraUc5dzBCQVFzRkFBT0NBZ0VBamcvdjRtSVA3Z0JWQ3c0cGVtdEduM1BTdERoL2FCOXZiV3lqQXl4U05hYUgKSDBuSUQ1cTV3b3c5dWVCaURmalRQbmhiZjNQNzY4SEc4b0wvKzlDK1ZtLzBsaUZCZCswL0RhYXlLcEFORk1MQgpCVitzMmFkV1JoUXVjTFFmWFB3dW04UnliV3Y4MndrUmtXQ0NkT0JhQXZBTXVUZ2swOFN3Skl5UWZWZ3BrM25ZCjBPd2pGd1NBYWR2ZXZmK0xvRC85TDhSOU5FdC9uNFdKZStMdEVhbW85RVZiK2wrY1lxeXh5dWJBVlkwWTZCTTIKR1hxQWgzRkVXMmFRTXB3b3VoLzVTN3c1b1NNWU42bWlZMW9qa2k4Z1BtMCs0K0NJTFBXaC9mcjJxME8vYlB0YgpUcisrblBNbVo4b3Y5ZXBOR0l1cWh0azVqYTIvSnVZK1JXNDZJUmM4UXBGMUV5VWFlMDJFNlUyVmFjczdHZ2UyCkNlU0lOa29MRkZtaUtCZkluL0hBY2hsbWU5YUw2RGxKOXdBcmVCREgzRThrSDdnUkRXYlNLMi9RRDBIcWFjK0UKZ2VHSHdwZy84T3RCT0hVTW5NN2VMT1hCSkZjSm9zV2YwWG5FZ1M0dWJnYUhncURFdThwOFBFN3JwQ3h0VU51cgp0K3gyeE9OSS9yQldnZGJwNTFsUHI3bzgxOXpQSkN2WVpxMVBwMXN0OGZiM1JsVVNXdmJRTVBGdEdBeWFCeStHCjBSZ1o5V1B0eUVZZ25IQWI1L0RxNDZzbmU5L1FuUHd3R3BqdjFzMW9FM1pGUWpodm5HaXM4K2RxUnhrM1laQWsKeWlEZ2hXN2FudHpZTDlTMUNDOHNWZ1ZPd0ZKd2ZGWHBkaWlyMzVtUWx5U0czMDFWNEZzUlYrWjBjRnA0TmkwPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
avi_labels: '""'
avi_disable_static_route_sync: true
avi_cni_plugin: antrea
avi_control_plane_ha_provider: false
avi_management_cluster_vip_network_name: K8s-Frontend
avi_management_cluster_vip_network_cidr: 192.168.220.0/23
avi_control_plane_endpoint_port: 6443
type: tkg.tanzu.vmware.com/addonYou can now apply the manifest which will create the AKO Operator addon secret and trigger the installation of the AKO Operator package:
kubectl apply -f ako-operator-addon-manifest.yaml
secret/tkg-mgmt-ako-operator-addon createdYou can see that the AKO Operator package and the load-balancer-and-ingress-service package are now installed:
tanzu package installed list -A
- Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
ako-operator ako-operator.tanzu.vmware.com Reconcile succeeded tkg-system
antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system
load-balancer-and-ingress-service load-balancer-and-ingress-service.tanzu.vmware.com Reconcile succeeded tkg-system
metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system
pinniped pinniped.tanzu.vmware.com Reconcile succeeded tkg-system
tanzu-addons-manager addons-manager.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-systemThe load-balancer-and-ingress-service package will automatically be installed to any workload clusters as well:
tanzu package installed list -A
/ Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system
load-balancer-and-ingress-service load-balancer-and-ingress-service.tanzu.vmware.com Reconcile failed: Error (see .status.usefulErrorMessage for details) tkg-system
metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system
pinniped pinniped.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-systemYou can see that it’s not in a healthy state though. If you dig further into it you’ll see a little more detail about why:
tanzu package installed get load-balancer-and-ingress-service -n tkg-system
/ Retrieving installation details for load-balancer-and-ingress-service...
NAME: load-balancer-and-ingress-service
PACKAGE-NAME: load-balancer-and-ingress-service.tanzu.vmware.com
PACKAGE-VERSION:
STATUS: Reconcile failed: Error (see .status.usefulErrorMessage for details)
CONDITIONS: [{ReconcileFailed True Error (see .status.usefulErrorMessage for details)}]
USEFUL-ERROR-MESSAGE: kapp: Error: Applying update statefulset/ako (apps/v1) namespace: avi-system:
Updating resource statefulset/ako (apps/v1) namespace: avi-system:
StatefulSet.apps "ako" is invalid: spec:
Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden (reason: Invalid)The package is trying to make a change to the existing ako statefulset that is now allowed. This is easy enough to work around by deleting the existing stateful set and letting the package framework recreate it.
kubectl -n avi-system delete statefulsets.apps ako
statefulset.apps "ako" deleted
tanzu package installed get load-balancer-and-ingress-service -n tkg-system
- Retrieving installation details for load-balancer-and-ingress-service...
NAME: load-balancer-and-ingress-service
PACKAGE-NAME: load-balancer-and-ingress-service.tanzu.vmware.com
PACKAGE-VERSION:
STATUS: Reconcile succeeded
CONDITIONS: [{ReconcileSucceeded True }]
USEFUL-ERROR-MESSAGE:This is a known issue affecting TKG 1.4 and should be addressed in a future version.
One other oddity I noticed was that sometimes my enovy service got a new IP address from NSX ALB. To get back to the original IP address it was as easy as editing the service and specifying loadBalancerIP: <previous IP> and then deleting any envoy virtual services in NSX ALB. A new envoy virtual service will be created after a few minutes with the desired IP address.
Update the extensions
NOTE: These instructions are specific to coming from a TKG 1.3 installation. The extension framework was changed slightly in 1.3.1. If you are using these instructions but are running TKG 1.3.1, you do not need to delete the extensions prior to installing the packages…i.e. you don’t want to run and of the commands similar to kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/ingress/contour/contour-extension.yaml as they will remove the installed app and any stateful data.
From several of my previous posts, you may know that I have installed all of the available TKG extensions into my workload cluster (Cert Manager, Harbor, Contour, External DNS, Fluentbit, Prometheus and Grafana). The extension framework has been removed in TKG 1.4 and all of these applications are now installed as “Packages” The package framework makes great use of the kapp-controller and should ease installation and lifecycle management going forward. You can read more about kapp-controller and packages in general at https://carvel.dev/kapp-controller/.
You can see what extensions are installed via the kubectl get apps command (run from the context of the cluster in question):
kubectl get apps -A
NAMESPACE NAME DESCRIPTION SINCE-DEPLOY AGE
tanzu-system-ingress contour Reconcile succeeded 57s 161d
tanzu-system-logging fluent-bit Reconcile succeeded 57s 160d
tanzu-system-monitoring grafana Reconcile succeeded 80s 160d
tanzu-system-monitoring prometheus Reconcile succeeded 9s 160d
tanzu-system-registry harbor Reconcile succeeded 98s 160d
tanzu-system-service-discovery external-dns Reconcile succeeded 59s 161d
tkg-system antrea Reconcile succeeded 4m7s 161d
tkg-system load-balancer-and-ingress-service Reconcile succeeded 1m8s 3m4s
tkg-system metrics-server Reconcile succeeded 41s 161d
tkg-system pinniped Reconcile succeeded 4m46s 161d
tkg-system vsphere-cpi Reconcile succeeded 61s 161d
tkg-system vsphere-csi Reconcile succeeded 3m22s 161dYou’ll need to make a note of value in the NAME column as it will be used later when we install the replacement package.
To determine what packages are available, you can use the new tanzu package command:
tanzu package available list -A
/ Retrieving available packages...
NAME DISPLAY-NAME SHORT-DESCRIPTION NAMESPACE
cert-manager.tanzu.vmware.com cert-manager Certificate management tanzu-package-repo-global
contour.tanzu.vmware.com Contour An ingress controller tanzu-package-repo-global
external-dns.tanzu.vmware.com external-dns This package provides DNS synchronization functionality. tanzu-package-repo-global
fluent-bit.tanzu.vmware.com fluent-bit Fluent Bit is a fast Log Processor and Forwarder tanzu-package-repo-global
grafana.tanzu.vmware.com grafana Visualization and analytics software tanzu-package-repo-global
harbor.tanzu.vmware.com Harbor OCI Registry tanzu-package-repo-global
multus-cni.tanzu.vmware.com multus-cni This package provides the ability for enabling attaching multiple network interfaces to pods in Kubernetes tanzu-package-repo-global
prometheus.tanzu.vmware.com prometheus A time series database for your metrics tanzu-package-repo-global
addons-manager.tanzu.vmware.com tanzu-addons-manager This package provides TKG addons lifecycle management capabilities. tkg-system
ako-operator.tanzu.vmware.com ako-operator NSX Advanced Load Balancer using ako-operator tkg-system
antrea.tanzu.vmware.com antrea networking and network security solution for containers tkg-system
calico.tanzu.vmware.com calico Networking and network security solution for containers. tkg-system
kapp-controller.tanzu.vmware.com kapp-controller Kubernetes package manager tkg-system
load-balancer-and-ingress-service.tanzu.vmware.com load-balancer-and-ingress-service Provides L4+L7 load balancing for TKG clusters running on vSphere tkg-system
metrics-server.tanzu.vmware.com metrics-server Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines. tkg-system
pinniped.tanzu.vmware.com pinniped Pinniped provides identity services to Kubernetes. tkg-system
vsphere-cpi.tanzu.vmware.com vsphere-cpi The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management. Cluster API Provider for vSphere is a concrete implementation of Cluster API for vSphere. tkg-system
vsphere-csi.tanzu.vmware.com vsphere-csi vSphere CSI provider tkg-systemThe tanzu package command is welcome addition and doesn’t rely on separate manifest files like the extension framework did. The list of packages is maintained by VMware and updated as new versions or existing packages or new packages are added. To see what packages are currently installed you can run the tanzu package installed list command:
tanzu package installed list -A
/ Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system
load-balancer-and-ingress-service load-balancer-and-ingress-service.tanzu.vmware.com Reconcile succeeded tkg-system
metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system
pinniped pinniped.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-systemThese are all packages that were installed/upgraded during the upgrade to version 1.4 and all reside in the tkg-system namespace.
cert-manager
You should start with cert-manager as it is needed for most of the other extensions.
First check to see what version is available to install:
tanzu package available list cert-manager.tanzu.vmware.com -A
/ Retrieving package versions for cert-manager.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 2020-11-24T18:00:00Z tanzu-package-repo-globalWith the version available, the cert-manager package can be installed. cert-manager was not an app in 1.3, only a deployment, hence it not showing up in the kubectl get apps -A output earlier. The only thing we need to be sure of with the cert-manger package installation is to install the package to the same namespace where cert-manger is currently installed (cert-manager in this example).
tanzu package install cert-manager -p cert-manager.tanzu.vmware.com -n cert-manager -v 1.1.0+vmware.1-tkg.2
/ Installing package 'cert-manager.tanzu.vmware.com'
| Getting namespace 'cert-manager'
| Getting package metadata for 'cert-manager.tanzu.vmware.com'
| Creating service account 'cert-manager-cert-manager-sa'
| Creating cluster admin role 'cert-manager-cert-manager-cluster-role'
| Creating cluster role binding 'cert-manager-cert-manager-cluster-rolebinding'
- Creating package resource
| Package install status: Reconciling
Added installed package 'cert-manager' in namespace 'cert-manager'And that’s it for cert-manger…very easy. You can check the kubectl get apps -A output to see that cert-manager is now listed
kubectl get apps -A
NAMESPACE NAME DESCRIPTION SINCE-DEPLOY AGE
cert-manager cert-manager Reconcile succeeded 24s 9m22s
tanzu-system-ingress contour Reconcile succeeded 57s 161d
tanzu-system-logging fluent-bit Reconcile succeeded 57s 160d
tanzu-system-monitoring grafana Reconcile succeeded 80s 160d
tanzu-system-monitoring prometheus Reconcile succeeded 9s 160d
tanzu-system-registry harbor Reconcile succeeded 98s 160d
tanzu-system-service-discovery external-dns Reconcile succeeded 59s 161d
tkg-system antrea Reconcile succeeded 4m7s 161d
tkg-system load-balancer-and-ingress-service Reconcile succeeded 1m8s 6m52s
tkg-system metrics-server Reconcile succeeded 41s 161d
tkg-system pinniped Reconcile succeeded 4m46s 161d
tkg-system vsphere-cpi Reconcile succeeded 61s 161d
tkg-system vsphere-csi Reconcile succeeded 3m22s 161dAnd the tanzu package installed list -A command shows it as well:
tanzu package installed list -A
- Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded cert-manager
antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system
load-balancer-and-ingress-service load-balancer-and-ingress-service.tanzu.vmware.com Reconcile succeeded tkg-system
metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system
pinniped pinniped.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-systemThe pods in the cert-manager namespace were recreated during this process.
For the rest of the extensions, the process is a little more complicated as you likely have made customizations to the default deployment (do you remember modifying those data-values.yaml files?). You’ll need to make sure to obtain a copy of the data-values secret for each extension prior to installing the new package as you will need to use the same customizaios.
Contour
You need to remove the Contour extension so that the legacy extension framework does not try to reconcile Contour while the package framework is also reconciling it. This will have no effect on the running Contour application.
kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/ingress/contour/contour-extension.yaml
extension.clusters.tmc.cloud.vmware.com "contour" deletedYou will need to get the available version first as was done for cert-manager.
tanzu package available list contour.tanzu.vmware.com -A
- Retrieving package versions for contour.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 2021-07-23T18:00:00Z tanzu-package-repo-globalYou can run the following command to get the data-values secret for Contour:
kubectl get secret contour-data-values -n tanzu-system-ingress -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-contour-data-values.yamlAnd in my case, this returned the following:
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
infrastructure_provider: "vsphere"
contour:
image:
repository: projects.registry.vmware.com/tkg
envoy:
image:
repository: projects.registry.vmware.com/tkg
service:
type: "LoadBalancer"The one change from the default that I made here was to set the service type to LoadBalancer instead of NodePort. With this information, I can create the needed configuration file for deploying the Contour package.
From the documentation, the default configuration file for deploying Contour with NSX Advanced Load Balancer in the mix is the following:
---
infrastructure_provider: vsphere
namespace: tanzu-system-ingress
contour:
configFileContents: {}
useProxyProtocol: false
replicas: 2
pspNames: "vmware-system-restricted"
logLevel: info
envoy:
service:
type: LoadBalancer
annotations: {}
nodePorts:
http: null
https: null
externalTrafficPolicy: Cluster
disableWait: false
hostPorts:
enable: true
http: 80
https: 443
hostNetwork: false
terminationGracePeriodSeconds: 300
logLevel: info
pspNames: null
certificates:
duration: 8760h
renewBefore: 360hYou can see that the service type is already set to LoadBalancer so there is no need to make any changes. If you have installed the 1.3 extension to a different namespace than the default, you’ll want to update that in this manifest.
There is a slick command noted in the Harbor docs for pulling down the Harbor package files so that you can get a clean copy of the Harbor data-values file for editing. You can use a similar process for all packages if you prefer. For example, the following will pull down the Contour package files:
imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages contour.tanzu.vmware.com.1.17.1+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/contour-package-1.17.1
Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/contour@sha256:73dc13131e6c1cfa8d3b56aeacd97734447acdf1ab8c0862e936623ca744e7c4'
Extracting layer 'sha256:93c1f3e88f0e0181e11a38a4e04ac16c21c5949622917b6c72682cc497ab3e44' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
SucceededYou can then navigate to the /tmp/contour-package-1.17.1/config folder and make a copy of the values.yaml (contour-data-values.yaml) and edit it:
#@data/values
---
infrastructure_provider: vsphere
namespace: tanzu-system-ingress
contour:
configFileContents: {}
useProxyProtocol: false
replicas: 2
pspNames: "vmware-system-restricted"
logLevel: info
envoy:
service:
type: null
annotations: {}
nodePorts:
http: null
https: null
externalTrafficPolicy: Cluster
aws:
LBType: classic
disableWait: false
hostPorts:
enable: true
http: 80
https: 443
hostNetwork: false
terminationGracePeriodSeconds: 300
logLevel: info
pspNames: null
certificates:
duration: 8760h
renewBefore: 360hIn my case, only a few changes are needed:
envoy.service.type: nullneeds to beenvoy.service.type: LoadBalancerenvoy.service.awsandenvoy.service.aws.LBType: classicneed to be removed
The last thing to do with the data-values file is to remove any comments:
yq -i eval '... comments=""' contour-data-values.yamlAfter creating a file with the needed contents, you can install the package:
tanzu package install contour -p contour.tanzu.vmware.com -v 1.17.1+vmware.1-tkg.1 -f contour-data-values.yaml -n tanzu-system-ingress
- Installing package 'contour.tanzu.vmware.com'
| Getting namespace 'tanzu-system-ingress'
\ Getting package metadata for 'contour.tanzu.vmware.com'
/ Creating service account 'contour-tanzu-system-ingress-sa'
| Creating cluster admin role 'contour-tanzu-system-ingress-cluster-role'
| Creating cluster role binding 'contour-tanzu-system-ingress-cluster-rolebinding'
| Creating secret 'contour-tanzu-system-ingress-values'
- Creating package resource
| Package install status: Reconciling
Added installed package 'contour' in namespace 'tanzu-system-ingress'As with the cert-manager package, you’ll see all of the Contour pods getting deleted and recreated but Contour’s configuration and functionality should be unchanged after the upgrade.
Harbor
Harbor turned out to be one of the more involved items to upgrade to a package given then number of configuration changes that were made when I initially configured it.
As with Contour, you need to remove the Harbor extension so that the legacy extension framework does not try to reconcile Harbor while the package framework is also reconciling it.
kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/registry/harbor/harbor-extension.yaml
extension.clusters.tmc.cloud.vmware.com "harbor" deletedGet the available version(s) of the Harbor package:
tanzu package available list harbor.tanzu.vmware.com -A
\ Retrieving package versions for harbor.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
harbor.tanzu.vmware.com 2.2.3+vmware.1-tkg.1 2021-07-07T18:00:00Z tanzu-package-repo-globalAs with the other two extensions that have been upgraded to packages already, you need get a copy of the data-values secret so that it can be used when configuring the new package.
kubectl get secret harbor-data-values -n tanzu-system-registry -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-harbor-data-values.yamlPer my earlier comment, you can see that there is a lot in this secret and a several items that have custom values:
tkg-1.3-harbor-data-values.yaml#@data/values
#@overlay/match-child-defaults missing_ok=True
---
# Docker images setting
image:
repository: projects.registry.vmware.com/tkg/harbor
tag: v2.1.3_vmware.1
pullPolicy: IfNotPresent
# The namespace to install Harbor
namespace: tanzu-system-registry
# The FQDN for accessing Harbor admin UI and Registry service.
hostname: harbor.corp.tanzu
# The network port of the Envoy service in Contour or other Ingress Controller.
port:
https: 443
# [Optional] The certificate for the ingress if you want to use your own TLS certificate.
# We will issue the certificate by cert-manager when it's empty.
tlsCertificate:
# [Required] the certificate
tls.crt: |
-----BEGIN CERTIFICATE-----
MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
+LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
R22CcKK+kduUjv0X
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
+QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
+CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
-----END PRIVATE KEY-----
# [Optional] the certificate of CA, this enables the download
# link on portal to download the certificate of CA
ca.crt: |
-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
MRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3JwMRkw
FwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIwMDgxOTE3MjA0NFoXDTMwMDgx
OTE3MzAzNVowSDEVMBMGCgmSJomT8ixkARkWBXRhbnp1MRQwEgYKCZImiZPyLGQB
GRYEY29ycDEZMBcGA1UEAxMQQ09OVFJPTENFTlRFUi1DQTCCAiIwDQYJKoZIhvcN
AQEBBQADggIPADCCAgoCggIBALKIdX7643PzvtVXlqNIwDuNq+rhcHF0fjR414j+
1IGQUuXrykjhSDthPP+8BGN7mBgHT8AjAS1b95xc8B0S2Fhln3AoRE9z03GtfsBu
FSBRUVwAifX6oXu97WzffhqPtxZfLJXbhOomjlkX6iffAs2TOLUx2Oj4w2vybhzj
lcA70ai+0Sl6axSo3lMZ4KkuZ2WgfEcaDjjj33/pV3/bnFK+7ydPttc2Tek5xsI8
XNMirIVxUiUT4YLy4WLiS200JUfbp1ZnMvnbQ8Jv1QnZl9W7WmBPcgxR4AAub0K4
vZLXu6MXiboTlzkMB/YthCkTNlJcKkhHf60YR/T6Sx1T2nupyBa4deo5UGPzhRiJ
pN37uqqAdK1qMDpCjARjS6U7Lf9JKjfiriLzLeyAjP8kaN4TdHSZd0pcQoZSxexQ
9n+4E4MQm4EJ4DrVZCilsyL2BdETcHXKPc7q+Db4XM7jPKNG5GP1EMV4Xohv58yZ
/rRfmK64gar8AMnOKT2AP681qdZs7lljONcXUALzlX5TqIchYT0DVQmFLYoMBeZz
0l21QjbK0YWnPza6Yi/N4m6rFbEB4WXiqhYSkxzrMvocVUgd4AAP1vfHNnFEsnUR
nSsiglFH/xlyO3cBFrmoZAxbA2091XHWhB4c0mQEI3hOqAB8UoFGBrQpmQ+LesoC
1LZ9AgMBAAGjUTBPMAsGA1UdDwQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
DgQWBBTFAxSvY64Q5adhm8IYecHBAUuobzAQBgkrBgEEAYI3FQEEAwIBADANBgkq
hkiG9w0BAQsFAAOCAgEAjg/v4mIP7gBVCw4pemtGn3PStDh/aB9vbWyjAyxSNaaH
H0nID5q5wow9ueBiDfjTPnhbf3P768HG8oL/+9C+Vm/0liFBd+0/DaayKpANFMLB
BV+s2adWRhQucLQfXPwum8RybWv82wkRkWCCdOBaAvAMuTgk08SwJIyQfVgpk3nY
0OwjFwSAadvevf+LoD/9L8R9NEt/n4WJe+LtEamo9EVb+l+cYqyxyubAVY0Y6BM2
GXqAh3FEW2aQMpwouh/5S7w5oSMYN6miY1ojki8gPm0+4+CILPWh/fr2q0O/bPtb
Tr++nPMmZ8ov9epNGIuqhtk5ja2/JuY+RW46IRc8QpF1EyUae02E6U2Vacs7Gge2
CeSINkoLFFmiKBfIn/HAchlme9aL6DlJ9wAreBDH3E8kH7gRDWbSK2/QD0Hqac+E
geGHwpg/8OtBOHUMnM7eLOXBJFcJosWf0XnEgS4ubgaHgqDEu8p8PE7rpCxtUNur
t+x2xONI/rBWgdbp51lPr7o819zPJCvYZq1Pp1st8fb3RlUSWvbQMPFtGAyaBy+G
0RgZ9WPtyEYgnHAb5/Dq46sne9/QnPwwGpjv1s1oE3ZFQjhvnGis8+dqRxk3YZAk
yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
-----END CERTIFICATE-----
# Use contour http proxy instead of the ingress when it's true
enableContourHttpProxy: true
# [Required] The initial password of Harbor admin.
harborAdminPassword: VMware1!
# [Required] The secret key used for encryption. Must be a string of 16 chars.
secretKey: VMware1!VMware1!
database:
# [Required] The initial password of the postgres database.
password: VMware1!VMware1!
core:
replicas: 1
# [Required] Secret is used when core server communicates with other components.
secret: VMware1!VMware1!
# [Required] The XSRF key. Must be a string of 32 chars.
xsrfKey: VMware1!VMware1!VMware1!VMware1!
jobservice:
replicas: 1
# [Required] Secret is used when job service communicates with other components.
secret: VMware1!VMware1!
registry:
replicas: 1
# [Required] Secret is used to secure the upload state from client
# and registry storage backend.
# See: https://github.com/docker/distribution/blob/master/docs/configuration.md#http
secret: VMware1!
notary:
# Whether to install Notary
enabled: true
clair:
# Whether to install Clair scanner
enabled: true
replicas: 1
# The interval of clair updaters, the unit is hour, set to 0 to
# disable the updaters
updatersInterval: 12
trivy:
# enabled the flag to enable Trivy scanner
enabled: true
replicas: 1
# gitHubToken the GitHub access token to download Trivy DB
gitHubToken: ""
# skipUpdate the flag to disable Trivy DB downloads from GitHub
#
# You might want to set the value of this flag to `true` in test or CI/CD environments to avoid GitHub rate limiting issues.
# If the value is set to `true` you have to manually download the `trivy.db` file and mount it in the
# `/home/scanner/.cache/trivy/db/trivy.db` path.
skipUpdate: false
# The persistence is always enabled and a default StorageClass
# is needed in the k8s cluster to provision volumes dynamicly.
# Specify another StorageClass in the "storageClass" or set "existingClaim"
# if you have already existing persistent volumes to use
#
# For storing images and charts, you can also use "azure", "gcs", "s3",
# "swift" or "oss". Set it in the "imageChartStorage" section
persistence:
persistentVolumeClaim:
registry:
# Use the existing PVC which must be created manually before bound,
# and specify the "subPath" if the PVC is shared with other components
existingClaim: ""
# Specify the "storageClass" used to provision the volume. Or the default
# StorageClass will be used(the default).
# Set it to "-" to disable dynamic provisioning
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 20Gi
jobservice:
existingClaim: ""
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
database:
existingClaim: ""
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
redis:
existingClaim: ""
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
trivy:
existingClaim: ""
storageClass: ""
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
# Define which storage backend is used for registry and chartmuseum to store
# images and charts. Refer to
# https://github.com/docker/distribution/blob/master/docs/configuration.md#storage
# for the detail.
imageChartStorage:
# Specify whether to disable `redirect` for images and chart storage, for
# backends which not supported it (such as using minio for `s3` storage type), please disable
# it. To disable redirects, simply set `disableredirect` to `true` instead.
# Refer to
# https://github.com/docker/distribution/blob/master/docs/configuration.md#redirect
# for the detail.
disableredirect: false
# Specify the "caBundleSecretName" if the storage service uses a self-signed certificate.
# The secret must contain keys named "ca.crt" which will be injected into the trust store
# of registry's and chartmuseum's containers.
# caBundleSecretName:
# Specify the type of storage: "filesystem", "azure", "gcs", "s3", "swift",
# "oss" and fill the information needed in the corresponding section. The type
# must be "filesystem" if you want to use persistent volumes for registry
# and chartmuseum
type: filesystem
filesystem:
rootdirectory: /storage
#maxthreads: 100
azure:
accountname: accountname # required
accountkey: base64encodedaccountkey # required
container: containername # required
realm: core.windows.net # optional
gcs:
bucket: bucketname # required
# The base64 encoded json file which contains the key
encodedkey: base64-encoded-json-key-file # optional
rootdirectory: null # optional
chunksize: 5242880 # optional
s3:
region: us-west-1 # required
bucket: bucketname # required
accesskey: null # eg, awsaccesskey
secretkey: null # eg, awssecretkey
regionendpoint: null # optional, eg, http://myobjects.local
encrypt: false # optional
keyid: null # eg, mykeyid
secure: true # optional
v4auth: true # optional
chunksize: null # optional
rootdirectory: null # optional
storageclass: STANDARD # optional
swift:
authurl: https://storage.myprovider.com/v3/auth
username: username
password: password
container: containername
region: null # eg, fr
tenant: null # eg, tenantname
tenantid: null # eg, tenantid
domain: null # eg, domainname
domainid: null # eg, domainid
trustid: null # eg, trustid
insecureskipverify: null # bool eg, false
chunksize: null # eg, 5M
prefix: null # eg
secretkey: null # eg, secretkey
accesskey: null # eg, accesskey
authversion: null # eg, 3
endpointtype: null # eg, public
tempurlcontainerkey: null # eg, false
tempurlmethods: null # eg
oss:
accesskeyid: accesskeyid
accesskeysecret: accesskeysecret
region: regionname
bucket: bucketname
endpoint: null # eg, endpoint
internal: null # eg, false
encrypt: null # eg, false
secure: null # eg, true
chunksize: null # eg, 10M
rootdirectory: null # eg, rootdirectory
# The http/https network proxy for clair, core, jobservice, trivy
proxy:
httpProxy:
httpsProxy:
noProxy: 127.0.0.1,localhost,.local,.internalYou can use the imgpkg command to pull down the default configuration information needed for deploying the Harbor package. In this example, I’m using the version information obtained earlier to pull the correct version and to create a directory under /tmp to store it.
imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages harbor.tanzu.vmware.com.2.2.3+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/harbor-package-2.2.3
Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/harbor@sha256:c4fe69a762c3081e2f181dead7d79224b7012db8b74e3ca847798f030f1749ee'
Extracting layer 'sha256:daca49ca6a24c1660418741b7c5dbb786cfadca9b25a6161e85ac7713d931fcc' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
SucceededNow you need to make a copy of the /tmp/harbor-package-2.2.3/config/values.yaml file (harbor-data-values.yaml) and update it with the information from the tkg-1.3-harbor-data-values.yaml file that was created from the Harbor data-values secret earlier. Your installation will likely differ from mine but the following are the items I had to set:
- hostname
- tlsCertificate.tls.crt
- tlsCertificate.tls.key
- tlsCertificate.ca.crt
- harborAdminPassword
- secretKey
- database.password
- core.secret
- core.xsrfKey
- jobservice.secret
- registry.secret
- persistence.persistentVolumeClaim.registry.size
There are numerous other items that can be configured so please be sure to read the Harbor package documentation to make sure you’re not missing anything.
The last thing to do with the data-values file is to remove any comments:
yq -i eval '... comments=""' harbor-data-values.yamlOnce you’re satisfied with your data-values file, you can proceed with deploying the Harbor package:
tanzu package install harbor -p harbor.tanzu.vmware.com -v 2.2.3+vmware.1-tkg.1 -f harbor-data-values.yaml -n tanzu-system-registry
- Installing package 'harbor.tanzu.vmware.com'
| Getting namespace 'tanzu-system-registry'
| Getting package metadata for 'harbor.tanzu.vmware.com'
| Creating service account 'harbor-tanzu-system-registry-sa'
| Creating cluster admin role 'harbor-tanzu-system-registry-cluster-role'
| Creating cluster role binding 'harbor-tanzu-system-registry-cluster-rolebinding'
| Creating secret 'harbor-tanzu-system-registry-values'
- Creating package resource
| Package install status: Reconciling
Please consider using 'tanzu package installed update' to update the installed package with correct settings
Error: timed out waiting for the conditionIt took a bit longer than the tanzu package install command was willing to wait the first time around and the command failed, even though the reconciliation was still going on in the background. I was able to check this by running the tanzu package installed get command:
tanzu package installed get harbor -n tanzu-system-registry
| Retrieving installation details for harbor...
NAME: harbor
PACKAGE-NAME: harbor.tanzu.vmware.com
PACKAGE-VERSION: 2.2.3+vmware.1-tkg.1
STATUS: Reconciling
CONDITIONS: [{Reconciling True }]And by looking at the events in the tanzu-system-registry namespace (kubectl -n tanzu-system-registry get events -w) I was able to see that there was still a lot of activity with creating pods and attaching persistent volumes. Once this was all done, the reconciliation of the Harbor package completed successfully.
Another thing to look out for since I saw it as well…as of the time of this writing, I could not get Harbor to completely reconcile with Notary enabled. The status of the app and package were both “reconciling” and harbor-notary-signer pod was in a CrashLoopBackOff state. If you take a look at the logs for this pod, you’ll see entries similar to:
{"level":"fatal","msg":"Could not read config at :/etc/notary/server-config.postgres.json, viper error: open : no such file or directory","time":"2021-09-11T16:14:43Z"}If you encounter the same thing and can live without Notary, you can update the Harbor data-values file and set notary.enabled to false and then update the package via the following command:
tanzu package installed update harbor -p harbor.tanzu.vmware.com -v 2.2.3+vmware.1-tkg.1 -f harbor-data-values.yaml -n tanzu-system-registryAnd lastly regarding Notary, one of my coworkers came up with this overlay that will allow you to run Notary successfully in TKG 1.4 (until this issue is permanently fixed):
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.and_op(overlay.subset({"kind": "Deployment"}), overlay.subset({"metadata": {"name": "harbor-notary-signer"}}))
---
spec:
template:
spec:
containers:
#@overlay/match by="name",expects="0+"
- name: notary-signer
image: projects.registry.vmware.com/tkg/harbor/notary-signer-photon@sha256:4dfbf3777c26c615acfb466b98033c0406766692e9c32f3bb08873a0295e24d1You can save this overlay as overlay-notary-signer-image-fix.yaml. You’ll need to create a secret with this overlay and then update the installed package, per the following two commands:
kubectl -n tanzu-system-registry create secret generic harbor-notary-singer-image-overlay -o yaml --dry-run=client --from-file=overlay-notary-signer-image-fix.yaml | kubectl apply -f -kubectl -n tanzu-system-registry annotate packageinstalls harbor ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=harbor-notary-singer-image-overlayYou’ll see that the Notary pod is running fine with this workaround in place:
kubectl -n tanzu-system-registry get po
NAME READY STATUS RESTARTS AGE
harbor-core-6c66bc897b-9srkh 1/1 Running 0 3m12s
harbor-database-0 1/1 Running 0 17h
harbor-jobservice-599f69b686-wdvqx 1/1 Running 0 17h
harbor-notary-server-cc8b5f5c9-j9tkv 1/1 Running 0 3m12s
harbor-notary-signer-7457dd5946-679m6 1/1 Running 0 3m12s
harbor-portal-5c9bf895d7-fvd4r 1/1 Running 0 17h
harbor-redis-0 1/1 Running 0 17h
harbor-registry-697cb78c56-9mrw7 2/2 Running 0 17h
harbor-trivy-0 1/1 Running 0 17hFluent-Bit
As with the previous extensions, you need to remove the Fluent-Bit extension so that the legacy extension framework does not try to reconcile Fluent-Bit while the package framework is also reconciling it. Since Fluent-Bit has no stateful data, you can actually remove it entirely and then reinstall it in the package format. You will need to get the contents of the current data-values secret first though.
kubectl get secret fluent-bit-data-values -n tanzu-system-logging -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-fluent-bit-data-values.yamlAnd in my case, this returned the following (syslog configuration, sending data to vRealize Log Insight):
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
logging:
image:
repository: projects.registry.vmware.com/tkg
tkg:
instance_name: "tkg-mgmt"
cluster_name: "tkg-wld"
fluent_bit:
output_plugin: "syslog"
syslog:
host: "vrli-01a.corp.tanzu"
port: "514"
mode: "udp"
format: "rfc5424"Removing the extension is fairly simple:
kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/logging/fluent-bit/fluent-bit-extension.yaml
extension.clusters.tmc.cloud.vmware.com "fluent-bit" deleted
kubectl delete app fluent-bit -n tanzu-system-logging
app.kappctrl.k14s.io "fluent-bit" deleted
kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/logging/fluent-bit/namespace-role.yaml
namespace "tanzu-system-logging" deleted
serviceaccount "fluent-bit-extension-sa" deleted
role.rbac.authorization.k8s.io "fluent-bit-extension-role" deleted
rolebinding.rbac.authorization.k8s.io "fluent-bit-extension-rolebinding" deleted
clusterrole.rbac.authorization.k8s.io "fluent-bit-extension-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "fluent-bit-extension-cluster-rolebinding" deletedYou will need to get the available Fluent-Bit version as was done for the previous packages.
tanzu package available list fluent-bit.tanzu.vmware.com -A
- Retrieving package versions for fluent-bit.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
fluent-bit.tanzu.vmware.com 1.7.5+vmware.1-tkg.1 2021-05-13T18:00:00Z tanzu-package-repo-globalYou can use the imgpkg command to pull down the default configuration information needed for deploying the Fluent-Bit package. In this example, I’m using the version information obtained earlier to pull the correct version and to create a directory under /tmp to store it.
imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages fluent-bit.tanzu.vmware.com.1.7.5+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/fluent-bit-package-1.7.5
Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/fluent-bit@sha256:c83d038c57f244aae2819dd77dc5184bb3e1ec96524d3f6a09fe8a244b7bc9e4'
Extracting layer 'sha256:48456e4452a35786ed148a3f1a9ede1427f8fccb0079a0eb0904820211b6cebd' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
SucceededYou can then navigate to the /tmp/fluent-bit-package-1.7.5/config folder and make a copy of the values.yaml file (fluent-bit-data-values.yaml) and edit it. The following is what this file looks like by default:
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
namespace: "tanzu-system-logging"
#! Required params for supported output plugins
fluent_bit:
config:
#! https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/variables
service: |
[Service]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
outputs: |
[OUTPUT]
Name stdout
Match *
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name modify
Match *
Rename message text
parsers: |
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
streams: ""
plugins: ""
#! optional configuration for the daemonset
daemonset:
resources: { }
#! limits:
#! cpu: 100m
#! memory: 128Mi
#! requests:
#! cpu: 100m
#! memory: 128Mi
podAnnotations: { }
podLabels: { }
Unfortunately, this default file will result in a fluent-bit implementation that doesn’t do very much. I did a comparison between what gets deployed in the TKG 1.3 Fluent-Bit pods and the TKG 1.4 Fluent-Bit pods and came up with the following changes to the default config file:
- Add the following to the
outputs:section (this was very specific to my configuration, you will need to update this based on your specific output type):
[OUTPUT]
Name syslog
Match kube.*
Host vrli-01a.corp.tanzu
Port 514
Mode udp
Syslog_Format rfc5424
Syslog_Hostname_key tkg_cluster
Syslog_Appname_key pod_name
Syslog_Procid_key container_name
Syslog_Message_key message
Syslog_SD_key k8s
Syslog_SD_key labels
Syslog_SD_key annotations
Syslog_SD_key tkg
[OUTPUT]
Name syslog
Match kube_systemd.*
Host vrli-01a.corp.tanzu
Port 514
Mode udp
Syslog_Format rfc5424
Syslog_Hostname_key tkg_cluster
Syslog_Appname_key tkg_instance
Syslog_Message_key MESSAGE
Syslog_SD_key systemd- Remove the following from the
filters:section:
[FILTER]
Name modify
Match *
Rename message text- Add the following to the
filters:section:
[FILTER]
Name record_modifier
Match *
Record tkg_cluster tkg-wld
Record tkg_instance tkg-mgmt
[FILTER]
Name nest
Match kube.*
Operation nest
Wildcard tkg_instance*
Nest_Under tkg
[FILTER]
Name nest
Match kube_systemd.*
Operation nest
Wildcard SYSTEMD*
Nest_Under systemd
[FILTER]
Name modify
Match kube.*
Copy kubernetes k8s
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_Under kubernetes- Remove the following from the
inputs:section:
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On- Add the following to the
inputs:section:
[INPUT]
Name tail
Tag audit.*
Path /var/log/audit/audit.log
Parser logfmt
DB /var/log/flb_system_audit.db
Mem_Buf_Limit 50MB
Refresh_Interval 10
Skip_Long_Lines On
[INPUT]
Name systemd
Tag kube_systemd.*
Path /var/log/journal
DB /var/log/flb_kube_systemd.db
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Read_From_Tail On
Strip_Underscores On
[INPUT]
Name tail
Tag apiserver_audit.*
Path /var/log/kubernetes/audit.log
Parser json
DB /var/log/flb_kube_audit.db
Mem_Buf_Limit 50MB
Refresh_Interval 10
Skip_Long_Lines On - Add the following to the
parsers:section:
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name docker-daemon
Format regex
Regex time="(?<time>[^ ]*)" level=(?<level>[^ ]*) msg="(?<msg>[^ ].*)"
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name logfmt
Format logfmt
[PARSER]
Name syslog-rfc5424
Format regex
Regex ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name syslog-rfc3164-local
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
Time_Keep On
[PARSER]
Name syslog-rfc3164
Format regex
Regex /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
Time_Key time
Time_Format %b %d %H:%M:%S
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name kube-custom
Format regex
Regex (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$The last thing to do with the data-values file is to remove any comments:
yq -i eval '... comments=""' fluent-bit-data-values.yamlMy completed fluent-bit-data-values.yaml file looks like the following:
namespace: "tanzu-system-logging"
fluent_bit:
config:
service: |
[Service]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
outputs: |
[OUTPUT]
Name stdout
Match *
[OUTPUT]
Name syslog
Match kube.*
Host vrli-01a.corp.tanzu
Port 514
Mode udp
Syslog_Format rfc5424
Syslog_Hostname_key tkg_cluster
Syslog_Appname_key pod_name
Syslog_Procid_key container_name
Syslog_Message_key message
Syslog_SD_key k8s
Syslog_SD_key labels
Syslog_SD_key annotations
Syslog_SD_key tkg
[OUTPUT]
Name syslog
Match kube_systemd.*
Host vrli-01a.corp.tanzu
Port 514
Mode udp
Syslog_Format rfc5424
Syslog_Hostname_key tkg_cluster
Syslog_Appname_key tkg_instance
Syslog_Message_key MESSAGE
Syslog_SD_key systemd
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name tail
Tag audit.*
Path /var/log/audit/audit.log
Parser logfmt
DB /var/log/flb_system_audit.db
Mem_Buf_Limit 50MB
Refresh_Interval 10
Skip_Long_Lines On
[INPUT]
Name systemd
Tag kube_systemd.*
Path /var/log/journal
DB /var/log/flb_kube_systemd.db
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Read_From_Tail On
Strip_Underscores On
[INPUT]
Name tail
Tag apiserver_audit.*
Path /var/log/kubernetes/audit.log
Parser json
DB /var/log/flb_kube_audit.db
Mem_Buf_Limit 50MB
Refresh_Interval 10
Skip_Long_Lines On
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name record_modifier
Match *
Record tkg_cluster tkg-wld
Record tkg_instance tkg-mgmt
[FILTER]
Name nest
Match kube.*
Operation nest
Wildcard tkg_instance*
Nest_Under tkg
[FILTER]
Name nest
Match kube_systemd.*
Operation nest
Wildcard SYSTEMD*
Nest_Under systemd
[FILTER]
Name modify
Match kube.*
Copy kubernetes k8s
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_Under kubernetes
parsers: |
[PARSER]
Name apache
Format regex
Regex ^(?[^ ]*) [^ ]* (?[^ ]*) \[(?
After creating a file with the needed contents, you can install the package:
tanzu package install fluent-bit -p fluent-bit.tanzu.vmware.com -n tanzu-system-logging -v 1.7.5+vmware.1-tkg.1 -f fluent-bit-data-values.yaml --create-namespace
- Installing package 'fluent-bit.tanzu.vmware.com'
| Creating namespace 'tanzu-system-logging'
| Getting package metadata for 'fluent-bit.tanzu.vmware.com'
| Creating service account 'fluent-bit-tanzu-system-logging-sa'
| Creating cluster admin role 'fluent-bit-tanzu-system-logging-cluster-role'
| Creating cluster role binding 'fluent-bit-tanzu-system-logging-cluster-rolebinding'
| Creating secret 'fluent-bit-tanzu-system-logging-values'
\ Creating package resource
| Package install status: Reconciling
Added installed package 'fluent-bit' in namespace 'tanzu-system-logging'External DNS
As with Fluent-Bit, External DNS is also stateless so it can be completely removed prior to installing the new package but you need to get a copy of the data-values secret first.
kubectl get secret external-dns-data-values -n tanzu-system-service-discovery -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-external-dns-data-values.yamlThe following were the contents of this secret:
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
externalDns:
image:
repository: projects.registry.vmware.com/tkg
deployment:
#@overlay/replace
args:
- --txt-owner-id=k8s
- --provider=rfc2136
- --rfc2136-host=192.168.110.10
- --rfc2136-port=53
- --rfc2136-zone=corp.tanzu
- --rfc2136-insecure
- --rfc2136-tsig-axfr
- --source=service
- --source=contour-httpproxy
- --domain-filter=corp.tanzuAs with Fluent-Bit, removing the External DNS extension is three very similar commands:
kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/service-discovery/external-dns/external-dns-extension.yaml
extension.clusters.tmc.cloud.vmware.com "external-dns" deleted
kubectl -n tanzu-system-service-discovery delete app external-dns
app.kappctrl.k14s.io "external-dns" deleted
kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/service-discovery/external-dns/namespace-role.yaml
namespace "tanzu-system-service-discovery" deleted
serviceaccount "external-dns-extension-sa" deleted
role.rbac.authorization.k8s.io "external-dns-extension-role" deleted
rolebinding.rbac.authorization.k8s.io "tanzu-system-service-discovery-rolebinding" deleted
clusterrole.rbac.authorization.k8s.io "external-dns-extension-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "external-dns-extension-cluster-rolebinding" deletedDon’t worry, none of your DNS records created by External DNS will be affected by removing the extension.
Get the available version(s) of the External DNS package:
tanzu package available list external-dns.tanzu.vmware.com -A
- Retrieving package versions for external-dns.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
external-dns.tanzu.vmware.com 0.8.0+vmware.1-tkg.1 2021-06-11T18:00:00Z tanzu-package-repo-globalUse imgpkg to pull down a default data-values file:
imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages external-dns.tanzu.vmware.com.0.8.0+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/external-dns-package-0.8.0
Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/external-dns@sha256:ae8cd950bbeb94b8ff66e41eaf62bcc4470f1f1ab424032a996a0193f5245961'
Extracting layer 'sha256:5227e0d571e7a5c02af43260bac2c783f02f1938e67070784a0e8870b0c80255' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
SucceededWhich returns the following (unhelpful) values.yaml file:
#@data/values
---
#! The namespace in which to deploy ExternalDNS.
namespace: tanzu-system-service-discovery
#! Deployment related configuration
deployment:
#! (REQUIRED) Args passed via command-line to external-dns
#! For more guidance on configuration options for your desired DNS provider, consult the
#! ExternalDNS docs at https://github.com/kubernetes-sigs/external-dns#running-externaldns
args: []
#! Environment variables to pass to external-dns
env: []
#! Security context of the external-dns container
securityContext: {}
#! Volume mounts of the external-dns container
volumeMounts: []
#! Volume of the external-dns pod
volumes: []In this case, you’ll be a lot better off using the various example files noted in the External DNS documentation.
As you would have seen in the original External DNS data-values secret, I’m using rfc2136 DNS, specifically Active Directory DNS. Our existing instructions for External DNS are very bind-specific when it comes to rfc2136 but there are instructions available for Active Directory DNS at Secure Updates Using RFC3645 (GSS-TSIG). I had actually tried to use these steps in the What I wanted to do section of my previous post, How to configure external-dns with Microsoft DNS in TKG 1.3 (plus Harbor and Contour). Due to a now-resolved issue, it didn’t work out but with the newer (0.8.0) version of External DNS, it did.
A configmap is needed that defines the kerberos configuration that External DNS will use…this is fairly generic and the only things I customized were the domain/realm name and the kdc/admin_server addresses:
apiVersion: v1
kind: ConfigMap
metadata:
name: krb5.conf
namespace: tanzu-system-service-discovery
data:
krb5.conf: |
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_ccache_name = KEYRING:persistent:%{uid}
default_realm = CORP.TANZU
[realms]
CORP.TANZU = {
kdc = controlcenter.corp.tanzu
admin_server = controlcenter.corp.tanzu
}
[domain_realm]
corp.tanzu = CORP.TANZU
.corp.tanzu = CORP.TANZUNow you can re-create the tanzu-system-service-discovery namespace and create the noted configmap:
kubectl create ns tanzu-system-service-discovery
namespace/tanzu-system-service-discovery createdkubectl apply -f external-dns-krb5-cm.yaml
configmap/krb5.conf createdAnd the External DNS data-values file looks like the following:
namespace: tanzu-system-service-discovery
deployment:
args:
- --provider=rfc2136
- --rfc2136-host=controlcenter.corp.tanzu
- --rfc2136-port=53
- --rfc2136-zone=corp.tanzu
- --rfc2136-gss-tsig
- --rfc2136-kerberos-realm=corp.tanzu
- --rfc2136-kerberos-username=administrator
- --rfc2136-kerberos-password=VMware1!
- --rfc2136-tsig-axfr
- --source=service
- --source=ingress
- --source=contour-httpproxy
- --domain-filter=corp.tanzu
- --txt-owner-id=k8s
- --txt-prefix=external-dns-
- --registry=txt
- --policy=upsert-only
env: []
securityContext: {}
volumeMounts:
- name: kerberos-config-volume
mountPath: /etc/krb5.conf
subPath: krb5.conf
volumes:
- name: kerberos-config-volume
configMap:
defaultMode: 420
name: krb5.confYou can see from this file that the /etc/krb5.conf file in the external-dns pod will be created from krb5.conf section of the configmap created earlier.
Now you should be ready to deploy the External DNS package:
tanzu package install external-dns -p external-dns.tanzu.vmware.com -n tanzu-system-service-discovery -v 0.8.0+vmware.1-tkg.1 -f external-dns-data-values.yaml
| Installing package 'external-dns.tanzu.vmware.com'
- Creating namespace 'tanzu-system-service-discovery'
/ Getting package metadata for 'external-dns.tanzu.vmware.com'
| Creating service account 'external-dns-tanzu-system-service-discovery-sa'
| Creating cluster admin role 'external-dns-tanzu-system-service-discovery-cluster-role'
| Creating cluster role binding 'external-dns-tanzu-system-service-discovery-cluster-rolebinding'
| Creating secret 'external-dns-tanzu-system-service-discovery-values'
- Creating package resource
- Package install status: Reconciling
Added installed package 'external-dns' in namespace 'tanzu-system-service-discovery'To validate that External DNS was working, I redeployed Harbor while External DNS was uninstalled. The installation was successful but no DNS record was created, as expected. After the External DNS package was installed, I was able to see the following in the logs for the External DNS pod:
time="2021-09-12T20:45:24Z" level=info msg="Instantiating new Kubernetes client"
time="2021-09-12T20:45:24Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-09-12T20:45:24Z" level=info msg="Created Kubernetes client https://100.64.0.1:443"
time="2021-09-12T20:45:26Z" level=info msg="Created Dynamic Kubernetes client https://100.64.0.1:443"
time="2021-09-12T20:45:27Z" level=info msg="Configured RFC2136 with zone 'corp.tanzu.' and nameserver 'controlcenter.corp.tanzu:53'"
time="2021-09-12T20:45:32Z" level=info msg="Adding RR: harbor.corp.tanzu 0 A 192.168.220.2"
time="2021-09-12T20:45:32Z" level=info msg="Adding RR: external-dns-harbor.corp.tanzu 0 TXT \"heritage=external-dns,external-dns/owner=k8s,external-dns/resource=HTTPProxy/tanzu-system-registry/harbor-httpproxy\""Additionally, I was able to set the dynamic update policy in Active Directory back to Secure only and everything still worked as expected.
Prometheus
As with previous packages, you need to remove the Prometheus extension so that the legacy extension framework does not try to reconcile Prometheus while the package framework is also reconciling it.
kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/monitoring/prometheus/prometheus-extension.yaml
extension.clusters.tmc.cloud.vmware.com "prometheus" deletedGet the available version(s) of the Prometheus package:
tanzu package available list prometheus.tanzu.vmware.com -A
- Retrieving package versions for prometheus.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
prometheus.tanzu.vmware.com 2.27.0+vmware.1-tkg.1 2021-05-12T18:00:00Z tanzu-package-repo-global
As with the other extensions that have been upgraded to packages already, you need get a copy of the data-values secret so that it can be used when configuring the new package.
kubectl get secret prometheus-data-values -n tanzu-system-monitoring -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-prometheus-data-values.yamlThe only things that have been customized in here are the hostname and certificate data (I’m using a wildcard certificate) and ingress has been enabled.
tkg-1.3-prometheus-data-values.yaml#@data/values
#@overlay/match-child-defaults missing_ok=True
---
monitoring:
ingress:
enabled: true
virtual_host_fqdn: "prometheus.corp.tanzu"
prometheus_prefix: "/"
alertmanager_prefix: "/alertmanager/"
tlsCertificate:
tls.crt: |
-----BEGIN CERTIFICATE-----
MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
+LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
R22CcKK+kduUjv0X
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
+QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
+CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
-----END PRIVATE KEY-----
prometheus_server:
image:
repository: projects.registry.vmware.com/tkg/prometheus
alertmanager:
image:
repository: projects.registry.vmware.com/tkg/prometheus
kube_state_metrics:
image:
repository: projects.registry.vmware.com/tkg/prometheus
node_exporter:
image:
repository: projects.registry.vmware.com/tkg/prometheus
pushgateway:
image:
repository: projects.registry.vmware.com/tkg/prometheus
cadvisor:
image:
repository: projects.registry.vmware.com/tkg/prometheus
prometheus_server_configmap_reload:
image:
repository: projects.registry.vmware.com/tkg/prometheus
prometheus_server_init_container:
image:
repository: projects.registry.vmware.com/tkg/prometheusYou can use the imgpkg command to pull down the default configuration information needed for deploying the Prometheus package. In this example, I’m using the version information obtained earlier to pull the correct version and to create a directory under /tmp to store it.
imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages prometheus.tanzu.vmware.com.2.27.0+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/prometheus-package-2.27.0
Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/prometheus@sha256:27af034c1c77bcae4e1f7f6d3883286e34419ea2e88222642af17393cd34e46a'
Extracting layer 'sha256:44798ebd112b55ea792f0198cf220a7eaed37c2abc531d6cf8efe89aadc8bff2' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
Succeeded
Now you need to make a copy of the /tmp/prometheus-package-2.27.0/config/values.yaml file (prometheus-data-values.yaml) and update it with the information from the tkg-1.3-prometheus-data-values.yaml file that was created from the Prometheus data-values secret earlier.
The last thing to do with the data-values file is to remove any comments:
yq -i eval '... comments=""' prometheus-data-values.yamlMy final prometheus-data-values.yaml file looked like the following (truncated to just the part I had to modify since it was really long):
ingress:
enabled: true
virtual_host_fqdn: "prometheus.corp.tanzu"
prometheus_prefix: "/"
alertmanager_prefix: "/alertmanager/"
prometheusServicePort: 80
alertmanagerServicePort: 80
tlsCertificate:
tls.crt: |
-----BEGIN CERTIFICATE-----
MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
+LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
R22CcKK+kduUjv0X
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
+QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
+CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
-----END PRIVATE KEY-----
ca.crt:I thought I was ready to install the package but realized that there was a difference between how the Prometheus package would be deployed and how the Prometheus extension was installed. The extension provisioned a PVC of 8GB in size but the package provisions a PVC of 150 GB in size. In order to keep using the same PVC, it would need to be resized. Unfortunately, the implementation of vSphere CSI included with TKG 1.4 does not include the resizer sidecar to the vsphere-csi-controller pod. Luckily, we have documentation for enabling the resizer sidecar at Enable Offline Volume Expansion for vSphere CSI (vSphere 7). And don’t let the title mislead you…if you are running vSphere 7.0 U2 or later, you can actually do online volume expansion as well.
With this in mind, there are a couple of options here. The Prometheus upgrade documentation calls for completely removing the Prometheus extension prior to installing the package…this will certainly avoid the issue of volume expansion but results in all historical Prometheus data being lost. Another option would be to edit the storage: "150Gi" line in the prometheus-data-values.yaml file to be storage: "8Gi". I actually chose to go the route of installing the resizer sidecar and let the PVC be increased to 150GB during the package installation.
At this point, you should be ready to install the package:
tanzu package install prometheus -p prometheus.tanzu.vmware.com -v 2.27.0+vmware.1-tkg.1 -f prometheus-data-values.yaml -n tanzu-system-monitoring
- Installing package 'prometheus.tanzu.vmware.com'
| Getting namespace 'tanzu-system-monitoring'
| Getting package metadata for 'prometheus.tanzu.vmware.com'
| Creating service account 'prometheus-tanzu-system-monitoring-sa'
| Creating cluster admin role 'prometheus-tanzu-system-monitoring-cluster-role'
| Creating cluster role binding 'prometheus-tanzu-system-monitoring-cluster-rolebinding'
| Creating secret 'prometheus-tanzu-system-monitoring-values'
- Creating package resource
\ Package install status: Reconciling
Added installed package 'prometheus' in namespace 'tanzu-system-monitoring'If you choose to go the same route that I did, you can validate that the PVC was resized:
kubectl -n tanzu-system-monitoring get events |grep -i volume
15m Normal Resizing persistentvolumeclaim/prometheus-server External resizer is resizing volume pvc-eac3c942-170b-4ce9-ab9b-36db528dcf71
14m Normal FileSystemResizeRequired persistentvolumeclaim/prometheus-server Require file system resize of volume on node
13m Normal FileSystemResizeSuccessful persistentvolumeclaim/prometheus-server MountVolume.NodeExpandVolume succeeded for volume "pvc-eac3c942-170b-4ce9-ab9b-36db528dcf71"
kubectl -n tanzu-system-monitoring get pvc prometheus-server
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-server Bound pvc-eac3c942-170b-4ce9-ab9b-36db528dcf71 150Gi RWO default 163dGrafana
I had a bit of a hiccup with Grafana while learning that the package/app framework does not work with multiple items installed to the same namespace. The namespace is effectively “owned” by the first app deployed there and subsequent apps will run afoul of ownership errors, similar to the following:
stderr: |-
kapp: Error: Ownership errors:
- Resource 'namespace/tanzu-system-monitoring (v1) cluster' is already associated with a different app 'prometheus-ctrl' namespace: tanzu-system-monitoring (label 'kapp.k14s.io/app=1617374954129473715')The documentation for upgrading Grafana calls out uninstalling Grafana completely before installing the Grafana package (as was noted for Prometheus). The one big difference between the extension and the package is that the package deploys its app to a different namespace….tanzu-system-dashboards. This method would certainly avoid the issue of multiple apps in the same namespace (apparently only a new concern for 1.4). You could follow the instructions and deploy Prometheus and Grafana as documented but lose all historical data or pick one to keep in tanzu-system-monitoring and put the other in a new namespace. I chose to go with leaving Prometheus in the tanzu-system-monitoring namespace and put Grafana in the tanzu-system-dashboards namespace.
As with the other extensions that have been upgraded to packages already, you need get a copy of the data-values secret so that it can be used when configuring the new package.
kubectl get secret grafana-data-values -n tanzu-system-monitoring -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > tkg-1.3-grafana-data-values.yamlAs with Prometheus, the items that have been customized in here are the hostname, certificate data (I’m using a wildcard certificate), admin_password and ingress has been enabled.
tkg-1.3-grafana-data-values.yaml#@data/values
#@overlay/match-child-defaults missing_ok=True
---
monitoring:
grafana:
image:
repository: "projects.registry.vmware.com/tkg/grafana"
ingress:
enabled: true
virtual_host_fqdn: "grafana.corp.tanzu"
prefix: "/"
tlsCertificate:
tls.crt: |
-----BEGIN CERTIFICATE-----
MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
+LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
R22CcKK+kduUjv0X
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
+QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
+CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
-----END PRIVATE KEY-----
secret:
admin_password: "Vk13YXJlMSE="
grafana_init_container:
image:
repository: "projects.registry.vmware.com/tkg/grafana"
grafana_sc_dashboard:
image:
repository: "projects.registry.vmware.com/tkg/grafana"As with the external-dns package, you need to remove the Grafana extension completely.
kubectl -n tanzu-system-monitoring delete app grafana
app.kappctrl.k14s.io "grafana" deleted
kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/monitoring/grafana/namespace-role.yaml
kubectl -n tanzu-system-monitoring delete sa grafana-extension-sa
serviceaccount "grafana-extension-sa" deleted
kubectl -n tanzu-system-monitoring delete role grafana-extension-role
role.rbac.authorization.k8s.io "grafana-extension-role" deleted
kubectl -n tanzu-system-monitoring delete rolebinding grafana-extension-rolebinding
rolebinding.rbac.authorization.k8s.io "grafana-extension-rolebinding" deleted
kubectl delete clusterrole grafana-extension-cluster-role
clusterrole.rbac.authorization.k8s.io "grafana-extension-cluster-role" deleted
kubectl delete clusterrolebinding grafana-extension-cluster-rolebinding
clusterrolebinding.rbac.authorization.k8s.io "grafana-extension-cluster-rolebinding" deletedletedThe documentation calls for running kubectl delete -f ~/tkg-extensions-v1.3.0+vmware.1/extensions/monitoring/grafana/grafana-extension.yaml which will delete all RBAC objects but will also delete the namespace. This would cause problems for Prometheus so I opted to manually delete each RBAC object.
Get the available version(s) of the Grafana package:
tanzu package available list grafana.tanzu.vmware.com -A
- Retrieving package versions for grafana.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
grafana.tanzu.vmware.com 7.5.7+vmware.1-tkg.1 2021-05-19T18:00:00Z tanzu-package-repo-globalYou can use the imgpkg command to pull down the default configuration information needed for deploying the Grafana package. In this example, I’m using the version information obtained earlier to pull the correct version and to create a directory under /tmp to store it.
imgpkg pull -b $(kubectl -n tanzu-package-repo-global get packages grafana.tanzu.vmware.com.7.5.7+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}') -o /tmp/grafana-package-7.5.7
Pulling bundle 'projects.registry.vmware.com/tkg/packages/standard/grafana@sha256:561e6afe31bf627f89dbb92a7dfded25128037fc407efc7d0190f2c6bc6deec7'
Extracting layer 'sha256:a6deb6b8b1e485dce2e45bea45a6055d502b945dc0abd22c47e54b9763c0486b' (1/1)
Locating image lock file images...
One or more images not found in bundle repo; skipping lock file update
SucceededNow you need to make a copy of the /tmp/grafana-package-7.5.7/config/values.yaml file (grafana-data-values.yaml) and update it with the information from the tkg-1.3-prometheus-data-values.yaml file that was created from the Prometheus data-values secret earlier. Some additional notes: the admin_password value has to be base64 encoded before being entered…echo -n "<password>" | base64, the namespace: value needs to be changed from tanzu-system-monitoring to tanzu-system-dashboards and the service.type: value needs to be changed from LoadBalancer to NodePort.
The last thing to do with the data-values file is to remove any comments:
yq -i eval '... comments=""' grafana-data-values.yamlMy final grafana-data-values.yaml file looked like the following:
namespace: tanzu-system-dashboards
grafana:
deployment:
replicas: 1
containers:
resources: {}
podAnnotations: {}
podLabels: {}
k8sSidecar:
containers:
resources: {}
service:
type: NodePort
port: 80
targetPort: 3000
labels: {}
annotations: {}
config:
grafana_ini: |
[analytics]
check_for_updates = false
[grafana_net]
url = https://grafana.com
[log]
mode = console
[paths]
data = /var/lib/grafana/data
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
datasource_yaml: |-
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: prometheus-server.tanzu-system-monitoring.svc.cluster.local
access: proxy
isDefault: true
dashboardProvider_yaml: |-
apiVersion: 1
providers:
- name: 'sidecarDashboardProvider'
orgId: 1
folder: ''
folderUid: ''
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: false
options:
path: /tmp/dashboards
foldersFromFilesStructure: true
pvc:
annotations: {}
storageClassName: null
accessMode: ReadWriteOnce
storage: "2Gi"
secret:
type: "Opaque"
admin_user: "YWRtaW4="
admin_password: "Vk13YXJlMSE="
ingress:
enabled: true
virtual_host_fqdn: "grafana.corp.tanzu"
prefix: "/"
servicePort: 80
tlsCertificate:
tls.crt: |
-----BEGIN CERTIFICATE-----
MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
+LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
R22CcKK+kduUjv0X
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
+QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
+CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
-----END PRIVATE KEY-----
ca.crt:At this point, you should be ready to install the package:
tanzu package install grafana -p grafana.tanzu.vmware.com -v 7.5.7+vmware.1-tkg.1 -f grafana-data-values.yaml -n tanzu-system-dashboards --create-namespace
- Installing package 'grafana.tanzu.vmware.com'
| Getting namespace 'tanzu-system-monitoring'
| Getting package metadata for 'grafana.tanzu.vmware.com'
| Creating service account 'grafana-tanzu-system-monitoring-sa'
| Creating cluster admin role 'grafana-tanzu-system-monitoring-cluster-role'
| Creating cluster role binding 'grafana-tanzu-system-monitoring-cluster-rolebinding'
| Creating secret 'grafana-tanzu-system-monitoring-values'
- Creating package resource
Added installed package 'grafana' in namespace 'tanzu-system-monitoring'Troubleshooting package installation issues
As you can probably guess, I had to troubleshoot a number of issues during the package installation/upgrade process. The following are a few of the commands that I would run to help determine what was going on:
tanzu package installed list -A– this will list the status of all packages and quickly show any that were not health:
tanzu package installed list -A
- Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded cert-manager
grafana grafana.tanzu.vmware.com 7.5.7+vmware.1-tkg.1 Reconcile failed: Error (see .status.usefulErrorMessage for details) tanzu-system-dashboards
contour contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 Reconcile succeeded tanzu-system-ingress
fluent-bit fluent-bit.tanzu.vmware.com 1.7.5+vmware.1-tkg.1 Reconcile succeeded tanzu-system-logging
prometheus prometheus.tanzu.vmware.com 2.27.0+vmware.1-tkg.1 Reconcile succeeded tanzu-system-monitoring
harbor harbor.tanzu.vmware.com 2.2.3+vmware.1-tkg.1 Reconcile succeeded tanzu-system-registry
external-dns external-dns.tanzu.vmware.com 0.8.0+vmware.1-tkg.1 Reconcile succeeded tanzu-system-service-discovery
antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system
load-balancer-and-ingress-service load-balancer-and-ingress-service.tanzu.vmware.com Reconcile succeeded tkg-system
metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system
pinniped pinniped.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-system- tanzu package installed get <package name> -n <namespace> – this will give more details on any installed package and bubble up any relevant errors:
tanzu package installed get grafana -n tanzu-system-monitoring
/ Retrieving installation details for grafana...
NAME: grafana
PACKAGE-NAME: grafana.tanzu.vmware.com
PACKAGE-VERSION: 7.5.7+vmware.1-tkg.1
STATUS: Reconcile failed: Error (see .status.usefulErrorMessage for details)
CONDITIONS: [{ReconcileFailed True Error (see .status.usefulErrorMessage for details)}]
USEFUL-ERROR-MESSAGE: kapp: Error: Ownership errors:
- Resource 'namespace/tanzu-system-monitoring (v1) cluster' is already associated with a different app 'prometheus-ctrl' namespace: tanzu-system-monitoring (label 'kapp.k14s.io/app=1617374954129473715')
kubectl package installed updated <installed package name> -p <public package name> -v <version> -f <configuration file> -n <namespace>– this will allow you to make changes to an already installed package:
tanzu package installed update grafana -p grafana.tanzu.vmware.com -v 7.5.7+vmware.1-tkg.1 -f grafana-data-values.yaml -n tanzu-system-dashboards
| Updating package 'grafana'
- Getting package install for 'grafana'
| Updating secret 'grafana-tanzu-system-dashboards-values'
| Updating package install for 'grafana'
Updated package install 'grafana' in namespace 'tanzu-system-dashboards'
kubectl get app -A– shows the status of the apps deployed via the package framework, should mimic thetanzu package installed list -Aoutput:
kubectl get app -A
NAMESPACE NAME DESCRIPTION SINCE-DEPLOY AGE
cert-manager cert-manager Reconcile succeeded 32s 2d2h
tanzu-system-ingress contour Reconcile succeeded 41s 28h
tanzu-system-logging fluent-bit Reconcile succeeded 4h49m 11h
tanzu-system-monitoring grafana Reconcile failed: Deploying: Error (see .status.usefulErrorMessage for details) 58s 19m
tanzu-system-monitoring prometheus Reconcile succeeded 71s 163d
tanzu-system-registry harbor Reconcile succeeded 74s 120m
tanzu-system-service-discovery external-dns Reconcile succeeded 63s 5h16m
tkg-system antrea Reconcile succeeded 95s 163d
tkg-system load-balancer-and-ingress-service Reconcile succeeded 5m17s 73m
tkg-system metrics-server Reconcile succeeded 5m32s 163d
tkg-system pinniped Reconcile succeeded 2m5s 163d
tkg-system vsphere-cpi Reconcile succeeded 82s 163d
tkg-system vsphere-csi Reconcile succeeded 106s 163d - kubectl get packageinstalls -A – yet another way of looking at installed packages:
kubectl get packageinstalls -A
NAMESPACE NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
cert-manager cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded 31h
tanzu-system-dashboards grafana grafana.tanzu.vmware.com 7.5.7+vmware.1-tkg.1 Reconcile succeeded 26m
tanzu-system-ingress contour contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 Reconcile succeeded 30h
tanzu-system-logging fluent-bit fluent-bit.tanzu.vmware.com 1.7.5+vmware.1-tkg.1 Reconcile succeeded 11h
tanzu-system-monitoring prometheus prometheus.tanzu.vmware.com 2.27.0+vmware.1-tkg.1 Reconcile succeeded 3h34m
tanzu-system-registry harbor harbor.tanzu.vmware.com 2.2.3+vmware.1-tkg.1 Reconcile succeeded 30h
tanzu-system-service-discovery external-dns external-dns.tanzu.vmware.com 0.8.0+vmware.1-tkg.1 Reconcile succeeded 5h21m
tkg-system antrea antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 Reconcile succeeded 31h
tkg-system load-balancer-and-ingress-service load-balancer-and-ingress-service.tanzu.vmware.com 1.4.3+vmware.1-tkg.1 Reconcile succeeded 31h
tkg-system metrics-server metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 Reconcile succeeded 31h
tkg-system pinniped pinniped.tanzu.vmware.com 0.4.4+vmware.1-tkg.1 Reconcile succeeded 31h
tkg-system vsphere-cpi vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 Reconcile succeeded 31h
tkg-system vsphere-csi vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 Reconcile succeeded 31hkubectl get events -n <namespace>– useful for seeing if a particular pod (or other resource) was having some issue while coming up.kubectl get po -n <namespace>– also useful for seeing what pod(s) might not be fully functional.
One last thing that you might notice if you take a closer look at any of the apps that were not newly deployed but upgraded via the package framework. There will be lots of annotations and labels on them related to the older extension framework, per the following example:
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"kappctrl.k14s.io/v1alpha1","kind":"App","metadata":{"annotations":{"tmc.cloud.vmware.com/extension-version":"v2.17.1_vmware.1","tmc.cloud.vmware.com/orphan-resource":"true"},"labels":{"tmc-extension-name":"prometheus","tmc-extension-version":"v2.17.1_vmware.1"},"name":"prometheus","namespace":"tanzu-system-monitoring"},"spec":{"deploy":[{"kapp":{"rawOptions":["--wait-timeout=5m"]}}],"fetch":[{"image":{"url":"projects.registry.vmware.com/tkg/tkg-extensions-templates:v1.3.0_vmware.1"}}],"serviceAccountName":"prometheus-extension-sa","syncPeriod":"5m","template":[{"ytt":{"ignoreUnknownComments":true,"inline":{"pathsFrom":[{"secretRef":{"name":"prometheus-data-values"}}]},"paths":["tkg-extensions/common","tkg-extensions/monitoring/prometheus"]}}]}}
tmc.cloud.vmware.com/extension-version: v2.17.1_vmware.1
tmc.cloud.vmware.com/orphan-resource: "true"
creationTimestamp: "2021-04-02T14:49:10Z"
finalizers:
- finalizers.kapp-ctrl.k14s.io/delete
generation: 4
labels:
tmc-extension-name: prometheus
tmc-extension-version: v2.17.1_vmware.1
name: prometheus
The annotations and finalizers like these can be removed if desired.
Pingback: How to configure external-dns with Microsoft DNS in TKG 1.3 (plus Harbor and Contour) – Little Stuff
Does this guide also apply to Supervisor clusters? “tanzu management-cluster get” does not work for me as there seems to be no management cluster? My test environment works fine, yet im unable to find my management cluster with tanzu cli.
Hello Bart. This was not written with supervisor clusters in mind and was not tested against it but there is no reason you should not be able to use the tanzu cli to register your supervisor cluster and then manager the lifecycle of your Tanzu Kubernetes clusters. https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-tanzu-k8s-clusters-connect-vsphere7.html should provide more information on that front.
Pingback: Migrating a TKG cluster control-plane endpoint from kube-vip to NSX-ALB – Little Stuff
Pingback: Installing a TKG 1.4 management cluster on vSphere with NSX Advanced Load Balancer – Little Stuff
Hi thank you for your blog.
I have a question.. would you have a answer?
i need to ytt test. but i don’t know how to ytt test for tanzu 1.4.
let me know how to ytt test for tanzu 1.4?
Hi im dokyung..
i need to labels but i can’t. because i want not backup statefulset.
thank you