Upgrading a TKGI 1.11 Management Console installation to 1.12

I’ve been through several TKGI installs over the last couple of years but not too many upgrades. The last time I went through one was going from 1.7 to 1.8, as documented in my previous post, A Walk-through of Upgrading Tanzu Kubernetes Grid Integrated Edition (Enterprise PKS) from 1.7 to 1.8. Much has changed since then and I’m happy to see that the process is much simpler with the introduction and maturation of the TKGI Management Console (TKGIMC). Since my last TKGI install, version 1.11, was done with the TKGIMC, I was able to take advantage of the much more streamlined process.

There is a brief overview of the upgrade process via the TKGIMC at Upgrade Tanzu Kubernetes Grid Integrated Edition Management Console but I wanted to go into it in much more detail, hence this post.

As with any upgrade, be sure to take a backup of all components and have a tested disaster recovery plan prior to starting.

Deploy the new TKGIMC OVA

After you’ve downloaded the new TKGIMC OVA from VMware, you can proceed with the upgrade process. In my environment, I have a static IP assigned to my TKGIMC appliance and want to keep using the same one. With that in mind, the old appliance will need to be updated with a new, temporary IP address.

Shutdown the old TKGIMC VM and then navigate to Configure, Settings, vApp Options. You should see the old IP address in the Properties section:

Click the Set Value link with this item selected and enter the new, temporary IP address:

Click the OK button and then you can power the old VM back on. The TKGIMC UI should soon be accessible at the new IP address.

Now you can deploy the new TKGIMC OVA while using your desired IP address.

You’ll want to make sure that you’ve got everything you need for the next few screens so that you can get it entered right the first time. You can change just about anything here after deployment but it’s always nice to get it right the first time:

It’s a pretty decent sized OVA (almost 14GB) so the deployment will likely take a little while to complete.

Once the VM is fully deployed, power it on and log in.

Click the Upgrade button.

Enter the temporary IP address for the old TKGIMC instance and the login credentials and then click the Connect button.

You can see the old and new versions in play on this screen. Note that you’ll have to manually update any Windows stemcells you have uploaded. If everything looks good, click Next to proceed.

If any items are red here you will need to drill down into them to correct any issues or enter appropriate information if a new, mandatory field is present. When you’re ready to proceed, click the Generate Upgrade Configuration button.

You can review the upgrade configuration in detail here if you need to before proceeding. The following is an example of what this upgrade configuration might look like:

upgrade configuration yaml
auth:
  auth_type: ldap
  cluster_client_access_token_lifetime: 600
  cluster_client_refresh_token_lifetime: 21600
  ldap_cert: |-
    -----BEGIN CERTIFICATE-----
    MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
    MRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3JwMRkw
    FwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIwMDgxOTE3MjA0NFoXDTMwMDgx
    OTE3MzAzNVowSDEVMBMGCgmSJomT8ixkARkWBXRhbnp1MRQwEgYKCZImiZPyLGQB
    GRYEY29ycDEZMBcGA1UEAxMQQ09OVFJPTENFTlRFUi1DQTCCAiIwDQYJKoZIhvcN
    AQEBBQADggIPADCCAgoCggIBALKIdX7643PzvtVXlqNIwDuNq+rhcHF0fjR414j+
    1IGQUuXrykjhSDthPP+8BGN7mBgHT8AjAS1b95xc8B0S2Fhln3AoRE9z03GtfsBu
    FSBRUVwAifX6oXu97WzffhqPtxZfLJXbhOomjlkX6iffAs2TOLUx2Oj4w2vybhzj
    lcA70ai+0Sl6axSo3lMZ4KkuZ2WgfEcaDjjj33/pV3/bnFK+7ydPttc2Tek5xsI8
    XNMirIVxUiUT4YLy4WLiS200JUfbp1ZnMvnbQ8Jv1QnZl9W7WmBPcgxR4AAub0K4
    vZLXu6MXiboTlzkMB/YthCkTNlJcKkhHf60YR/T6Sx1T2nupyBa4deo5UGPzhRiJ
    pN37uqqAdK1qMDpCjARjS6U7Lf9JKjfiriLzLeyAjP8kaN4TdHSZd0pcQoZSxexQ
    9n+4E4MQm4EJ4DrVZCilsyL2BdETcHXKPc7q+Db4XM7jPKNG5GP1EMV4Xohv58yZ
    /rRfmK64gar8AMnOKT2AP681qdZs7lljONcXUALzlX5TqIchYT0DVQmFLYoMBeZz
    0l21QjbK0YWnPza6Yi/N4m6rFbEB4WXiqhYSkxzrMvocVUgd4AAP1vfHNnFEsnUR
    nSsiglFH/xlyO3cBFrmoZAxbA2091XHWhB4c0mQEI3hOqAB8UoFGBrQpmQ+LesoC
    1LZ9AgMBAAGjUTBPMAsGA1UdDwQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
    DgQWBBTFAxSvY64Q5adhm8IYecHBAUuobzAQBgkrBgEEAYI3FQEEAwIBADANBgkq
    hkiG9w0BAQsFAAOCAgEAjg/v4mIP7gBVCw4pemtGn3PStDh/aB9vbWyjAyxSNaaH
    H0nID5q5wow9ueBiDfjTPnhbf3P768HG8oL/+9C+Vm/0liFBd+0/DaayKpANFMLB
    BV+s2adWRhQucLQfXPwum8RybWv82wkRkWCCdOBaAvAMuTgk08SwJIyQfVgpk3nY
    0OwjFwSAadvevf+LoD/9L8R9NEt/n4WJe+LtEamo9EVb+l+cYqyxyubAVY0Y6BM2
    GXqAh3FEW2aQMpwouh/5S7w5oSMYN6miY1ojki8gPm0+4+CILPWh/fr2q0O/bPtb
    Tr++nPMmZ8ov9epNGIuqhtk5ja2/JuY+RW46IRc8QpF1EyUae02E6U2Vacs7Gge2
    CeSINkoLFFmiKBfIn/HAchlme9aL6DlJ9wAreBDH3E8kH7gRDWbSK2/QD0Hqac+E
    geGHwpg/8OtBOHUMnM7eLOXBJFcJosWf0XnEgS4ubgaHgqDEu8p8PE7rpCxtUNur
    t+x2xONI/rBWgdbp51lPr7o819zPJCvYZq1Pp1st8fb3RlUSWvbQMPFtGAyaBy+G
    0RgZ9WPtyEYgnHAb5/Dq46sne9/QnPwwGpjv1s1oE3ZFQjhvnGis8+dqRxk3YZAk
    yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
    -----END CERTIFICATE-----
  ldap_cert_altname: ""
  ldap_email_attribute: mail
  ldap_email_domains: ""
  ldap_external_group_whitelist: '*'
  ldap_firstname_attribute: ""
  ldap_group_search_base: cn=Users,dc=corp,dc=tanzu
  ldap_group_search_filter: member={0}
  ldap_lastname_attribute: ""
  ldap_password: VMware1!
  ldap_referrals: follow
  ldap_url: ldaps://controlcenter.corp.tanzu:636
  ldap_user_search_base: cn=Users,dc=corp,dc=tanzu
  ldap_user_search_filter: userPrincipalName={0}
  ldap_username: cn=Administrator,cn=Users,dc=corp,dc=tanzu
  oidc_groups_claim: roles
  oidc_groups_prefix: 'oidc:'
  oidc_username_claim: user_name
  oidc_username_prefix: 'oidc:'
  pks_cli_access_token_lifetime: 600
  pks_cli_refresh_token_lifetime: 21600
  saml_default_identity_provider: false
  saml_display_name: ""
  saml_email_attribute: ""
  saml_external_groups_attribute: ""
  saml_first_name_attribute: ""
  saml_last_name_attribute: ""
  saml_name_id_format: ""
  saml_pks_cli_autoapprove: false
  saml_pks_cluster_client_autoapprove: false
  saml_require_signed_assertions: false
  saml_sign_auth_requests: false
  saml_signature_algorithm: ""
  saml_sso_name: ""
  saml_sso_url: ""
  saml_sso_xml: ""
  uaa_as_cluster_oidc_provider: true
availability_zones:
- for_management_only: true
  name: TKGI-MGMT-1
  resources:
  - resgroup-13006
- for_management_only: false
  name: TKGI-COMP-1
  resources:
  - resgroup-13007
bosh_persistent_disk_type: "51200"
bosh_trust_opsman_ca: false
bosh_trusted_root_certs: |-
  -----BEGIN CERTIFICATE-----
  MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
  MRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3JwMRkw
  FwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIwMDgxOTE3MjA0NFoXDTMwMDgx
  OTE3MzAzNVowSDEVMBMGCgmSJomT8ixkARkWBXRhbnp1MRQwEgYKCZImiZPyLGQB
  GRYEY29ycDEZMBcGA1UEAxMQQ09OVFJPTENFTlRFUi1DQTCCAiIwDQYJKoZIhvcN
  AQEBBQADggIPADCCAgoCggIBALKIdX7643PzvtVXlqNIwDuNq+rhcHF0fjR414j+
  1IGQUuXrykjhSDthPP+8BGN7mBgHT8AjAS1b95xc8B0S2Fhln3AoRE9z03GtfsBu
  FSBRUVwAifX6oXu97WzffhqPtxZfLJXbhOomjlkX6iffAs2TOLUx2Oj4w2vybhzj
  lcA70ai+0Sl6axSo3lMZ4KkuZ2WgfEcaDjjj33/pV3/bnFK+7ydPttc2Tek5xsI8
  XNMirIVxUiUT4YLy4WLiS200JUfbp1ZnMvnbQ8Jv1QnZl9W7WmBPcgxR4AAub0K4
  vZLXu6MXiboTlzkMB/YthCkTNlJcKkhHf60YR/T6Sx1T2nupyBa4deo5UGPzhRiJ
  pN37uqqAdK1qMDpCjARjS6U7Lf9JKjfiriLzLeyAjP8kaN4TdHSZd0pcQoZSxexQ
  9n+4E4MQm4EJ4DrVZCilsyL2BdETcHXKPc7q+Db4XM7jPKNG5GP1EMV4Xohv58yZ
  /rRfmK64gar8AMnOKT2AP681qdZs7lljONcXUALzlX5TqIchYT0DVQmFLYoMBeZz
  0l21QjbK0YWnPza6Yi/N4m6rFbEB4WXiqhYSkxzrMvocVUgd4AAP1vfHNnFEsnUR
  nSsiglFH/xlyO3cBFrmoZAxbA2091XHWhB4c0mQEI3hOqAB8UoFGBrQpmQ+LesoC
  1LZ9AgMBAAGjUTBPMAsGA1UdDwQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
  DgQWBBTFAxSvY64Q5adhm8IYecHBAUuobzAQBgkrBgEEAYI3FQEEAwIBADANBgkq
  hkiG9w0BAQsFAAOCAgEAjg/v4mIP7gBVCw4pemtGn3PStDh/aB9vbWyjAyxSNaaH
  H0nID5q5wow9ueBiDfjTPnhbf3P768HG8oL/+9C+Vm/0liFBd+0/DaayKpANFMLB
  BV+s2adWRhQucLQfXPwum8RybWv82wkRkWCCdOBaAvAMuTgk08SwJIyQfVgpk3nY
  0OwjFwSAadvevf+LoD/9L8R9NEt/n4WJe+LtEamo9EVb+l+cYqyxyubAVY0Y6BM2
  GXqAh3FEW2aQMpwouh/5S7w5oSMYN6miY1ojki8gPm0+4+CILPWh/fr2q0O/bPtb
  Tr++nPMmZ8ov9epNGIuqhtk5ja2/JuY+RW46IRc8QpF1EyUae02E6U2Vacs7Gge2
  CeSINkoLFFmiKBfIn/HAchlme9aL6DlJ9wAreBDH3E8kH7gRDWbSK2/QD0Hqac+E
  geGHwpg/8OtBOHUMnM7eLOXBJFcJosWf0XnEgS4ubgaHgqDEu8p8PE7rpCxtUNur
  t+x2xONI/rBWgdbp51lPr7o819zPJCvYZq1Pp1st8fb3RlUSWvbQMPFtGAyaBy+G
  0RgZ9WPtyEYgnHAb5/Dq46sne9/QnPwwGpjv1s1oE3ZFQjhvnGis8+dqRxk3YZAk
  yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
  -----END CERTIFICATE-----
bosh_vm_type: large.disk
dns_servers: ""
enable_telemetry: false
ephemeral_storage:
- datastore-4001
harbor:
  address_pool1_cidr: ""
  address_pool1_size: ""
  address_pool2_cidr: ""
  address_pool2_size: ""
  address_pool3_cidr: ""
  address_pool3_size: ""
  admin_password: 
  api_ca: |-
    -----BEGIN CERTIFICATE-----
    MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
    MRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3JwMRkw
    FwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIwMDgxOTE3MjA0NFoXDTMwMDgx
    OTE3MzAzNVowSDEVMBMGCgmSJomT8ixkARkWBXRhbnp1MRQwEgYKCZImiZPyLGQB
    GRYEY29ycDEZMBcGA1UEAxMQQ09OVFJPTENFTlRFUi1DQTCCAiIwDQYJKoZIhvcN
    AQEBBQADggIPADCCAgoCggIBALKIdX7643PzvtVXlqNIwDuNq+rhcHF0fjR414j+
    1IGQUuXrykjhSDthPP+8BGN7mBgHT8AjAS1b95xc8B0S2Fhln3AoRE9z03GtfsBu
    FSBRUVwAifX6oXu97WzffhqPtxZfLJXbhOomjlkX6iffAs2TOLUx2Oj4w2vybhzj
    lcA70ai+0Sl6axSo3lMZ4KkuZ2WgfEcaDjjj33/pV3/bnFK+7ydPttc2Tek5xsI8
    XNMirIVxUiUT4YLy4WLiS200JUfbp1ZnMvnbQ8Jv1QnZl9W7WmBPcgxR4AAub0K4
    vZLXu6MXiboTlzkMB/YthCkTNlJcKkhHf60YR/T6Sx1T2nupyBa4deo5UGPzhRiJ
    pN37uqqAdK1qMDpCjARjS6U7Lf9JKjfiriLzLeyAjP8kaN4TdHSZd0pcQoZSxexQ
    9n+4E4MQm4EJ4DrVZCilsyL2BdETcHXKPc7q+Db4XM7jPKNG5GP1EMV4Xohv58yZ
    /rRfmK64gar8AMnOKT2AP681qdZs7lljONcXUALzlX5TqIchYT0DVQmFLYoMBeZz
    0l21QjbK0YWnPza6Yi/N4m6rFbEB4WXiqhYSkxzrMvocVUgd4AAP1vfHNnFEsnUR
    nSsiglFH/xlyO3cBFrmoZAxbA2091XHWhB4c0mQEI3hOqAB8UoFGBrQpmQ+LesoC
    1LZ9AgMBAAGjUTBPMAsGA1UdDwQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
    DgQWBBTFAxSvY64Q5adhm8IYecHBAUuobzAQBgkrBgEEAYI3FQEEAwIBADANBgkq
    hkiG9w0BAQsFAAOCAgEAjg/v4mIP7gBVCw4pemtGn3PStDh/aB9vbWyjAyxSNaaH
    H0nID5q5wow9ueBiDfjTPnhbf3P768HG8oL/+9C+Vm/0liFBd+0/DaayKpANFMLB
    BV+s2adWRhQucLQfXPwum8RybWv82wkRkWCCdOBaAvAMuTgk08SwJIyQfVgpk3nY
    0OwjFwSAadvevf+LoD/9L8R9NEt/n4WJe+LtEamo9EVb+l+cYqyxyubAVY0Y6BM2
    GXqAh3FEW2aQMpwouh/5S7w5oSMYN6miY1ojki8gPm0+4+CILPWh/fr2q0O/bPtb
    Tr++nPMmZ8ov9epNGIuqhtk5ja2/JuY+RW46IRc8QpF1EyUae02E6U2Vacs7Gge2
    CeSINkoLFFmiKBfIn/HAchlme9aL6DlJ9wAreBDH3E8kH7gRDWbSK2/QD0Hqac+E
    geGHwpg/8OtBOHUMnM7eLOXBJFcJosWf0XnEgS4ubgaHgqDEu8p8PE7rpCxtUNur
    t+x2xONI/rBWgdbp51lPr7o819zPJCvYZq1Pp1st8fb3RlUSWvbQMPFtGAyaBy+G
    0RgZ9WPtyEYgnHAb5/Dq46sne9/QnPwwGpjv1s1oE3ZFQjhvnGis8+dqRxk3YZAk
    yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
    -----END CERTIFICATE-----
  api_cert: |-
    -----BEGIN CERTIFICATE-----
    MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
    ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
    MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
    MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
    EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
    S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
    BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
    z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
    lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
    FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
    H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
    uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
    F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
    H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
    JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
    Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
    +LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
    AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
    AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
    ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
    VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
    LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
    Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
    dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
    RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
    AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
    JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
    Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
    ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
    AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
    CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
    8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
    RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
    aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
    s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
    n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
    hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
    5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
    hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
    LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
    R22CcKK+kduUjv0X
    -----END CERTIFICATE-----
  api_private_key: |-
    -----BEGIN PRIVATE KEY-----
    MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
    1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
    ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
    UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
    dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
    8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
    0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
    f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
    7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
    J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
    ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
    BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
    OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
    RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
    IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
    CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
    lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
    jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
    oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
    ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
    TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
    +QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
    aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
    ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
    XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
    +CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
    62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
    R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
    Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
    fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
    O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
    73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
    3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
    8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
    pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
    Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
    GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
    ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
    L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
    csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
    Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
    6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
    NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
    a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
    uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
    eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
    cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
    l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
    zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
    I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
    -----END PRIVATE KEY-----
  auth_mode: ldap
  enabled: true
  harbor_http_proxy: ""
  harbor_https_proxy: ""
  harbor_no_proxy_addresses: ""
  hostname: harbor.corp.tanzu
  log_insight_enabled: false
  log_insight_host: ""
  log_insight_port: 514
  log_insight_protocol: tcp
  manual_ssl_certs: true
  notary_enabled: true
  post_deploy_enable_smoke_test: true
  pre_delete_enable_deregister_uaa: true
  storage_aws_access_key: ""
  storage_aws_enable_v4auth: true
  storage_aws_region: ""
  storage_aws_s3_url: ""
  storage_aws_secret: ""
  storage_aws_secure_mode: true
  storage_gcs_service_account_key: ""
  storage_local_persistent_disk_type: "20480"
  storage_nfs_share: ""
  storage_s3_bucket_name: ""
  storage_s3_bucket_root_dir: ""
  storage_s3_chunksize: 5242880
  storage_type: local
  trivy_enabled: true
  trivy_github_token: ""
  trivy_skip_update: false
  use_default_network_settings: true
  vm_type: medium.disk
identity_manual_ssl_certs: true
identity_pks_api_host: tkgi.corp.tanzu
k8s_pv_storage:
- datastore-4001
log_insight:
  ca_cert: ""
  enable_ssl: true
  enable_ssl_cert_validation: false
  enabled: false
  host: ""
  rate_limiting: 0
logsink:
  deploy_log_sink_resources: true
  deploy_metric_sink_resources: true
network:
  active_t0_edge_node: ""
  active_t0_edge_node_ip: ""
  additional_dep_reserved_ip_ranges: ""
  additional_svc_reserved_ip_ranges: ""
  autoprovision_nsx: false
  dep_dns: 192.168.110.10
  dep_network_cidr: ""
  dep_network_gateway: ""
  dep_network_moid: network-o17004
  dep_network_name: ls-tkgi-mgmt
  dep_network_type: OpaqueNetwork
  dep_reserved_ip_range_from: 172.31.0.10
  dep_reserved_ip_range_to: 172.31.0.10
  enable_outbound_traffic: true
  enabled_nsx_policy: false
  external_portgroup_gateway: ""
  external_portgroup_netmask: ""
  external_portgroup_subnet: ""
  external_vlan_id: 0
  flannel_pod_network_cidr: 10.200.0.0/16
  flannel_service_network_cidr: 10.100.200.0/24
  floating_ips_range: ""
  nsx_ca_crt: |-
    -----BEGIN CERTIFICATE-----
    MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
    MRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3JwMRkw
    FwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIwMDgxOTE3MjA0NFoXDTMwMDgx
    OTE3MzAzNVowSDEVMBMGCgmSJomT8ixkARkWBXRhbnp1MRQwEgYKCZImiZPyLGQB
    GRYEY29ycDEZMBcGA1UEAxMQQ09OVFJPTENFTlRFUi1DQTCCAiIwDQYJKoZIhvcN
    AQEBBQADggIPADCCAgoCggIBALKIdX7643PzvtVXlqNIwDuNq+rhcHF0fjR414j+
    1IGQUuXrykjhSDthPP+8BGN7mBgHT8AjAS1b95xc8B0S2Fhln3AoRE9z03GtfsBu
    FSBRUVwAifX6oXu97WzffhqPtxZfLJXbhOomjlkX6iffAs2TOLUx2Oj4w2vybhzj
    lcA70ai+0Sl6axSo3lMZ4KkuZ2WgfEcaDjjj33/pV3/bnFK+7ydPttc2Tek5xsI8
    XNMirIVxUiUT4YLy4WLiS200JUfbp1ZnMvnbQ8Jv1QnZl9W7WmBPcgxR4AAub0K4
    vZLXu6MXiboTlzkMB/YthCkTNlJcKkhHf60YR/T6Sx1T2nupyBa4deo5UGPzhRiJ
    pN37uqqAdK1qMDpCjARjS6U7Lf9JKjfiriLzLeyAjP8kaN4TdHSZd0pcQoZSxexQ
    9n+4E4MQm4EJ4DrVZCilsyL2BdETcHXKPc7q+Db4XM7jPKNG5GP1EMV4Xohv58yZ
    /rRfmK64gar8AMnOKT2AP681qdZs7lljONcXUALzlX5TqIchYT0DVQmFLYoMBeZz
    0l21QjbK0YWnPza6Yi/N4m6rFbEB4WXiqhYSkxzrMvocVUgd4AAP1vfHNnFEsnUR
    nSsiglFH/xlyO3cBFrmoZAxbA2091XHWhB4c0mQEI3hOqAB8UoFGBrQpmQ+LesoC
    1LZ9AgMBAAGjUTBPMAsGA1UdDwQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
    DgQWBBTFAxSvY64Q5adhm8IYecHBAUuobzAQBgkrBgEEAYI3FQEEAwIBADANBgkq
    hkiG9w0BAQsFAAOCAgEAjg/v4mIP7gBVCw4pemtGn3PStDh/aB9vbWyjAyxSNaaH
    H0nID5q5wow9ueBiDfjTPnhbf3P768HG8oL/+9C+Vm/0liFBd+0/DaayKpANFMLB
    BV+s2adWRhQucLQfXPwum8RybWv82wkRkWCCdOBaAvAMuTgk08SwJIyQfVgpk3nY
    0OwjFwSAadvevf+LoD/9L8R9NEt/n4WJe+LtEamo9EVb+l+cYqyxyubAVY0Y6BM2
    GXqAh3FEW2aQMpwouh/5S7w5oSMYN6miY1ojki8gPm0+4+CILPWh/fr2q0O/bPtb
    Tr++nPMmZ8ov9epNGIuqhtk5ja2/JuY+RW46IRc8QpF1EyUae02E6U2Vacs7Gge2
    CeSINkoLFFmiKBfIn/HAchlme9aL6DlJ9wAreBDH3E8kH7gRDWbSK2/QD0Hqac+E
    geGHwpg/8OtBOHUMnM7eLOXBJFcJosWf0XnEgS4ubgaHgqDEu8p8PE7rpCxtUNur
    t+x2xONI/rBWgdbp51lPr7o819zPJCvYZq1Pp1st8fb3RlUSWvbQMPFtGAyaBy+G
    0RgZ9WPtyEYgnHAb5/Dq46sne9/QnPwwGpjv1s1oE3ZFQjhvnGis8+dqRxk3YZAk
    yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
    -----END CERTIFICATE-----
  nsx_dns: 192.168.110.10
  nsx_enable_http_proxy: false
  nsx_fip_id: ee88a04b-58a5-45e6-988e-11bd041fba1d
  nsx_hybrid_nat: false
  nsx_manual_ssl_certs: true
  nsx_nat_mode: true
  nsx_node_cidr: 10.20.0.0/16
  nsx_node_ip_block_id: 1fe8c07b-4f3e-4f14-9c01-533c3bfc8946
  nsx_pod_cidr: 10.10.0.0/16
  nsx_pod_ip_block_id: ad4a8c09-fff2-4887-81da-9a75ff64f0a1
  nsx_service_network_cidr: 10.100.200.0/24
  nsx_t0_id: f32981b4-5858-4a0a-b7e1-c7aab619e426
  nsx_verify_ssl_certs: true
  proxy_http_password: ""
  proxy_http_url: ""
  proxy_http_user: ""
  proxy_https_password: ""
  proxy_https_url: ""
  proxy_https_user: ""
  proxy_no_proxy_addresses: ""
  standby_t0_edge_node: ""
  standby_t0_edge_node_ip: ""
  svc_dns: ""
  svc_network_cidr: 172.16.101.0/24
  svc_network_gateway: ""
  svc_network_name: ""
  svc_reserved_ip_range_from: ""
  svc_reserved_ip_range_to: ""
  t0_edge_node_lb_ip: ""
  t0_ha_mode_active_active: false
  use_antrea: false
  use_nsx: true
nsx_host: nsxmanager.corp.tanzu
nsx_password: 
nsx_username: admin
ntp_servers: 192.168.100.1
opsman_fqdn: opsman.corp.tanzu
opsman_private_key: ""
opsman_root_cert: ""
permanent_storage:
- datastore-4001
pks_api_cert: |-
  -----BEGIN CERTIFICATE-----
  MIIHiDCCBXCgAwIBAgITHQAAAAkDm8eswM8dBgAAAAAACTANBgkqhkiG9w0BAQsF
  ADBIMRUwEwYKCZImiZPyLGQBGRYFdGFuenUxFDASBgoJkiaJk/IsZAEZFgRjb3Jw
  MRkwFwYDVQQDExBDT05UUk9MQ0VOVEVSLUNBMB4XDTIxMDIwOTE2MjUyNFoXDTIz
  MDIwOTE2MjUyNFowcjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
  EjAQBgNVBAcTCVBhbG8gQWx0bzEPMA0GA1UEChMGVk13YXJlMRIwEAYDVQQLEwlU
  S0dJIDEuMTAxFTATBgNVBAMMDCouY29ycC50YW56dTCCAiIwDQYJKoZIhvcNAQEB
  BQADggIPADCCAgoCggIBAL53OJVD9TUtRpLWHqLr9OpUqF8HZOMG/L8/t8QeDzAE
  z2jIGC+ImCS2QUtf+t6NFU+1pDrcjZgfJE4KWoowQlCfwKPQr4YFSyoMN44RcmOD
  lSTfYEcjrLWWn+XyU1AUDilhoceTfdZei/1Q3mXxUZLmkrqVOjucjhpOr2gmlD55
  FEYeJBplBySsdcg9x0ey1+d/Ly7F2v4IWr91hDyNIJleUBbpF/atjhAazrRM9NLz
  H9lp7FE/EEskN1ZzChpQGdcamUEcIlr4ROTw2Jsc9zL9AEw8JoxjYlH7oIEHPVN9
  uwa7Ni3Yq9VWWFjZfhNXZQaz8aSQLpUHAgrTFPDkJNcebMFjnNR5exjTcffCV2I1
  F0pgh4A5KvGHjn2j13mtxa8W7wGtBssNFmN/q1rKnGLjwwMRI1g78KWS1zuA3Hc8
  H67YpoOV4LA31uYsdaTQvc16Qb81DKaiYAvuI4B/+f6OCEAvNIX3C/Ee3XXZB5j8
  JAtpTtBasbxAFplntvljfjlcgbsdJ+lKMUInX4xfv0J0dFTOi+xZ2BJhNhXukONV
  Po9jWNhPh1JyvhjvOWm8Mn24KcShpmiKdxKWzsEA9S5cN7RVtkMUOvcWrf8zQaKM
  +LmZ6/EBId2eLB94OxLB0n1VeFXsxNe42xUEGxieY/4LCG7laTvRdRWi4aCwK/VJ
  AgMBAAGjggI/MIICOzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
  AwEwFwYDVR0RBBAwDoIMKi5jb3JwLnRhbnp1MB0GA1UdDgQWBBSb5YTxUuhj65vF
  ZwneQHFNhGJa+jAfBgNVHSMEGDAWgBTFAxSvY64Q5adhm8IYecHBAUuobzCB0wYD
  VR0fBIHLMIHIMIHFoIHCoIG/hoG8bGRhcDovLy9DTj1DT05UUk9MQ0VOVEVSLUNB
  LENOPWNvbnRyb2xjZW50ZXIsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZp
  Y2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9Y29ycCxEQz10YW56
  dT9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JM
  RGlzdHJpYnV0aW9uUG9pbnQwgcEGCCsGAQUFBwEBBIG0MIGxMIGuBggrBgEFBQcw
  AoaBoWxkYXA6Ly8vQ049Q09OVFJPTENFTlRFUi1DQSxDTj1BSUEsQ049UHVibGlj
  JTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixE
  Qz1jb3JwLERDPXRhbnp1P2NBQ2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1j
  ZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MCEGCSsGAQQBgjcUAgQUHhIAVwBlAGIAUwBl
  AHIAdgBlAHIwDQYJKoZIhvcNAQELBQADggIBAAJq4Ix7Kd+Nz9ksBsdLbYOITux3
  CznnBSALkUAu5aL5PfJM2ww0Z54aOO1PH74jxc/1GQ5MM+xdd12JRklwB76MLXST
  8gWrWp229rCDA57qR5NgPY44rRM935WnnMoopQjdJTBveYvzFs8202E6yf4REdsx
  RVr7T9fhPz/hkR3tblTdinKeMM1QLN4C2NUjeqXSciham6KpwfPvcB4Ifhfb0PP7
  aQ6xbeEyGCc7y2Hj/DP52o64shGvEj4nM72xQhHT/1huXUuX3b1FH1+c8luZsker
  s2hrbUwJiMaOP4dY1NhhLsJJMDHr9RZSEgNVl7XHtpMM0Qp4nYL4Xz6W85phqTgF
  n8yt+NOeYEt7zuA9kK1/RSTTErXXpNfwTiJWQg3GqYlQ+mfwmjbAaCZ8r802ueNI
  hXZjvRtg/uHyl/GYp/WVemygw1XUAUIosUOEY7v+rvvPurN9K0qgcD5zTl/bsV/y
  5EFc+Q0KzSIV5CLfejwVJs40QdupWffXHOYqm49zT8ejffEExUBxXH/b4rooumkc
  hpsrx5hbo/XJvS7ZbXCH/k8kDq8+9o4QEVjqYyVwA/F3+/Mv2ywGLwKY5B+WvJQt
  LrxsDU58LYfVcwKSuryS5Rv9Kh0tZcFH2zpzQJDgMoZqPqZHFxhiV+w4KAD7WQxd
  R22CcKK+kduUjv0X
  -----END CERTIFICATE-----
pks_api_private_key: |-
  -----BEGIN PRIVATE KEY-----
  MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC+dziVQ/U1LUaS
  1h6i6/TqVKhfB2TjBvy/P7fEHg8wBM9oyBgviJgktkFLX/rejRVPtaQ63I2YHyRO
  ClqKMEJQn8Cj0K+GBUsqDDeOEXJjg5Uk32BHI6y1lp/l8lNQFA4pYaHHk33WXov9
  UN5l8VGS5pK6lTo7nI4aTq9oJpQ+eRRGHiQaZQckrHXIPcdHstfnfy8uxdr+CFq/
  dYQ8jSCZXlAW6Rf2rY4QGs60TPTS8x/ZaexRPxBLJDdWcwoaUBnXGplBHCJa+ETk
  8NibHPcy/QBMPCaMY2JR+6CBBz1TfbsGuzYt2KvVVlhY2X4TV2UGs/GkkC6VBwIK
  0xTw5CTXHmzBY5zUeXsY03H3wldiNRdKYIeAOSrxh459o9d5rcWvFu8BrQbLDRZj
  f6taypxi48MDESNYO/Clktc7gNx3PB+u2KaDleCwN9bmLHWk0L3NekG/NQymomAL
  7iOAf/n+jghALzSF9wvxHt112QeY/CQLaU7QWrG8QBaZZ7b5Y345XIG7HSfpSjFC
  J1+MX79CdHRUzovsWdgSYTYV7pDjVT6PY1jYT4dScr4Y7zlpvDJ9uCnEoaZoincS
  ls7BAPUuXDe0VbZDFDr3Fq3/M0GijPi5mevxASHdniwfeDsSwdJ9VXhV7MTXuNsV
  BBsYnmP+Cwhu5Wk70XUVouGgsCv1SQIDAQABAoICAQCE843Ay943j3Iq/2IFUfX1
  OMELDItE2lTFX0H0mRL67vCk8L/JNm0Ve09awRXKEetlZ6LLH7eLD3n1K88FlShF
  RS5ga0SKpdlQ8ZQ6DD2v72LFiVOYdPOTEiBtj9jOFiHIiwk12ePGJttLKQ8FVA0g
  IOkdaxtqDx82h+RzLDLg5P3c8B89eXYiCGxzKYSYrON/Cc2ytZPnLYfDC9IRvmWa
  CTaYt37txzpaTYwqWWmwctuxlPnLwNyrxw0FwGm18mIHP97ojy4AGDtnICPjKrX3
  lpmFnZs+9gTku2PPjXEmfaZ2zWnFWPChi5NB+hfCgofXxPYRbD/H8UtgqPV+LZL0
  jP6SV1DTbSVl3AIrFtCwWGzpTYLN9kxqRNIlU3hHZorErXQo0GebOqwKEjP81TFQ
  oBcBeRoWTkzZAl4oqS2FywXPEuUP21ChjK0jjW/VN267A23cS4IBhR7Cn825eg7/
  ZRcW82/i9EfyxFXi1dTRBP98gEj2ls6coLmbPIZHNqG73DMxk9cI3LfjGGu3rbfR
  TbqaNzsznYluiHlVWzjk3JKLvCBonGSD6ZhJ5OZkak8+Vl+rzI+9rWBJTBZihlwy
  +QeHZGq9htTAQ5F7KO9Tn7D79e7wIwZf2DHMweRGX9dUfkOaHCwwcSjVH/yziXtR
  aJ85akugMq/BciUBrffl/QKCAQEA8mmyuAbVecVRpcRVvH6auEAkL0yLp8rfAnm0
  ToD/yQE2GOPSB4uucamPBd33wNQA5UDUc0GmqqOAN6ne2fT3ZtzebRobdpvuYUu1
  XWlCvribR1zLgsglF2gTlOwOO1exZOlwax8a93xeBN3cQBXTisfXoMPbE80DTEqi
  +CPNnMBXKP6JOabDaWD+XvfNEe7PlQcgL7BKiujj98Ui/J/ebnITgpksGDNyaUIG
  62RwZeQOka6i6SMuCaD+Da3LjFfg9MAyOC1hsfst2puTkkJE+tJHRHOMSj6bZuW1
  R6rwe6SqrnhaBdDdUzZmONOVaJBCrQ9YE3ZiECH3cS3OQvDDDwKCAQEAySQmb+R7
  Ouk9IOh1HAZkLMWAPZ6FsZ916iMPnwcv7UADocRCzz3QtQOcI7lv242rrhLhVG/H
  fZMZh9aAL0Wm8i1/mrpAIYyMiNQJvWs4aY2bb6rlIL9mR0iThbVZXfyeRLvfyMpy
  O6PWYt8WF9dWLE7v3bW5Maqtqc+OND+1TGWTd0eZSSQgnR1VeFVNZSFG88vpmJDR
  73POVbMEKpxYfe7heZ0dApcb/IA1a3Zqqz0cZ4uqu1dWehwtb40dYmaqswbY6ke8
  3HKGQSBmlxWUF7Nn9Zg79u5YVW2jLOoMgUv3dDGAOIHlC97soA6NtoH/VhzY635+
  8+sX3wktvgXiJwKCAQEAhDCzbrr7So4ZegXYoxN/F56SnOBm/7cXaWgotO6PjXMF
  pwkFDWxUUlMeVRq38gUp/9ocgEV6t261iqUtizmUeBlVibVE6KcblR8N5cRyy0Is
  Gvw1VjoCUANHOlyHXkDx0Y+i6CdsMy00r/60DpZYZ0OXCGoFW4TemYnR2PLdOu+A
  GDDFcBTKVvq3e94xi+foduIN4TOHUryxI/nynEQprZyzmvIgI4pah5+j2lVJHacB
  ctwCppOylTmfkKIHb560Y4MzX4MP1VidpqpUDNvqdcSZbHB+PjZp0/DLrCtBPIuN
  L9sdbDJ7ntb5Y1+uB/kzAuBtLR/PVfDP2H4cDlDwbQKCAQBaRz51REDHLT6BkbRW
  csvtiGvJvGfXVHIRN9FgGFK7ktrOdY9jAyS0yjz/j9CT459lzxWR12Xbh/WSkYUR
  Mpr+4cr/QI9eP34oP7traD92qNdWJIcYzq9yWTHVdpL461SCFy0XKz5gZGXqFKUO
  6FjGJFvm0BSiJTAzInR6IQoXkxPAGsPDH1MAEdV14BuPw4LcE+7xyjZf2kOHFYVO
  NsRFKb3L3ufRbM9j4ouXgxvXZeNk2jw0P7wRrKn8AoNo0hnVpsIfTTmIXGLDwm4p
  a8b/aEfF5KEtcMb2+PGfTCF2uwkC/uDE/BA45sKgCEg03V4kYWg/MpR6mE8rjSwZ
  uPxLAoIBAQCPo0w3rHZMdBmaM9TCWfgCvMNuwhlWfPjQnYgpiPrVqnue1RdMLthY
  eETTu+yxL+IiA5u46iJmM60p1nPw2IXWODJ634AbWZgNmZifqWr4Hm8DXCzrYBNp
  cwB/NhcyJGX1eTZdooiwzdqukU5TCDRnPxNPv+TVUFcIsdmPHSJlZgkXiKh0JMbN
  l8JjE2rjKUQc61kIoll+MNDoW5uCakeb5K0SRuxPHpGC1+6hzW3zQNa+kGd85b5y
  zkrdsBEJ8/YXb9g+lId2Qpaj1MO6lgJTwLUkKTsDZ9hBindmBVAlAnVQzRxKeald
  I2b/u5gfwdfn/3z+JNpdcdc1A4cX7Qdi
  -----END PRIVATE KEY-----
pks_db_persistent_disk_type: "10240"
pks_db_vm_type: medium.disk
pks_enable_db_ha: false
pks_enable_quotas: false
pks_enable_vsphere_csi: false
pks_persistent_disk_type: "10240"
pks_run_upgrade_clusters_errand: false
pks_telemetry:
  customerNumber: ""
  installation_purpose: not_provided
  selector: disabled
pks_vm_instance_number: 1
pks_vm_type: medium.disk
plans:
- addons-spec: ""
  admission_pod_security_policy: false
  admission_security_context_deny: false
  allow_privileged_containers: true
  apply_addons_azs: []
  apply_addons_instances: 1
  apply_addons_lifecycle: errand
  apply_addons_networks: []
  apply_addons_vm_type: medium.disk
  cluster_services_linux_ha_enabled: false
  description: This plan will configure a lightweight Kubernetes cluster. Not recommended
    for production workloads.
  kubelet_custimzation_hard_eviction_threshold: ""
  kubelet_custimzation_system_reserved: ""
  kubelet_drain_delete_local_data: true
  kubelet_drain_force: true
  kubelet_drain_force_node: false
  kubelet_drain_grace_period: 10
  kubelet_drain_ignore_daemonsets: true
  kubelet_drain_timeout: 0
  master_azs:
  - enabled: true
    name: TKGI-COMP-1
  master_instances: 1
  master_networks: []
  master_persistent_disk_type: "10240"
  master_vm_extensions: []
  master_vm_type: medium.disk
  name: linux-small
  plan_id: ""
  post_deploy_errands: []
  pre_delete_errands: []
  windows_worker_os_enabled: false
  worker_azs:
  - enabled: true
    name: TKGI-COMP-1
  worker_instances: 1
  worker_max_instances: 50
  worker_networks: []
  worker_persistent_disk_type: "20480"
  worker_vm_extensions: []
  worker_vm_type: medium.disk
- addons-spec: ""
  admission_pod_security_policy: false
  admission_security_context_deny: false
  allow_privileged_containers: true
  apply_addons_azs: []
  apply_addons_instances: 1
  apply_addons_lifecycle: errand
  apply_addons_networks: []
  apply_addons_vm_type: medium.disk
  cluster_services_linux_ha_enabled: false
  description: 'Example: This plan will configure a medium sized Kubernetes cluster,
    suitable for more pods.'
  kubelet_custimzation_hard_eviction_threshold: ""
  kubelet_custimzation_system_reserved: ""
  kubelet_drain_delete_local_data: true
  kubelet_drain_force: true
  kubelet_drain_force_node: false
  kubelet_drain_grace_period: 10
  kubelet_drain_ignore_daemonsets: true
  kubelet_drain_timeout: 0
  master_azs:
  - enabled: true
    name: TKGI-COMP-1
  master_instances: 3
  master_networks: []
  master_persistent_disk_type: "10240"
  master_vm_extensions: []
  master_vm_type: medium.disk
  name: linux-medium
  plan_id: ""
  post_deploy_errands: []
  pre_delete_errands: []
  windows_worker_os_enabled: false
  worker_azs:
  - enabled: true
    name: TKGI-COMP-1
  worker_instances: 5
  worker_max_instances: 50
  worker_networks: []
  worker_persistent_disk_type: "51200"
  worker_vm_extensions: []
  worker_vm_type: medium.disk
- addons-spec: ""
  admission_pod_security_policy: false
  admission_security_context_deny: false
  allow_privileged_containers: false
  apply_addons_azs: []
  apply_addons_instances: 1
  apply_addons_lifecycle: errand
  apply_addons_networks: []
  apply_addons_vm_type: medium.disk
  cluster_services_linux_ha_enabled: false
  description: small windows plan
  kubelet_custimzation_hard_eviction_threshold: ""
  kubelet_custimzation_system_reserved: ""
  kubelet_drain_delete_local_data: true
  kubelet_drain_force: true
  kubelet_drain_force_node: false
  kubelet_drain_grace_period: 10
  kubelet_drain_ignore_daemonsets: true
  kubelet_drain_timeout: 0
  master_azs:
  - enabled: true
    name: TKGI-COMP-1
  master_instances: 1
  master_networks: []
  master_persistent_disk_type: "10240"
  master_vm_extensions: []
  master_vm_type: large.disk
  name: windows-small
  plan_id: ""
  post_deploy_errands: []
  pre_delete_errands: []
  windows_worker_os_enabled: true
  worker_azs:
  - enabled: true
    name: TKGI-COMP-1
  worker_instances: 1
  worker_max_instances: 50
  worker_networks: []
  worker_persistent_disk_type: "20480"
  worker_vm_extensions: []
  worker_vm_type: xlarge
- addons-spec: ""
  admission_pod_security_policy: false
  admission_security_context_deny: false
  allow_privileged_containers: false
  apply_addons_azs: []
  apply_addons_instances: 1
  apply_addons_lifecycle: errand
  apply_addons_networks: []
  apply_addons_vm_type: medium.disk
  cluster_services_linux_ha_enabled: false
  description: medium Windows plan
  kubelet_custimzation_hard_eviction_threshold: ""
  kubelet_custimzation_system_reserved: ""
  kubelet_drain_delete_local_data: true
  kubelet_drain_force: true
  kubelet_drain_force_node: false
  kubelet_drain_grace_period: 10
  kubelet_drain_ignore_daemonsets: true
  kubelet_drain_timeout: 0
  master_azs:
  - enabled: true
    name: TKGI-COMP-1
  master_instances: 3
  master_networks: []
  master_persistent_disk_type: "10240"
  master_vm_extensions: []
  master_vm_type: large.disk
  name: windows-medium
  plan_id: ""
  post_deploy_errands: []
  pre_delete_errands: []
  windows_worker_os_enabled: true
  worker_azs:
  - enabled: true
    name: TKGI-COMP-1
  worker_instances: 5
  worker_max_instances: 50
  worker_networks: []
  worker_persistent_disk_type: "51200"
  worker_vm_extensions: []
  worker_vm_type: xlarge
syslog:
  address: ""
  enable_tls: false
  enabled: false
  permitted_peer: ""
  port: 0
  protocol: tcp
  tls_cert: ""
tmc_integration:
  enabled: false
  tmc_api_token: ""
  tmc_api_url: ""
  tmc_cluster_name_prefix: tkgi-
  tmc_group_name: ""
use_thin_provisioned_disks: true
vcenter_datacenter: datacenter-1001
vcenter_host: vcsa-01a.corp.tanzu
vcenter_password: 
vcenter_username: administrator@vsphere.local
vrops:
  enabled: false
wavefront:
  enabled: false
  pks_wavefront_http_proxy: ""
  token: ""
  url: ""
worker_max_in_flight: 1

When you’re ready to proceed, click the Apply Upgrade Configuration button.

Per the warning, be sure to get any changes that might have been made manually in Opsman synchronized with TKGIMC. You can click the Continue button when you’re ready to proceed.

Upgrade Opsman

And the upgrade is underway. The old opsman VM will be suspended in preparation for the new one to be deployed:

And once that task is finished, a new opsman VM will get deployed:

And you can see the same task in the TKGIMC UI:

The new opsman VM will power on and have the same IP address as the original opsman VM (172.31.0.2 in this case):

The existing NAT rule in NSX-T will allow for the same access to the opsman UI as before (via the 10.40.14.34 address):

The original opsman VM should be deleted automatically.

Upgrade bosh

With the new opsman deployed, you should now be able to login and follow the progress there as well.

You can click on the Show Progress button to see the details on what is happening:

After a few minutes, you should see a new stemcell getting deployed:

This will be used for the new bosh VM. You should see the old bosh VM get deleted and a new one created:

Back in the Opsman UI you can see more details about the current operation:

There will be a lot more activity like this along with numerous tasks in vCenter while the new bosh VM is being configured.

I noticed that by the time the installation of the new bosh VM got to this point, I was able to access it again and run bosh commands:

bosh vms

Using environment '172.31.0.3' as client 'ops_manager'

Task 294
Task 295
Task 293
Task 294 done

Task 295 done

Task 293 done

Deployment 'harbor-container-registry-4cc226eed5c63d07ac2d'

Instance                                         Process State  AZ           IPs         VM CID                                   VM Type      Active  Stemcell
harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61  running        TKGI-MGMT-1  172.31.0.6  vm-dcf29dfb-51b7-4e58-8c00-ac50de618222  medium.disk  true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125

1 vms

Deployment 'pivotal-container-service-5d7c44d55652fb4f3c22'

Instance                                                        Process State  AZ           IPs         VM CID                                   VM Type      Active  Stemcell
pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a  running        TKGI-MGMT-1  172.31.0.5  vm-b8514659-825b-4a09-bbf9-0f7d45e4c011  medium.disk  true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125
pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64                     running        TKGI-MGMT-1  172.31.0.4  vm-cd957e99-d010-41bc-861f-23a6a90f4c35  medium.disk  true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125

2 vms

Deployment 'service-instance_2ed3bec8-1710-48f4-9917-19599e6cdcd0'

Instance                                     Process State  AZ           IPs         VM CID                                   VM Type      Active  Stemcell
master/c899fd74-5060-461f-bec1-05de2b29566e  running        TKGI-COMP-1  172.15.0.2  vm-19648717-f394-4d51-afe0-4c9ffd31ce65  medium.disk  true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125
worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2  running        TKGI-COMP-1  172.15.0.3  vm-cd711202-0ea9-4813-82e5-107974738b9b  medium.disk  true    bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125

2 vms

Succeeded

And very shortly after this point, the bosh portion of the upgrade was finished.

The very last thing to happen in this phase is the deletion of the old bosh VM:

You should see that the upgrade process has moved on to the TKGI phase in the TKGIMC UI:

Upgrade TKGI

There’s nothing to really see in vSphere or Opsman while the TKGI tile is being uploaded. Once this step is finished you’ll see activity very similar to what was observed with the bosh VM.

When the process moves on to the deploying phase, you can follow the progress in more detail in the Opsman UI.

You monitor the progress from the command line as well via the bosh task command:

bosh task
Using environment '172.31.0.3' as client 'ops_manager'

Task 327

Task 327 | 13:57:17 | Preparing deployment: Preparing deployment (00:00:11)
Task 327 | 13:57:28 | Preparing deployment: Rendering templates (00:00:08)
Task 327 | 13:57:37 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 327 | 13:57:37 | Compiling packages: golang-1-linux/f064db540b2fa2fe3e640f78d84dd00b8ee4052aec703d50763e9739f134c40e
Task 327 | 13:57:37 | Compiling packages: golang-1-linux/92acc26a97deb3429611aeef37c7f2d2697fd45dc0e8c14d62231d569112dd7b
Task 327 | 13:59:42 | Compiling packages: golang-1-linux/f064db540b2fa2fe3e640f78d84dd00b8ee4052aec703d50763e9739f134c40e (00:02:05)
Task 327 | 13:59:42 | Compiling packages: system-metrics-agent/40d78c7533904cebed45b39b3de555a9af9c71d087e1d2325d66514090806a92
Task 327 | 13:59:47 | Compiling packages: golang-1-linux/92acc26a97deb3429611aeef37c7f2d2697fd45dc0e8c14d62231d569112dd7b (00:02:10)
Task 327 | 13:59:47 | Compiling packages: bosh-dns/5d531fe59a5fa8e74d55db78ddbf0c30418e627046d3cf56549bdfbfaba176e3

The process will proceed with deploying a new stemcell for the TKGI VMs (API and DB VMs), cloning the stemcell (multiple times for the canary builds), configuring TKGI and then deleting the original VMs. When the process is complete you’ll be informed in the Opsman UI if you’re following along there.

 bosh task
Using environment '172.31.0.3' as client 'ops_manager'

Task 327

Task 327 | 13:57:17 | Preparing deployment: Preparing deployment (00:00:11)
Task 327 | 13:57:28 | Preparing deployment: Rendering templates (00:00:08)
Task 327 | 13:57:37 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 327 | 13:57:37 | Compiling packages: golang-1-linux/f064db540b2fa2fe3e640f78d84dd00b8ee4052aec703d50763e9739f134c40e
Task 327 | 13:57:37 | Compiling packages: golang-1-linux/92acc26a97deb3429611aeef37c7f2d2697fd45dc0e8c14d62231d569112dd7b
Task 327 | 13:59:42 | Compiling packages: golang-1-linux/f064db540b2fa2fe3e640f78d84dd00b8ee4052aec703d50763e9739f134c40e (00:02:05)
Task 327 | 13:59:42 | Compiling packages: system-metrics-agent/40d78c7533904cebed45b39b3de555a9af9c71d087e1d2325d66514090806a92
Task 327 | 13:59:47 | Compiling packages: golang-1-linux/92acc26a97deb3429611aeef37c7f2d2697fd45dc0e8c14d62231d569112dd7b (00:02:10)
Task 327 | 13:59:47 | Compiling packages: bosh-dns/5d531fe59a5fa8e74d55db78ddbf0c30418e627046d3cf56549bdfbfaba176e3
Task 327 | 14:00:15 | Compiling packages: system-metrics-agent/40d78c7533904cebed45b39b3de555a9af9c71d087e1d2325d66514090806a92 (00:00:33)
Task 327 | 14:00:37 | Compiling packages: bosh-dns/5d531fe59a5fa8e74d55db78ddbf0c30418e627046d3cf56549bdfbfaba176e3 (00:00:50)
Task 327 | 14:00:59 | Updating instance pks-db: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:00:59 | L executing pre-stop: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:01:00 | L executing drain: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:01:18 | L stopping jobs: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:01:26 | L executing post-stop: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:04:10 | L installing packages: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:04:36 | L configuring jobs: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:04:36 | L executing pre-start: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:05:13 | L starting jobs: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary)
Task 327 | 14:05:45 | L executing post-start: pks-db/c4761a65-e2c3-4244-bc8d-cf4b59451a64 (0) (canary) (00:06:16)
Task 327 | 14:07:15 | Updating instance pivotal-container-service: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:07:16 | L executing pre-stop: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:07:17 | L executing drain: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:07:18 | L stopping jobs: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:07:28 | L executing post-stop: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:10:04 | L installing packages: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:11:11 | L configuring jobs: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:11:11 | L executing pre-start: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:13:37 | L starting jobs: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary)
Task 327 | 14:14:08 | L executing post-start: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a (0) (canary) (00:07:31)

Task 327 Started  Thu Sep 23 13:57:17 UTC 2021
Task 327 Finished Thu Sep 23 14:14:46 UTC 2021
Task 327 Duration 00:17:29
Task 327 done

Succeeded

Upgrade Harbor

As with TKGI, while the Harbor tile is uploading you’re not going to see much activity. This is a fairly small tile so it should move on fairly quickly.

And just like the previous components, you can follow along in the Opsman UI or at the command line to get more details:

bosh task
Using environment '172.31.0.3' as client 'ops_manager'

Task 334

Task 334 | 14:20:20 | Preparing deployment: Preparing deployment (00:00:04)
Task 334 | 14:20:24 | Preparing deployment: Rendering templates (00:00:02)
Task 334 | 14:20:26 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 334 | 14:20:26 | Compiling packages: busybox/8a4c96b84928f6e9540e11eadaaa2d5a0afd5b90166c4e43ec0997a380eb51f9
Task 334 | 14:20:26 | Compiling packages: nfs-common/28863548db503f8be6a588950676eb13a536dd077567edb8f38f7d9de6403e8e
Task 334 | 14:20:26 | Compiling packages: smoke-test/249e495341e27147c0305e562d8ccce1b2a3b9fb818211cf9cbcfe3769e3e043
Task 334 | 14:20:26 | Compiling packages: python/2634ba233f5c4a1602f15712d146dd88451bd0661d757e8f1950045ee7022052

Once the Harbor upgrade is finished you should see notification of the completion wherever you are watching:

bosh task
Using environment '172.31.0.3' as client 'ops_manager'

Task 334

Task 334 | 14:20:20 | Preparing deployment: Preparing deployment (00:00:04)
Task 334 | 14:20:24 | Preparing deployment: Rendering templates (00:00:02)
Task 334 | 14:20:26 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 334 | 14:20:26 | Compiling packages: busybox/8a4c96b84928f6e9540e11eadaaa2d5a0afd5b90166c4e43ec0997a380eb51f9
Task 334 | 14:20:26 | Compiling packages: nfs-common/28863548db503f8be6a588950676eb13a536dd077567edb8f38f7d9de6403e8e
Task 334 | 14:20:26 | Compiling packages: smoke-test/249e495341e27147c0305e562d8ccce1b2a3b9fb818211cf9cbcfe3769e3e043
Task 334 | 14:20:26 | Compiling packages: python/2634ba233f5c4a1602f15712d146dd88451bd0661d757e8f1950045ee7022052
Task 334 | 14:22:02 | Compiling packages: smoke-test/249e495341e27147c0305e562d8ccce1b2a3b9fb818211cf9cbcfe3769e3e043 (00:01:36)
Task 334 | 14:22:02 | Compiling packages: uaa/d443ffe1547c43bd37d9f2973f12f877c93a0083d5b0da5c1617a623d09ed289 (00:00:18)
Task 334 | 14:22:20 | Compiling packages: wavefront/0d38ef4dfec0f35aeaac6f292e154b0bd9948eb3aa7682116820f5814a161efc
Task 334 | 14:22:32 | Compiling packages: nfs-common/28863548db503f8be6a588950676eb13a536dd077567edb8f38f7d9de6403e8e (00:02:06)
Task 334 | 14:22:32 | Compiling packages: docker-compose/a316e53e5ac4f708a82e4317c4e674c7fac92621061cac7bc253e41157462d3f
Task 334 | 14:22:34 | Compiling packages: busybox/8a4c96b84928f6e9540e11eadaaa2d5a0afd5b90166c4e43ec0997a380eb51f9 (00:02:08)
Task 334 | 14:22:34 | Compiling packages: common/1ee9dd03e6d818e2158fd430c06e81ca519f9fde706dcb8359fc22db1d2f0f50
Task 334 | 14:22:50 | Compiling packages: docker-compose/a316e53e5ac4f708a82e4317c4e674c7fac92621061cac7bc253e41157462d3f (00:00:18)
Task 334 | 14:22:52 | Compiling packages: common/1ee9dd03e6d818e2158fd430c06e81ca519f9fde706dcb8359fc22db1d2f0f50 (00:00:18)
Task 334 | 14:22:52 | Compiling packages: docker/3b37ad894d154483514c95232240e0a2529384cd22975f3738eaaea307a1690f
Task 334 | 14:22:52 | Compiling packages: harbor-common/c7fdde453d1de2011ac8de8bc593cb998fdbab565dab2ac130ce35fa9ee4bd77 (00:00:15)
Task 334 | 14:23:07 | Compiling packages: harbor-app/df55cfe8c9763b068684a7ed6e58434e16739a7a9646b4d95e0e7058348f9c18
Task 334 | 14:23:38 | Compiling packages: docker/3b37ad894d154483514c95232240e0a2529384cd22975f3738eaaea307a1690f (00:00:46)
Task 334 | 14:24:09 | Compiling packages: wavefront/0d38ef4dfec0f35aeaac6f292e154b0bd9948eb3aa7682116820f5814a161efc (00:01:49)
Task 334 | 14:25:09 | Compiling packages: python/2634ba233f5c4a1602f15712d146dd88451bd0661d757e8f1950045ee7022052 (00:04:43)
Task 334 | 14:26:11 | Compiling packages: harbor-app/df55cfe8c9763b068684a7ed6e58434e16739a7a9646b4d95e0e7058348f9c18 (00:03:04)
Task 334 | 14:26:43 | Updating instance harbor-app: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:26:43 | L executing pre-stop: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:26:44 | L executing drain: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:26:45 | L stopping jobs: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:27:36 | L executing post-stop: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:30:07 | L installing packages: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:31:05 | L configuring jobs: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:31:05 | L executing pre-start: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:35:45 | L starting jobs: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary)
Task 334 | 14:36:16 | L executing post-start: harbor-app/cc2959d0-b4d5-4f83-b592-80630b856c61 (0) (canary) (00:09:34)

Task 334 Started  Thu Sep 23 14:20:20 UTC 2021
Task 334 Finished Thu Sep 23 14:36:17 UTC 2021
Task 334 Duration 00:15:57
Task 334 done

Succeeded

I didn’t actually get to see the final stage show “Completed” in the TKGIMC UI since my login timed out but this was the last page I saw:

But after logging back in…

The upgrade was complete with all components showing the new versions.

Upgrade clusters

The upgrade process does not automatically upgrade any deployed clusters so this still remains to be done. I had a single linux cluster deployed and it still showed up as being a TKGI 1.11 cluster:

tkgi clusters

Upgrade is available to PKS Version: 1.12.0-build.42

PKS Version      Name           k8s Version  Plan Name    UUID                                  Status     Action
1.11.0-build.46  linux-cluster  1.20.6       linux-small  2ed3bec8-1710-48f4-9917-19599e6cdcd0  succeeded  CREATE
tkgi cluster linux-cluster

Upgrade is available to PKS Version: 1.12.0-build.42

PKS Version:              1.11.0-build.46
Name:                     linux-cluster
K8s Version:              1.20.6
Plan Name:                linux-small
UUID:                     2ed3bec8-1710-48f4-9917-19599e6cdcd0
Last Action:              CREATE
Last Action State:        succeeded
Last Action Description:  Instance provisioning completed
Kubernetes Master Host:   linux-cluster.corp.tanzu
Kubernetes Master Port:   8443
Worker Nodes:             1
Kubernetes Master IP(s):  10.40.14.42
Network Profile Name:
Kubernetes Profile Name:
Compute Profile Name:
Tags:

You can see similar information from the TKGIMC UI:

Having the TKGIMC in use makes upgrading clusters incredibly easy.

Just select the appropriate cluster and click the Upgrade button.

The best way to keep an eye on the upgrade is with the bosh and tkgi commands:

bosh task

Using environment '172.31.0.3' as client 'ops_manager'

Task 353

Task 353 | 14:56:35 | Deprecation: Global 'properties' are deprecated. Please define 'properties' at the job level.
Task 353 | 14:56:37 | Preparing deployment: Preparing deployment
Task 353 | 14:56:39 | Warning: DNS address not available for the link provider instance: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a
Task 353 | 14:56:39 | Warning: DNS address not available for the link provider instance: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a
Task 353 | 14:56:39 | Warning: DNS address not available for the link provider instance: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a
Task 353 | 14:56:50 | Preparing deployment: Preparing deployment (00:00:13)
Task 353 | 14:56:50 | Preparing deployment: Rendering templates (00:00:07)
Task 353 | 14:56:58 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 353 | 14:56:58 | Compiling packages: nsx-cni/3e9b647679f5ca57037c7eaf3d3292ca75b07fb5
Task 353 | 14:56:58 | Compiling packages: jq/c6a6daa7f64fc4775d11c0d4441d9fcf49506746
Task 353 | 14:56:58 | Compiling packages: nsx-python27/75df9f63298d0d2644c6030b160b9b7486a9c195
Task 353 | 14:56:58 | Compiling packages: nsx-cni-common/864a3d57e17b9bd533a7a9ac17af0fe1935b5926
tkgi tasks

ID                                    Type     Status       StartTime                      EndTime  Clusters
bf37a147-0ae9-4441-90f9-2cbdcbe662d9  UPGRADE  in progress  Thu, 23 Sep 2021 08:56:34 MDT  ---      linux-cluster

tkgi task bf37a147-0ae9-4441-90f9-2cbdcbe662d9

Your UPGRADE task is: in progress

Name           Status       Start time                     End time  isCanary
linux-cluster  in progress  Thu, 23 Sep 2021 08:56:34 MDT  ---       false

Just like with the other TKGI components, you should see a lot of activity in vSphere as the old nodes are deleted and the new nodes are created:

You can see that the upgrade is finished in multiple places:

bosh task

Using environment '172.31.0.3' as client 'ops_manager'

Task 353

Task 353 | 14:56:35 | Deprecation: Global 'properties' are deprecated. Please define 'properties' at the job level.
Task 353 | 14:56:37 | Preparing deployment: Preparing deployment
Task 353 | 14:56:39 | Warning: DNS address not available for the link provider instance: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a
Task 353 | 14:56:39 | Warning: DNS address not available for the link provider instance: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a
Task 353 | 14:56:39 | Warning: DNS address not available for the link provider instance: pivotal-container-service/fc1121b1-accc-48f0-9f30-ef72aec6959a
Task 353 | 14:56:50 | Preparing deployment: Preparing deployment (00:00:13)
Task 353 | 14:56:50 | Preparing deployment: Rendering templates (00:00:07)
Task 353 | 14:56:58 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 353 | 14:56:58 | Compiling packages: nsx-cni/3e9b647679f5ca57037c7eaf3d3292ca75b07fb5
Task 353 | 14:56:58 | Compiling packages: jq/c6a6daa7f64fc4775d11c0d4441d9fcf49506746
Task 353 | 14:56:58 | Compiling packages: nsx-python27/75df9f63298d0d2644c6030b160b9b7486a9c195
Task 353 | 14:56:58 | Compiling packages: nsx-cni-common/864a3d57e17b9bd533a7a9ac17af0fe1935b5926
Task 353 | 14:59:01 | Compiling packages: jq/c6a6daa7f64fc4775d11c0d4441d9fcf49506746 (00:02:03)
Task 353 | 14:59:01 | Compiling packages: ncp_rootfs/4fad3aa0fb55f62a11e4dc1fcd2d718ed5e109d1
Task 353 | 14:59:09 | Compiling packages: nsx-cni/3e9b647679f5ca57037c7eaf3d3292ca75b07fb5 (00:02:11)
Task 353 | 14:59:12 | Compiling packages: nsx-cni-common/864a3d57e17b9bd533a7a9ac17af0fe1935b5926 (00:02:14)
Task 353 | 15:00:19 | Compiling packages: ncp_rootfs/4fad3aa0fb55f62a11e4dc1fcd2d718ed5e109d1 (00:01:18)
Task 353 | 15:01:44 | Compiling packages: nsx-python27/75df9f63298d0d2644c6030b160b9b7486a9c195 (00:04:46)
Task 353 | 15:01:44 | Compiling packages: openvswitch/e350e39dbe0019e57494e4410d7f532ebfab177c (00:03:47)
Task 353 | 15:06:02 | Updating instance master: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:06:02 | L executing pre-stop: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:06:03 | L executing drain: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:06:05 | L stopping jobs: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:06:29 | L executing post-stop: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:09:00 | L installing packages: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:09:47 | L configuring jobs: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:09:47 | L executing pre-start: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:10:28 | L starting jobs: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary)
Task 353 | 15:10:53 | L executing post-start: master/c899fd74-5060-461f-bec1-05de2b29566e (0) (canary) (00:04:59)
Task 353 | 15:11:01 | Updating instance worker: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:11:02 | L executing pre-stop: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:11:41 | L executing drain: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:11:45 | L stopping jobs: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:12:03 | L executing post-stop: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:14:38 | L installing packages: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:15:50 | L configuring jobs: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:15:50 | L executing pre-start: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:16:17 | L starting jobs: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary)
Task 353 | 15:16:43 | L executing post-start: worker/745b1ef9-c38b-4da6-9dee-6a8a25c021e2 (0) (canary) (00:07:09)

Task 353 Started  Thu Sep 23 14:56:35 UTC 2021
Task 353 Finished Thu Sep 23 15:18:10 UTC 2021
Task 353 Duration 00:21:35
Task 353 done

Succeeded
tkgi task bf37a147-0ae9-4441-90f9-2cbdcbe662d9

Your UPGRADE task is: done

Name           Status     Start time                     End time                       isCanary
linux-cluster  succeeded  Thu, 23 Sep 2021 08:56:34 MDT  Thu, 23 Sep 2021 09:21:23 MDT  false

You can see here that the Kubernetes version has been upgraded to 1.21.3. From the command line, you can see the same and that the TKGI version has been updated:

tkgi clusters

PKS Version      Name           k8s Version  Plan Name    UUID                                  Status     Action
1.12.0-build.42  linux-cluster  1.21.3       linux-small  2ed3bec8-1710-48f4-9917-19599e6cdcd0  succeeded  UPGRADE

tkgi cluster linux-cluster

PKS Version:              1.12.0-build.42
Name:                     linux-cluster
K8s Version:              1.21.3
Plan Name:                linux-small
UUID:                     2ed3bec8-1710-48f4-9917-19599e6cdcd0
Last Action:              UPGRADE
Last Action State:        succeeded
Last Action Description:  Instance upgrade completed
Kubernetes Master Host:   linux-cluster.corp.tanzu
Kubernetes Master Port:   8443
Worker Nodes:             1
Kubernetes Master IP(s):  10.40.14.42
Network Profile Name:
Kubernetes Profile Name:
Compute Profile Name:
Tags:

What happened to my Windows plans?

I don’t have any Windows clusters deployed but I did create plans for them and upload a Windows stemcell (you can see more about this in my earlier post, TGKI 1.9 with Windows workers). However, when I went to deploy a Windows cluster in my upgraded TKGI environment, I quickly noticed that I had no Windows plans any longer.

tkgi plans

Name          ID                                    Description
linux-small   8A0E21A8-8072-4D80-B365-D1F502085560  This plan will configure a lightweight Kubernetes cluster. Not recommended for production workloads.
linux-medium  58375a45-17f7-4291-acf1-455bfdc8e371  Example: This plan will configure a medium sized Kubernetes cluster, suitable for more pods.

Windows plans need to in spots 11 through 13 and sure enough, these were inactive in Opsman on the TKGI tile (I was only using Plan 11 and Plan 12):

I started to dig further and saw that my Windows stemcell did not appear to be present:

Oddly enough, checking for the stemcell from the command line gave conflicting results:

bosh stemcells

Using environment '172.31.0.3' as client 'ops_manager'

Name                                      Version   OS             CPI                   CID
bosh-vsphere-esxi-ubuntu-xenial-go_agent  621.141*  ubuntu-xenial  e5fb7b6ede9a1242b8eb  sc-eed15e83-ea38-4eb3-99c4-8c34fafde669
~                                         621.125   ubuntu-xenial  e5fb7b6ede9a1242b8eb  sc-3ab644c7-6337-4090-8da1-99aa13a5a003
bosh-vsphere-esxi-windows2019-go_agent    2019.36   windows2019    e5fb7b6ede9a1242b8eb  sc-dfb4061a-429a-4fb8-828f-2bb52c755f56

(*) Currently deployed

3 stemcells

So bosh knows that there is a Windows stemcell present but Opsman does not.

Since this is all running in a vCD vApp it’s not terribly difficult to go back and double-check the configuration prior to starting the upgrade:

tkgi plans

Name            ID                                    Description
linux-small     8A0E21A8-8072-4D80-B365-D1F502085560  This plan will configure a lightweight Kubernetes cluster. Not recommended for production workloads.
linux-medium    58375a45-17f7-4291-acf1-455bfdc8e371  Example: This plan will configure a medium sized Kubernetes cluster, suitable for more pods.
windows-small   228fe8fa-8e98-44f1-aaf3-5f931b995840  small windows plan
windows-medium  612cc499-ff58-45be-acf9-2f9f72bfa236  medium Windows plan
bosh stemcells

Using environment '172.31.0.3' as client 'ops_manager'

Name                                      Version   OS             CPI                   CID
bosh-vsphere-esxi-ubuntu-xenial-go_agent  621.125*  ubuntu-xenial  e5fb7b6ede9a1242b8eb  sc-3ab644c7-6337-4090-8da1-99aa13a5a003
bosh-vsphere-esxi-windows2019-go_agent    2019.36   windows2019    e5fb7b6ede9a1242b8eb  sc-dfb4061a-429a-4fb8-828f-2bb52c755f56

(*) Currently deployed

2 stemcells

It turns out that this is not entirely unexpected. If you deploy TKGI via the TKGIMC, there is no ability to automate the configuration of Windows plans and upload Windows stemcells. This is largely due to the fact that the Windows stemcells are not managed/delivered by VMware as they must be manually created. To have Windows plans available via the TKGIMC, you must manually create them in Opsman, upload the appropriate Windows stemcell, Apply Changes and then synchronize the configuration in TKGIMC. I had done this in my 1.11 TKGIMC installation but didn’t realize that I would need to do it again after upgrading to 1.12. The Windows stemcell is no longer present in Opsman as this is a fresh deployment with just the settings from the old Opsman imported. The Windows plans cannot be created since there is no stemcell for them to use. If any Windows clusters had been deployed, the tkgi clusters command would actually error out as the Windows plans would no longer be available.

In the Opsman UI, navigate to the TKGI tile and then click on Plan 11. Fill out the information for the plan as appropriate:

Name: windows-small
Type: Windows
Description: small windows plan
Master/etcd node instances: 1
Master/etcd availability zones: tkg-comp-1
Master persistent disk size: 10 GB
Master/etcd VM type: medium.disk (cpu: 2, ram: 4 GB, disk: 32 GB)
Worker node instances: 1
Worker persistent disk size: 20 GB
Worker VM Type: automatic xlarge (cpu: 4, ram: 16GB, disk: 32GB)
Worker Availability Zone: tkg-comp-1
Errand VM type: medium.disk (cpu: 2, ram: 4 GB, disk: 32 GB)

Repeat for Plans 12 and 13 if you’re using them as well.

Navigate to Installation Dashboard and click on the Missing Stemcell link on the TKGI tile.

Click the Import Stemcell button. Select an appropriate Windows stemcell file. Once the file is imported, you can associate it with TKGI (these are typically much larger than the Linux stemcells so the import process may take a while):

Click the Save button here and then navigate to Installation Dashboard.

Click the Review Pending Changes button

You can see that the Windows stemcell will be uploaded for use with TKGI. Click the Apply Changes button.

When this is done, you should see the Windows plans and stemcell everywhere but the TKGIMC UI.

bosh stemcells

Using environment '172.31.0.3' as client 'ops_manager'

Name                                      Version   OS             CPI                   CID
bosh-vsphere-esxi-ubuntu-xenial-go_agent  621.141*  ubuntu-xenial  e5fb7b6ede9a1242b8eb  sc-eed15e83-ea38-4eb3-99c4-8c34fafde669
~                                         621.125   ubuntu-xenial  e5fb7b6ede9a1242b8eb  sc-3ab644c7-6337-4090-8da1-99aa13a5a003
bosh-vsphere-esxi-windows2019-go_agent    2019.40   windows2019    e5fb7b6ede9a1242b8eb  sc-150d5b7a-8dcb-48a6-b0ca-2c9af615be89
~                                         2019.36   windows2019    e5fb7b6ede9a1242b8eb  sc-dfb4061a-429a-4fb8-828f-2bb52c755f56

(*) Currently deployed

4 stemcells
tkgi plans

Name            ID                                    Description
linux-small     8A0E21A8-8072-4D80-B365-D1F502085560  This plan will configure a lightweight Kubernetes cluster. Not recommended for production workloads.
linux-medium    58375a45-17f7-4291-acf1-455bfdc8e371  Example: This plan will configure a medium sized Kubernetes cluster, suitable for more pods.
windows-small   228fe8fa-8e98-44f1-aaf3-5f931b995840  small windows plan
windows-medium  612cc499-ff58-45be-acf9-2f9f72bfa236  medium Windows plan

We still need to synchronize the changes made in Opsman to TKGIMC. This is done by navigating to the TKGI Configuration page and clicking on the Generate Configuration button.

Click on the Apply Configuration button.

The process will look very similar to what was observed during the upgrade, albeit much faster since very little is actually being done.

When this is finished, you should see the Windows plans as available options when creating a cluster in the TKGIMC UI:

Update the tkgi/pks/kubectl binaries

This is pretty minor but you don’t want to forget it. You can download the TKGi 1.12 pks/tkgi/kubectl binaries from Pivotal.

Once you have the files download, you can use the following steps to install them.

install kubectl-linux-amd64-1.21.3 /usr/local/bin/kubectl

kubectl version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6+vmware.1", GitCommit:"088f01db2ffab397a290be443902918b59ee032c", GitTreeState:"clean", BuildDate:"2021-04-17T01:01:00Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
install tkgi-linux-amd64-1.12.0-build.312 /usr/local/bin/tkgi

tkgi --version

PKS CLI version: 1.12.0-build.312
install pks-linux-amd64-1.12.0-build.312 /usr/local/bin/pks

pks --version

PKS CLI version: 1.12.0-build.312

Cleaning up

You can delete the old TKGIMC VM if you no longer need it.

If you have any unwanted Windows stemcells in bosh, you can delete them with the bosh delete-stemcell command.

bosh delete-stemcell bosh-vsphere-esxi-windows2019-go_agent/2019.36

Using environment '172.31.0.3' as client 'ops_manager'

Continue? [yN]: y

Task 438. Done

Succeeded

Leave a Comment

Your email address will not be published. Required fields are marked *