Upgrading NSX Advanced Load Balancer

I had just installed NSX ALB version 20.1.4 a few weeks ago (see Deploying NSX Advanced Load Balancer for use with Tanzu Kubernetes Grid and vSphere with Tanzu) and was happy to see a new version (20.1.5) out so soon as I was curious to see what the upgrade process looked like. This process was far easier than I thought it would be, regardless of the method you choose to employ. I opted for the CLI method first but then went through the same exercise in the GUI just for the experience. You can read all about the upgrade process in the Flexible Upgrades documentation.

You might not immediately think that there is a new version if you’re just checking for it at my.vmware.com as you’ll only see version 20.1.X with a release date of 2020-10-12. If you head over to the Avi Release Notes page, you’ll get more information about the current version though.

The easiest way to see your current NSX ALB version is on the Administration, Controller, System Update page.

When you go to download an available upgrade, there are several items to choose from. Look for the following item after you get passed off from my.vmware.com to portal.avipulse.vmware.com/software/vantage. The file is about 4GB in size so it shouldn’t take too long to download.

Before we get started, DISCLAIMER: Only the 20.1.3 version is currently supported for use with TKG. I have used both 20.1.4 and 20.1.5 without issue but you might run into support concerns if you stray from 20.1.3.

Upgrading via CLI

You’ll need to sftp the download .pkg file to the /tmp folder on the controller appliance. Once this is done ssh over as the admin user and launch the shell command.

admin@192-168-110-32:~$ shell
Login: admin
Password:

[admin:192-168-110-32]: >

Note that the prompt changed from admin@192-168-110-32:~$ to [admin:192-168-110-32]:. This is a good way to be sure that you’re in the right mode.

Use the upload command to get the .pkg file prepped for installation.

upload image filename /tmp/controller-20.1.5-9148.pkg
Starting image upload...

There was quite a bit of output but you can expand the following to get an idea of what it should look like. You can see from this output that it only took about three and a half minutes for the operation to complete:

image upload output

+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                                | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| uuid                                 | image-f5d6e974-5cee-4ef1-9e0c-b773c10fa6b9                                                                                                                                                                                                                                                                                                                                                                                                               |
| name                                 | 20.1.5-9148-20210415.070829                                                                                                                                                                                                                                                                                                                                                                                                                              |
| controller_info                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|   path                               | image://20.1.5-9148-20210415.070829/controller.pkg                                                                                                                                                                                                                                                                                                                                                                                                       |
|   build                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|     date                             | 2021-04-15 07:08:29 UTC                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|     tag                              | 20.1.5-9148-20210415.070829                                                                                                                                                                                                                                                                                                                                                                                                                              |
|     version                          | 20.1.5                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|     build_no                         | 9148                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|     min_version                      | 15.2                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|     product                          | controller                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|     product_name                     | Avi Cloud Controller                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|   hash                               | 8d2840ba69ec62ac06c1cc4f179fb3eec49d0483a19bc5d4519baf97bc66299d                                                                                                                                                                                                                                                                                                                                                                                         |
| se_info                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|   build                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|     date                             | 2021-04-15 07:08:29 UTC                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|     tag                              | 20.1.5-9148-20210415.070829                                                                                                                                                                                                                                                                                                                                                                                                                              |
|     version                          | 20.1.5                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|     build_no                         | 9148                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|     min_version                      | 15.2                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|     product                          | se                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|     product_name                     | Avi Service Engine                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| migrations                           |                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|   api_version                        | 16_4_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| versions                             |                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                                      | 14_2                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                      | 15_1                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                      | 15_1_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 15_2                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                      | 15_2_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 15_3                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                      | 16_1                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                      | 16_1_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_1_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_1_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_2                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                      | 16_2_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_2_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_2_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_3                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                      | 16_3_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_3_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_3_4                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_4                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_5                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_6                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_7                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_8                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_4_9                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_5_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_5_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_5_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 16_5_4                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_4                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_5                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_6                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_7                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_8                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_9                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_1_10                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_1_11                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_1_12                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_1_13                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_1_14                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_2_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_4                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_5                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_6                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_7                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_8                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_9                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 17_2_10                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_2_11                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_2_12                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_2_13                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_2_14                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_2_15                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_2_16                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 17_2_17                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 18_1_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_1_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_1_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_1_4                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_1_5                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_4                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_5                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_6                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_7                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_8                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_9                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 18_2_10                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 18_2_11                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 18_2_12                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                      | 20_1_1                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 20_1_2                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 20_1_3                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 20_1_4                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | 20_1_5                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                      | current_version                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|   max_active_versions                | 2                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|   controller_min_free_disk_size      | 10                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|   se_min_free_disk_size              | 5                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|   controller_host_min_free_disk_size | 10                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|   se_host_min_free_disk_size         | 5                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|   rollback_controller_disk_space     | 2                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|   rollback_se_disk_space             | 1                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|   se_min_total_disk                  | 10                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|   se_min_memory                      | 2                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|   se_min_cores                       | 1                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|   controller_min_total_disk          | 128                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|   controller_min_memory              | 12                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|   controller_min_cores               | 4                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| type                                 | IMAGE_TYPE_SYSTEM                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| status                               | SYSERR_SUCCESS                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| uber_bundle                          | False                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| cloud_info_values                    |                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                                      | {'cloud_name': 'openstack', 'cloud_data_values': [{'key': 'supported_releases', 'values': ['ussuri', 'train', 'stein', 'rocky', 'queens', 'pike', 'ocata']}, {'key': 'keystone_api_versions', 'values': ['3.8-3.14']}, {'key': 'glance_api_versions', 'values': ['2.5-2.9']}, {'key': 'nova_api_min_versions', 'values': ['2.1']}, {'key': 'nova_api_micro_max_versions', 'values': ['2.42-2.87']}, {'key': 'neutron_api_versions', 'values': ['2.0']}]} |
| tenant_uuid                          | admin                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Time Taken: 211.57266354560852 secs


You can validate that the image is ready with the show image command:

show image

+-----------------------------+--------------------------------------------+-------------------+
| Name                        | UUID                                       | Type              |
+-----------------------------+--------------------------------------------+-------------------+
| 20.1.4-9087-20210215.202012 | image-1bd25dc4-8d9f-472b-a87f-6fa96c7415e5 | IMAGE_TYPE_SYSTEM |
| 20.1.5-9148-20210415.070829 | image-f5d6e974-5cee-4ef1-9e0c-b773c10fa6b9 | IMAGE_TYPE_SYSTEM |
+-----------------------------+--------------------------------------------+-------------------+

You’ll use the upgrade system command to kick off your upgrade. There are several options that you can use with this depending on how you want the upgrade to proceed but I went with the simplest (none). The only mandatory option is image_ref and the name of the image returned from the show image command executed previously.

upgrade system image_ref 20.1.5-9148-20210415.070829

+-------------+------------------------------------------------------------------------------+
| Field       | Value                                                                        |
+-------------+------------------------------------------------------------------------------+
| status_code | SYSERR_UPGRADE_OPS_PREVIEW_RESPONSE                                          |
| status      | Checks preview for upgrade operations.                                       |
| checks      |                                                                              |
|             | Check Controller Cluster readiness for upgrade operations.                   |
|             | Check and inform user to take a backup prior to upgrade operations.          |
|             | Check if se linux is enabled on controller nodes.                            |
|             | Check if upgrade operation is already in progress.                           |
|             | Check ServiceEngineGroup has an ongoing upgrade operation.                   |
|             | Check image version compatibility for upgrade operations.                    |
|             | Check ServiceEngine reachability for upgrade operations.                     |
|             | Check Cloud readiness for upgrade operations.                                |
|             | Check ServiceEngine disk space for upgrade operations.                       |
|             | Check Controller Cluster disk space for upgrade operations.                  |
|             | Check and inform Virtual Service(s) disruption for upgrade operations.       |
|             | Check idempotent operations for upgrade operations.                          |
|             | Check active versions compatibility for upgrade operations.                  |
|             | Check ServiceEngineGroup error recovery options prior to upgrade operations. |
|             | Check Image state across Cluster members for upgrade operations.             |
|             | Checks for the patch in image bundle.                                        |
|             | Checks if Gslb Feature is enabled and provides feature specific messages.    |
|             | Checks the system configuration.                                             |
|             | Check total number of alerts for upgrade operations.                         |
|             | Checks if the cloud api versions are compatible after upgrade.               |
+-------------+------------------------------------------------------------------------------+
Starting upgrade
+--------------+---------------------------------------------------------------------------------------------------+
| Field        | Value                                                                                             |
+--------------+---------------------------------------------------------------------------------------------------+
| warning      | True                                                                                              |
| errors       |                                                                                                   |
|  maskable    | True                                                                                              |
|  status_code | SYSERR_MC_UPGRADE_VS_DISRUPTED_ERR                                                                |
|  reason      | Virtual Services disruption warning during upgrade operations.                                    |
|  details     |                                                                                                   |
|              | VS Errors: 'Warning: Traffic to following Virtual Services will be disrupted.' VS:tkg-wld--tanzu- |
|              | system-ingress-envoy Tenant:admin Reason:not scaledout                                            |
|  maskable    | True                                                                                              |
|  status_code | SYSERR_MC_BACKUP_ERR                                                                              |
|  reason      | Inform User to take configuration backup prior to upgrade operations.                             |
|  details     |                                                                                                   |
|              | Please take the backup before starting the upgrade operations.                                    |
+--------------+---------------------------------------------------------------------------------------------------+

Do you want to continue? [y/n]: 

I paused here as I was concerned about the “VS Error” message. I took a look at my single Virtual Service (envoy) and realized that it was only running a single instance on one SE.

Looks like I must have missed something when I originally installed NSX ALB. I had to drill down into the VS to see what the configuration looked like.

If you drill down in the VS and then hover over the VS name, you’ll get a pop-up that looks like the following:

That “not scaledout” message makes a little more sense now. I could see that this VS was only running on the AviTanzu-se-esywk SE but needed to run on both SEs. I clicked on the Scale Out button and was able to have a copy placed on the AviTanzu-se-zzdns SE also.

After clicking the Scale out button, I saw the AviTanzu-se-zzdnz VM being modified (an extra vNIC was being configured) in the vSphere Client.

And almost immediately, the VS showed both SEs.

Lastly, to make sure that this doesn’t happen again, I needed to modify the SE group configuration such that at least two instances of each VS were created. From the Infrastructure, Service Engine Group page, I had to click on the Edit button for the Default-Group SE Group.

I had to set the Scale per Virtual Service minimum value to 2.

It’s worth noting that you can simply make this change to the SE Group (without manually scaling out the VS) and it will automatically scale out any existing Virtual Services that aren’t already.

It turns out that I got into this situation because I was using an Enterprise license and moved off of the default Active/Standby HA node. In Active/Standby, an instance of each VS is placed on each SE (but not active on the standby node). In Active/Active or N+M, the default behavior is to only create one instance of each VS on the active node.

Now I felt better about proceeding with the upgrade. Out of curiosity I did go through this once without the VS scaled out and was able to confirm that there was an outage of about two minutes for all of the services behind the Contour/Envoy ingress.

Do you want to continue? [y/n]: y
Starting upgrade
+-------------+-----------------------------------------------------------------------------------------------------------+
| Field       | Value                                                                                                     |
+-------------+-----------------------------------------------------------------------------------------------------------+
| status_code | SYSERR_UPGRADE_SYSTEM_STARTED                                                                             |
| status      | 'Upgrade of System (Controller + All SEGroup(s)) started. Use 'show upgrade status' to check the status.' |
+-------------+-----------------------------------------------------------------------------------------------------------+

At this point, the upgrade was underway. You can run the show upgrade status command to see how far along things are and what components are invovled.

show upgrade status

+-------------+--------+-------+---------------------------------+-----------+-------+-------+
| Name        | Tenant | Cloud | State                           | Operation | Image | Patch |
+-------------+--------+-------+---------------------------------+-----------+-------+-------+
| cluster-0-1 | -      | -     | UPGRADE_FSM_IN_PROGRESS : (16)% | UPGRADE   | -     | -     |
+-------------+--------+-------+---------------------------------+-----------+-------+-------+

I did get kicked out of the UI after a couple of minutes and lost access via SSH as well. I attempted to login at the at the console and while this worked, trying to run the shell command failed with a “Could not connect to the controller” error. The system rebooted shortly afterward. Even after the reboot, launching the shell command gave different output than the first time. It’s nice enough to let you know that the controller is not ready but that you can at least check the upgrade status.

shell

Login: admin
Password:
Controller is not yet ready - Launching the shell unauthenticated
If the system is being upgraded, you can monitor upgrade status
Controller is not yet ready - Launching the shell unauthenticated
If the system is being upgraded, you can monitor upgrade status

I ran the show upgrade status several times as the process progressed to see where things were. Once the SEs were being upgraded (and likely a little earlier) I was able to get back in to the UI but was not permitted to make any kind of configuration change.

show upgrade status
+-------------------+--------+---------------+--------------------------------------------+-----------+-----------------------------+-------+
| Name              | Tenant | Cloud         | State                                      | Operation | Image                       | Patch |
+-------------------+--------+---------------+--------------------------------------------+-----------+-----------------------------+-------+
| cluster-0-1       | admin  | -             | UPGRADE_FSM_SE_UPGRADE_IN_PROGRESS : (83)% | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| Default-Group     | admin  | Default-Cloud | UPGRADE_FSM_IN_PROGRESS : (78)%            | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| AviTanzu-se-zzdnz | admin  | Default-Cloud | UPGRADE_FSM_IN_PROGRESS : (0)%             | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| AviTanzu-se-esywk | admin  | Default-Cloud | UPGRADE_FSM_IN_PROGRESS : (45)%            | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
+-------------------+--------+---------------+--------------------------------------------+-----------+-----------------------------+-------+
show upgrade status
+-------------------+--------+---------------+--------------------------------------------+-----------+-----------------------------+-------+
| Name              | Tenant | Cloud         | State                                      | Operation | Image                       | Patch |
+-------------------+--------+---------------+--------------------------------------------+-----------+-----------------------------+-------+
| cluster-0-1       | admin  | -             | UPGRADE_FSM_SE_UPGRADE_IN_PROGRESS : (83)% | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| Default-Group     | admin  | Default-Cloud | UPGRADE_FSM_IN_PROGRESS : (78)%            | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| AviTanzu-se-zzdnz | admin  | Default-Cloud | UPGRADE_FSM_IN_PROGRESS : (45)%            | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| AviTanzu-se-esywk | admin  | Default-Cloud | UPGRADE_FSM_COMPLETED                      | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
+-------------------+--------+---------------+--------------------------------------------+-----------+-----------------------------+-------+

When the process was finished, the show upgrade status command showed the following output:

show upgrade status
+-------------------+--------+---------------+-----------------------+-----------+-----------------------------+-------+
| Name              | Tenant | Cloud         | State                 | Operation | Image                       | Patch |
+-------------------+--------+---------------+-----------------------+-----------+-----------------------------+-------+
| cluster-0-1       | admin  | -             | UPGRADE_FSM_COMPLETED | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| Default-Group     | admin  | Default-Cloud | UPGRADE_FSM_COMPLETED | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| AviTanzu-se-zzdnz | admin  | Default-Cloud | UPGRADE_FSM_COMPLETED | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
| AviTanzu-se-esywk | admin  | Default-Cloud | UPGRADE_FSM_COMPLETED | UPGRADE   | 20.1.5-9148-20210415.070829 | -     |
+-------------------+--------+---------------+-----------------------+-----------+-----------------------------+-------+

And back in the UI we can see that the Administration, Controller, System Update page shows the current version as 20.1.5.

Upgrading via the UI

This is definitely the easier option but I’m glad I did it via the CLI first to get a little better idea of what’s happening behind the scenes.

From the Administration, Controller, Software page, click the Upload From Computer button and choose the .pkg file you downloaded earlier.

You’ll see a progress bar which will let you know when the upload is finished.

If the upload was successful, you should see the original 20.1.4 version and the new 20.1.5 version present.

Navigate to the Administration, Controller, System Update page. Select the new image and click on the Upgrade button.

You’ll be presented with options for proceeding with the upgrade. I simply accepted the defaults as they worked for me.

Since I had already scaled out the VS, the only warning was about taking a backup (which I had already done).

After clicking the Confirm button, the upgrade process was shown in the UI (for a short time).

Again, I very quickly lost access to the UI and saw the system reboot. While you wait for the UI to come back you can ssh to the controller VM (or login at the console) and run show upgrade status to see where things are. After a few minutes, the UI will come back and you can continue to follow the upgrade process there.

You can click on the blue info button to see more details about the upgrade and where things are.

And just as with the CLI method, you’ll see the current version updated to 20.1.5 when the upgrade is finished.

I tried out the Rollback process (note the button above) and while it was unavoidably disruptive to my VS traffic, it was a painless process. I hope to never truly need to use it but having it prominent and easy was reassuring.

Fix the ako-essential-role role

One last thing, in TKG, when NSX ALB is configured during the management cluster deployment, a role name ako-essential-role is created. This role has all of the privileges needs for the AKO user (<workload cluster name>-ako-user) to be able to interact with NSX ALB. After NSX ALB is upgraded to version 20.1.5, there is a new privilege that is needed that was not previously present. You can use the following process to modify the ako-essential-role role and include this new privilege.

You need to be on the NSX ALB controller VM and in the shell mode noted previously. The new privilege that is needed is named PERMISSION_L4POLICYSET and if you check the ako-essential-role role, you will see that it is not present:

show role ako-essential-role

+-------------------------+-------------------------------------------+
| Field                   | Value                                     |
+-------------------------+-------------------------------------------+
| uuid                    | role-faecbacf-301d-42c8-8760-ef6d1e8d7fa6 |
| name                    | ako-essential-role                        |
| privileges[1]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_VIRTUALSERVICE                 |
| privileges[2]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_POOL                           |
| privileges[3]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_POOLGROUP                      |
| privileges[4]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_HTTPPOLICYSET                  |
| privileges[5]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_NETWORKSECURITYPOLICY          |
| privileges[6]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_AUTOSCALE                      |
| privileges[7]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_DNSPOLICY                      |
| privileges[8]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_NETWORKPROFILE                 |
| privileges[9]           |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_APPLICATIONPROFILE             |
| privileges[10]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_APPLICATIONPERSISTENCEPROFILE  |
| privileges[11]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_HEALTHMONITOR                  |
| privileges[12]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_ANALYTICSPROFILE               |
| privileges[13]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_IPAMDNSPROVIDERPROFILE         |
| privileges[14]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_CUSTOMIPAMDNSPROFILE           |
| privileges[15]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_TRAFFICCLONEPROFILE            |
| privileges[16]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_IPADDRGROUP                    |
| privileges[17]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_STRINGGROUP                    |
| privileges[18]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_VSDATASCRIPTSET                |
| privileges[19]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_PROTOCOLPARSER                 |
| privileges[20]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_SSLPROFILE                     |
| privileges[21]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_AUTHPROFILE                    |
| privileges[22]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_PINGACCESSAGENT                |
| privileges[23]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_PKIPROFILE                     |
| privileges[24]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_SSLKEYANDCERTIFICATE           |
| privileges[25]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_CERTIFICATEMANAGEMENTPROFILE   |
| privileges[26]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_HARDWARESECURITYMODULEGROUP    |
| privileges[27]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_SSOPOLICY                      |
| privileges[28]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_NATPOLICY                      |
| privileges[29]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_WAFPROFILE                     |
| privileges[30]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_WAFPOLICY                      |
| privileges[31]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_WAFPOLICYPSMGROUP              |
| privileges[32]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_ERRORPAGEPROFILE               |
| privileges[33]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_ERRORPAGEBODY                  |
| privileges[34]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_ALERTCONFIG                    |
| privileges[35]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_ALERT                          |
| privileges[36]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_ACTIONGROUPCONFIG              |
| privileges[37]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_ALERTSYSLOGCONFIG              |
| privileges[38]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_ALERTEMAILCONFIG               |
| privileges[39]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_SNMPTRAPPROFILE                |
| privileges[40]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_TRAFFIC_CAPTURE                |
| privileges[41]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_CLOUD                          |
| privileges[42]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_SERVICEENGINE                  |
| privileges[43]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_SERVICEENGINEGROUP             |
| privileges[44]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_NETWORK                        |
| privileges[45]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_VRFCONTEXT                     |
| privileges[46]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_USER_CREDENTIAL                |
| privileges[47]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_SYSTEMCONFIGURATION            |
| privileges[48]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_CONTROLLER                     |
| privileges[49]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_REBOOT                         |
| privileges[50]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_TECHSUPPORT                    |
| privileges[51]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_INTERNAL                       |
| privileges[52]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_CONTROLLERSITE                 |
| privileges[53]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_IMAGE                          |
| privileges[54]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_USER                           |
| privileges[55]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_ROLE                           |
| privileges[56]          |                                           |
|   type                  | READ_ACCESS                               |
|   resource              | PERMISSION_TENANT                         |
| privileges[57]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_GSLB                           |
| privileges[58]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_GSLBSERVICE                    |
| privileges[59]          |                                           |
|   type                  | NO_ACCESS                                 |
|   resource              | PERMISSION_GSLBGEODBPROFILE               |
| allow_unlabelled_access | True                                      |
| tenant_ref              | admin                                     |
+-------------------------+-------------------------------------------+

Enter configure mode for the role:

configure role ako-essential-role

Updating an existing object. Currently, the object is:

You’ll see the same list of privileges that was previously shown in this output as well. You should also see that the prompt changes to [admin:192-168-110-32]: role>.

Edit the privileges for the role:

[admin:192-168-110-32]: role> privileges
New object being created

And you’ll see that the prompt changes again, this time to [admin:192-168-110-32]: role:privileges>.

Now we can actually add the new privilege to this role:

[admin:192-168-110-32]: role:privileges> type write_access
[admin:192-168-110-32]: role:privileges> resource permission_l4policyset
[admin:192-168-110-32]: role:privileges> save
[admin:192-168-110-32]: role> save

You’ll get the entire role configuration output to the screen again, this time with the following at the end:

| privileges[60]          |                                           |
|   type                  | WRITE_ACCESS                              |
|   resource              | PERMISSION_L4POLICYSET                    |

Leave a Comment

Your email address will not be published.