Building an IPV6 TKG 1.4 cluster

Leave a Comment / Install, Networking, TKG, vSphere / By

With the introduction of TKG 1.4, you now have the ability to deploy IPV6 Kubernetes clusters…and it’s not all that difficult to do either. I recently walked through the process and spent far more time working on getting a functional IPV6 vSphere environment with internet access than actually building out the TKG environment.

VMware has documented the process at IPv6 Networking (vSphere) but there are a few limitations that need to be called out:

  • This is for vSphere only (no Azure or AWS yet).
  • You cannot create an IPV6 workload cluster under a vSphere with Tanzu supervisor cluster.
  • You cannot create IPV6 workload clusters under an IPV4 management cluster (and vice versa).
  • You cannot register and IPV6 cluster to Tanzu Mission Control.
  • Your IPV6 clusters cannot participate in the Customer Experience Improvement Program (CEIP).
  • It is not supported to use NSX Advanced Load Balancer with an IPV6 cluster (for load balancer services or control plane endpoints).
  • Many of the available packages may not work or be unsupported in an IPV6 cluster.
  • If your vCenter Server is accessed through a proxy, you’ll have to use the VSPHERE_INSECURE: "true" flag when creating clusters.
  • You can only create IPV6-only clusters (no dual-stack clusters yet).
  • You must create the management cluster from a configuration file (no UI support yet).
  • You cannot upgrade/migrate an IPV4 cluster to IPV6 yet.
  • Though support for Windows nodes in TKG 1.4 is experimental, you cannot combine IPV6 and Windows nodes.

As you might guess, one of the first things you’ll need to have in place is a vSphere deployment that is configured for IPV6. VMware has documentation for this at Configuring vSphere for IPv6. Unfortunately, some of this documentation hasn’t been recently updated so you might find the names for a few things different if you’re on one of the latest vSphere versions. One issue that I ran into was wanting to have my vSphere deployment be dual-stack so that I could easily access it. Our documentation calls for configuring the IPV6 address from the VAMI UI but I observed that doing it this way automatically disables the IPV4 settings:

To get around this, I configured the IPV6 settings from the command line on the VCSA appliance:

/opt/vmware/share/vami/vami_config_net

 Main Menu

0)      Show Current Configuration (scroll with Shift-PgUp/PgDown)
1)      Exit this program
2)      Default Gateway
3)      Hostname
4)      DNS
5)      Proxy Server
6)      IP Address Allocation for eth0
Enter a menu number [0]: 6
Type Ctrl-C to go back to the Main Menu

Configure an IPv6 address for eth0? y/n [n]: y
Use a DHCPv6 Server instead of a static IPv6 address? y/n [n]: n
IPv6 Address []: fd53::22
Prefix []: 64
IPv6 Address:   fd53::22
Prefix:         64

Is this correct? y/n [y]: y

Configure an IPv4 address for eth0? y/n [n]: y
Use a DHCPv4 Server instead of a static IPv4 address? y/n [n]: n
IPv4 Address [192.168.110.22]:
Netmask [255.255.255.0]:
IPv4 Address:   192.168.110.22
Netmask:        255.255.255.0

Is this correct? y/n [y]: y

Reconfiguring eth0...
net.ipv6.conf.eth0.disable_ipv6 = 0
Network parameters successfully changed to requested values

 Main Menu

0)      Show Current Configuration (scroll with Shift-PgUp/PgDown)
1)      Exit this program
2)      Default Gateway
3)      Hostname
4)      DNS
5)      Proxy Server
6)      IP Address Allocation for eth0
Enter a menu number [0]: 2

Warning: if any of the interfaces for this VM use DHCP,
the Hostname, DNS, and Gateway parameters will be
overwritten by information from the DHCP server.

Type Ctrl-C to go back to the Main Menu

0)      eth0
Choose the interface to associate with default gateway [0]:
Gateway will be associated with eth0
IPv4 Default Gateway []: 192.168.110.1
IPv6 Default Gateway []: fd53::1
Reconfiguring eth0...
net.ipv6.conf.eth0.disable_ipv6 = 0
Network parameters successfully changed to requested values

 Main Menu

0)      Show Current Configuration (scroll with Shift-PgUp/PgDown)
1)      Exit this program
2)      Default Gateway
3)      Hostname
4)      DNS
5)      Proxy Server
6)      IP Address Allocation for eth0
Enter a menu number [0]: 0


Network Configuration for eth0
IPv4 Address:   192.168.110.22
Netmask:        255.255.255.0
IPv6 Address:   fd53::22
Prefix:         64

Global Configuration
IPv4 Gateway:   192.168.110.1
IPv6 Gateway:   fd53::1
Hostname:       vcsa-01a.corp.tanzu
DNS Servers:    127.0.0.1, fd53::11
Domain Name:
Search Path:    corp.tanzu
Proxy Server:

You’ll also need to make some changes on the system where you will run the tanzu command from so that docker will play nice with IPV6. The main two things are to ensure that your local system does not ignore IPV6 router advertisements (sudo sysctl net.ipv6.conf.eth0.accept_ra=2, replace eth0 as appropriate) and to create a MASQUERADE rule on the local system so that the bootstrap cluster can receive traffic from external resources (sudo ip6tables -t nat -A POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o docker0 -j MASQUERADE, updated IPV6 prefix as necessary).

One thing that I did not have to do is to configure docker to use an IPV6-capable DNS server…mine was already IPV6-capable. If you need to configure a different name server from your local system, you can follow the instructions at Specify DNS servers for Docker.

You’ll need to make sure that you have a DHCP server that is configured to allocate appropriate IPV6 addresses. In my labs, I’m usually using VyOS for networking and this was as simple as configuring an appropriate IPV6 interface (set interfaces ethernet eth1 address fd53::1/64) and then creating an IPV6 DHCP pool (set service dhcpv6-server shared-network-name 'IPV6DHCP' subnet fd53::/64 address-range start fd53::150 stop fd53::199; set service dhcpv6-server shared-network-name 'IPV6DHCP' subnet fd53::/64 name-server fd53::11).

If you will need to access any IPV4 resources from your IPV6-only cluster, you’ll also need a proxy in place.

I has all of this out of the way and thought I was ready to go when I ran into an issue I could not overcome. My labs are typically in a nested vCloud Director environment and I have little control over what happens outside of my own vApps. It turns out that I could not get IPV6 traffic out of the vApp due to a networking constraint at the org/vdc level. This is necessary for pulling down images from projects.registry.vmware.com. With this setback, I went in search of an alternate IPV6 vSphere environment and quickly found one ready to go (and it had IPV6 internet access).

I still had to run the sysctl and ip6tables commands but otherwise there was very little effort involved. The key piece to enabling IPV6 support for TKG 1.4 clusters is the TKG_IP_FAMILY: "ipv6" parameter in your cluster configuration file. With this in place, any tanzu management-cluster create or tanzu cluster create commands will do all of the work of getting your IPV6 cluster created.

What’s happening behind the scenes when the TKG_IP_FAMILY: "ipv6" parameter is set is that the vspheremachinetemplate object for each node is being modified such that it will get its address via dhcp6 instead of dhcp4. You can see this by examining the annotations on the vspheremachinetemplate objects:

kubectl get vspheremachinetemplates tkg-wld-control-plane -o jsonpath='{.metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration}' | jq '.spec.template.spec.network.devices'

[
  {
    "dhcp6": true,
    "networkName": "VM Network"
  }
]

If this were an IPV4 cluster, dhcp6 would be replaced with dhcp4.

Additionally, the vsphere-cpi-data-values secret has the ipFamily value set to ipv6. 

kubectl -n tkg-system get secrets vsphere-cpi-data-values -o jsonpath={.data.values\\.yaml} | base64 -d |grep ipFamily

  ipFamily: ipv6

The configuration file for my management cluster looked like the following:

VSPHERE_CONTROL_PLANE_ENDPOINT: "fd01:0:106:b:0:a:cccc:9bfd"
CLUSTER_NAME: tkg-mgmt
CLUSTER_PLAN: dev

VSPHERE_SERVER: "sc1-10-182-7-114.eng.vmware.com"

DEPLOY_TKG_ON_VSPHERE7: true

VSPHERE_INSECURE: "true"

VSPHERE_USERNAME: "administrator@vsphere.local"
VSPHERE_PASSWORD: <encoded:QWRtaW4hMjM=>
VSPHERE_DATACENTER: "dc0"
VSPHERE_DATASTORE: "sharedVmfs-0"
VSPHERE_FOLDER: "folder0"
VSPHERE_RESOURCE_POOL: "rp0"
VSPHERE_NETWORK: "VM Network"

VSPHERE_SSH_AUTHORIZED_KEY: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC5KYNeWQgVHrDHaEhBCLF1vIR0OAtUIJwjKYkY4E/5HhEu8fPFvBOIHPFTPrtkX4vzSiMFKE5WheKGQIpW3HHlRbmRPc9oe6nNKlsUfFAaJ7OKF146Gjpb7lWs/C34mjdtxSb1D/YcHSyqK5mxhyHAXPeK8lrxG5MLOJ3X2A3iUvXcBo1NdhRdLRWQmyjs16fnPx6840x9n5NqeiukFYIVhDMFErq42AkeewsWcbZQuwViSLk2cIc09eykAjaXMojCmSbjrj0kC3sbYX+HD2OWbKohTqqO6/UABtjYgTjIS4PqsXWk63dFdcxF6ukuO6ZHaiY7h3xX2rTg9pv1oT8WBR44TYgvyRp0Bhe0u2/n/PUTRfp22cOWTA2wG955g7jOd7RVGhtMHi9gFXeUS2KodO6C4XEXC7Y2qp9p9ARlNvu11QoaDyH3l0h57Me9we+3XQNuteV69TYrJnlgWecMa/x+rcaEkgr7LD61dY9sTuufttLBP2ro4EIWoBY6F1Ozvcp8lcgi/55uUGxwiKDA6gQ+UA/xtrKk60s6MvYMzOxJiUQbWYr3MJ3NSz6PJVXMvlsAac6U+vX4U9eJP6/C1YDyBaiT96cb/B9TkvpLrhPwqMZdYVomVHsdY7YriJB93MRinKaDJor1aIE/HMsMpbgFCNA7mma9x5HS/57Imw== admin@corp.local"
INFRASTRUCTURE_PROVIDER: vsphere
ENABLE_CEIP_PARTICIPATION: "false"
MACHINE_HEALTH_CHECK_ENABLED: "false"

VSPHERE_CONTROL_PLANE_NUM_CPUS: "4"
VSPHERE_CONTROL_PLANE_MEM_MIB: "16000"
VSPHERE_CONTROL_PLANE_DISK_GIB: "80"

VSPHERE_WORKER_NUM_CPUS: "4"
VSPHERE_WORKER_MEM_MIB: "16000"
VSPHERE_WORKER_DISK_GIB: "80"

TKG_IP_FAMILY: "ipv6"
CLUSTER_CIDR: "fd00:100:64::/48"
SERVICE_CIDR: "fd00:100:96::/108"

TKG_HTTP_PROXY: "http://[fd01:0:106:b:0:93ff:fe55:beec]:3128"
TKG_HTTPS_PROXY: "http://[fd01:0:106:b:0:93ff:fe55:beec]:3128"
TKG_NO_PROXY: "localhost,::/0,127.0.0.1,127.0.0.53,172.19.0.1"

A few notes about the items in here:

  • The value for VSPHERE_CONTROL_PLANE_ENDPOINT has to be a routable IPV6 address.
  • The three PROXY settings are optional but were required in this environment.
  • The CLUSTER_CIDR and SERVICE_CIDR values do not need to be set if the defaults are acceptable (the values in this example are the defaults).
  • As noted previously, TKG_IP_FAMILY has to be set to ipv6 and VSPHERE_INSECURE must be set to true (since a proxy is in use).

The command to create the cluster is the same as you would use when creating an IPV4 cluster:

tanzu management-cluster create -f tkg-mgmt.yaml

You should immediately start to see output similar to the following:

Validating the pre-requisites...

vSphere 7.0 Environment Detected.

You have connected to a vSphere 7.0 environment which does not have vSphere with Tanzu enabled. vSphere with Tanzu includes
an integrated Tanzu Kubernetes Grid Service which turns a vSphere cluster into a platform for running Kubernetes workloads in dedicated
resource pools. Configuring Tanzu Kubernetes Grid Service is done through vSphere HTML5 client.

Tanzu Kubernetes Grid Service is the preferred way to consume Tanzu Kubernetes Grid in vSphere 7.0 environments. Alternatively you may
deploy a non-integrated Tanzu Kubernetes Grid instance on vSphere 7.0.
Deploying TKG management cluster on vSphere 7.0 ...

Setting up management cluster...
Validating configuration...
Using infrastructure provider vsphere:v0.7.10
Generating cluster configuration...
Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /home/ubuntu/.kube-tkg/tmp/config_UM02AZZd
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available…

You should see a kind container running on your local system (this is the bootstrap cluster):

docker ps

CONTAINER ID        IMAGE                                                         COMMAND                  CREATED             STATUS              PORTS                 NAMES
c0b85b9d5d2d        projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1   "/usr/local/bin/entrâ¦"   15 seconds ago      Up 3 seconds        ::1:33821->6443/tcp   tkg-kind-c54ehis09c6tfrg5s9b0-control-plane

The location of the kubeconfig for the bootstrap cluster was noted in the tanzu management-cluster create output (/home/ubuntu/.kube-tkg/tmp/config_UM02AZZd). You can use this to see more details on the bootstrap cluster:


kubectl --kubeconfig=/home/ubuntu/.kube-tkg/tmp/config_UM02AZZd get nodes -o wide

NAME                                          STATUS   ROLES                  AGE   VERSION                               INTERNAL-IP             EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION      CONTAINER-RUNTIME
tkg-kind-c553ogc6n3g8siht63m0-control-plane   Ready    control-plane,master   46s   v1.21.2+vmware.1-360497810732255795   fc00:f853:ccd:e793::2   <none>        Ubuntu 20.04 LTS   4.4.0-142-generic   containerd://1.3.3-14-g449e9269

You can see that an IPV6 address has been assigned to the control plane node in the bootstrap cluster. If you suspect that the process is not proceeding, you can use this same kubeconfig file to check the status of any pods or to view the logs/events.

After a few minutes, the output from the tanzu management-cluster create command was updated and the process moved on to creating virtual machines:

Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Start creating management cluster...

After the control plane VM is provisioned it will be powered on and you can see that it has obtained an IPV6 address from the DHCP server:

As with an IPV4 cluster, the process will continue with creating any additional control plane and worker nodes. In this case, the cluster was only configured for a single control plane node and a single worker node.

The tanzu management-cluster create output continued to progress through the rest of the cluster installation:

Saving management cluster kubeconfig into /home/kubo/.kube/config
Installing providers on management cluster...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available…
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Waiting for the management cluster to get ready for move...
Waiting for addons installation...
Moving all Cluster API objects from bootstrap cluster to management cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Creating objects in the target cluster
Deleting objects from the source cluster
Waiting for additional components to be up and running...
Waiting for packages to be up and running...
Context set for management cluster tkg-mgmt as 'tkg-mgmt-admin@tkg-mgmt'.

Management cluster created!


You can now create your first workload cluster by running the following:

  tanzu cluster create [name] -f [file]


Some addons might be getting installed! Check their status by running the following:

  kubectl get apps -A

In the vSphere Client you should see both nodes created and running:

You can use the tanzu management-cluster get command to see more details on the new management cluster:

tanzu management-cluster get

  NAME      NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES
  tkg-mgmt  tkg-system  running  1/1           1/1      v1.21.2+vmware.1  management


Details:

NAME                                                         READY  SEVERITY  REASON  SINCE  MESSAGE
/tkg-mgmt                                                    True                     2m36s
├─ClusterInfrastructure - VSphereCluster/tkg-mgmt            True                     2m47s
├─ControlPlane - KubeadmControlPlane/tkg-mgmt-control-plane  True                     2m36s
│ └─Machine/tkg-mgmt-control-plane-6xfr7                     True                     2m39s
└─Workers
  └─MachineDeployment/tkg-mgmt-md-0
    └─Machine/tkg-mgmt-md-0-559c48d65d-n7ng5                 True                     2m44s


Providers:

  NAMESPACE                          NAME                    TYPE                    PROVIDERNAME  VERSION  WATCHNAMESPACE
  capi-kubeadm-bootstrap-system      bootstrap-kubeadm       BootstrapProvider       kubeadm       v0.3.23
  capi-kubeadm-control-plane-system  control-plane-kubeadm   ControlPlaneProvider    kubeadm       v0.3.23
  capi-system                        cluster-api             CoreProvider            cluster-api   v0.3.23
  capv-system                        infrastructure-vsphere  InfrastructureProvider  vsphere       v0.7.10

And inspecting the nodes in the cluster shows that they both have IPV6 addresses assigned:

kubectl get nodes -o wide

NAME                             STATUS   ROLES                  AGE     VERSION            INTERNAL-IP                  EXTERNAL-IP                  OS-IMAGE                 KERNEL-VERSION   CONTAINER-RUNTIME
tkg-mgmt-control-plane-6xfr7     Ready    control-plane,master   5m23s   v1.21.2+vmware.1   fd01:0:106:b:0:a:cccc:9bfd   fd01:0:106:b:0:a:cccc:9bfd   VMware Photon OS/Linux   4.19.198-1.ph3   containerd://1.4.6
tkg-mgmt-md-0-559c48d65d-n7ng5   Ready    <none>                 82s     v1.21.2+vmware.1   fd01:0:106:b:0:a:0:d67       fd01:0:106:b:0:a:0:d67       VMware Photon OS/Linux   4.19.198-1.ph3   containerd://1.4.6

With the management cluster created, you can now create a workload cluster. The process is the same as creating an IPV4 workload cluster and all you need is an appropriate configuration file, like the following:

VSPHERE_CONTROL_PLANE_ENDPOINT: "fd01:0:106:b:0:a:dddd:9bfd"
CLUSTER_PLAN: dev
CLUSTER_NAME: tkg-wld

VSPHERE_SERVER: "sc1-10-182-7-114.eng.vmware.com"

VSPHERE_INSECURE: "true"

VSPHERE_USERNAME: "administrator@vsphere.local"
VSPHERE_PASSWORD: <encoded:QWRtaW4hMjM=>
VSPHERE_DATACENTER: "dc0"
VSPHERE_DATASTORE: "sharedVmfs-0"
VSPHERE_FOLDER: "folder0"
VSPHERE_RESOURCE_POOL: "rp0"
VSPHERE_NETWORK: "VM Network"

VSPHERE_SSH_AUTHORIZED_KEY: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOjSCcFHRUmi2JF/tjZDbwg1ctA6U2NyWM75nOD0hyP2 kubo"
INFRASTRUCTURE_PROVIDER: vsphere
ENABLE_CEIP_PARTICIPATION: "false"
MACHINE_HEALTH_CHECK_ENABLED: "false"

# only necessary if multiple templates are uploaded
# VSPHERE_TEMPLATE: "ubuntu-2004-kube-v1.21.1"

VSPHERE_CONTROL_PLANE_NUM_CPUS: "4"
VSPHERE_CONTROL_PLANE_MEM_MIB: "16000"
VSPHERE_CONTROL_PLANE_DISK_GIB: "80"

VSPHERE_WORKER_NUM_CPUS: "4"
VSPHERE_WORKER_MEM_MIB: "16000"
VSPHERE_WORKER_DISK_GIB: "80"

TKG_IP_FAMILY: "ipv6"
CLUSTER_CIDR: "fd00:100:64::/48"
SERVICE_CIDR: "fd00:100:96::/108"

TKG_HTTP_PROXY: "http://[fd01:0:106:b:0:93ff:fe55:beec]:3128"
TKG_HTTPS_PROXY: "http://[fd01:0:106:b:0:93ff:fe55:beec]:3128"
TKG_NO_PROXY: "localhost,::/0,127.0.0.1,127.0.0.53,172.19.0.1"

This is almost identical to the management cluster configuration and all of the same notes are applicable here. Once your configuration file is ready you can proceed with creating the cluster:

tanzu cluster create -f wc-env-vars.yaml --tkr v1.21.2

Validating configuration...
Warning: Pinniped configuration not found. Skipping pinniped configuration in workload cluster. Please refer to the documentation to check if you can configure pinniped on workload cluster manually
Creating workload cluster 'tkg-wld'...
Waiting for cluster to be initialized...

And the final bit of output from the tanzu cluster create command:

Waiting for cluster nodes to be available...

Waiting for addons installation...
Waiting for packages to be up and running...

Workload cluster 'tkg-wld' created

You will need to get the kubeconfig for the new cluster to be able to access it:

tanzu cluster list

  NAME     NAMESPACE  STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES   PLAN
  tkg-wld  default    running  1/1           1/1      v1.21.2+vmware.1  <none>  dev

tanzu cluster kubeconfig get tkg-wld --admin
Credentials of cluster 'tkg-wld' have been saved

You can now access the cluster by running 'kubectl config use-context tkg-wld-admin@tkg-wld'

kubectl config use-context tkg-wld-admin@tkg-wld

Switched to context "tkg-wld-admin@tkg-wld".

And just as with the management cluster, you can take a closer look at the new workload cluster:

tanzu cluster get tkg-wld

  NAME     NAMESPACE  STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES
  tkg-wld  default    running  1/1           1/1      v1.21.2+vmware.1  <none>
ℹ

Details:

NAME                                                        READY  SEVERITY  REASON  SINCE  MESSAGE
/tkg-wld                                                    True                     16h
├─ClusterInfrastructure - VSphereCluster/tkg-wld            True                     16h
├─ControlPlane - KubeadmControlPlane/tkg-wld-control-plane  True                     16h
│ └─Machine/tkg-wld-control-plane-qxwgf                     True                     16h
└─Workers
  └─MachineDeployment/tkg-wld-md-0
    └─Machine/tkg-wld-md-0-896dc6f7b-m76dw                  True                     16h
kubectl get nodes -o wide

NAME                           STATUS   ROLES                  AGE     VERSION            INTERNAL-IP                  EXTERNAL-IP                  OS-IMAGE                 KERNEL-VERSION   CONTAINER-RUNTIME
tkg-wld-control-plane-qxwgf    Ready    control-plane,master   9m23s   v1.21.2+vmware.1   fd01:0:106:b:0:a:dddd:9bfd   fd01:0:106:b:0:a:dddd:9bfd   VMware Photon OS/Linux   4.19.198-1.ph3   containerd://1.4.6
tkg-wld-md-0-896dc6f7b-m76dw   Ready    <none>                 7m54s   v1.21.2+vmware.1   fd01:0:106:b:0:a:0:fe2       fd01:0:106:b:0:a:0:fe2       VMware Photon OS/Linux   4.19.198-1.ph3   containerd://1.4.6

From here you should be able to start deploying workloads.

With this being the first iteration of IPV6 support in TKG, there are some big limitations but we’ll hopefully see those removed in future versions. I’m especially looking forward to dual-stack support as this would allow for a lot more flexibility in the type of environments to which you can deploy TKG.

Leave a Comment

Your email address will not be published. Required fields are marked *