With the introduction of TKG 1.4, you now have the ability to deploy IPV6 Kubernetes clusters…and it’s not all that difficult to do either. I recently walked through the process and spent far more time working on getting a functional IPV6 vSphere environment with internet access than actually building out the TKG environment.
VMware has documented the process at IPv6 Networking (vSphere) but there are a few limitations that need to be called out:
- This is for vSphere only (no Azure or AWS yet).
- You cannot create an IPV6 workload cluster under a vSphere with Tanzu supervisor cluster.
- You cannot create IPV6 workload clusters under an IPV4 management cluster (and vice versa).
- You cannot register and IPV6 cluster to Tanzu Mission Control.
- Your IPV6 clusters cannot participate in the Customer Experience Improvement Program (CEIP).
- It is not supported to use NSX Advanced Load Balancer with an IPV6 cluster (for load balancer services or control plane endpoints).
- Many of the available packages may not work or be unsupported in an IPV6 cluster.
- If your vCenter Server is accessed through a proxy, you’ll have to use the
VSPHERE_INSECURE: "true"flag when creating clusters. - You can only create IPV6-only clusters (no dual-stack clusters yet).
- You must create the management cluster from a configuration file (no UI support yet).
- You cannot upgrade/migrate an IPV4 cluster to IPV6 yet.
- Though support for Windows nodes in TKG 1.4 is experimental, you cannot combine IPV6 and Windows nodes.
As you might guess, one of the first things you’ll need to have in place is a vSphere deployment that is configured for IPV6. VMware has documentation for this at Configuring vSphere for IPv6. Unfortunately, some of this documentation hasn’t been recently updated so you might find the names for a few things different if you’re on one of the latest vSphere versions. One issue that I ran into was wanting to have my vSphere deployment be dual-stack so that I could easily access it. Our documentation calls for configuring the IPV6 address from the VAMI UI but I observed that doing it this way automatically disables the IPV4 settings:
To get around this, I configured the IPV6 settings from the command line on the VCSA appliance:
/opt/vmware/share/vami/vami_config_net
Main Menu
0) Show Current Configuration (scroll with Shift-PgUp/PgDown)
1) Exit this program
2) Default Gateway
3) Hostname
4) DNS
5) Proxy Server
6) IP Address Allocation for eth0
Enter a menu number [0]: 6
Type Ctrl-C to go back to the Main Menu
Configure an IPv6 address for eth0? y/n [n]: y
Use a DHCPv6 Server instead of a static IPv6 address? y/n [n]: n
IPv6 Address []: fd53::22
Prefix []: 64
IPv6 Address: fd53::22
Prefix: 64
Is this correct? y/n [y]: y
Configure an IPv4 address for eth0? y/n [n]: y
Use a DHCPv4 Server instead of a static IPv4 address? y/n [n]: n
IPv4 Address [192.168.110.22]:
Netmask [255.255.255.0]:
IPv4 Address: 192.168.110.22
Netmask: 255.255.255.0
Is this correct? y/n [y]: y
Reconfiguring eth0...
net.ipv6.conf.eth0.disable_ipv6 = 0
Network parameters successfully changed to requested values
Main Menu
0) Show Current Configuration (scroll with Shift-PgUp/PgDown)
1) Exit this program
2) Default Gateway
3) Hostname
4) DNS
5) Proxy Server
6) IP Address Allocation for eth0
Enter a menu number [0]: 2
Warning: if any of the interfaces for this VM use DHCP,
the Hostname, DNS, and Gateway parameters will be
overwritten by information from the DHCP server.
Type Ctrl-C to go back to the Main Menu
0) eth0
Choose the interface to associate with default gateway [0]:
Gateway will be associated with eth0
IPv4 Default Gateway []: 192.168.110.1
IPv6 Default Gateway []: fd53::1
Reconfiguring eth0...
net.ipv6.conf.eth0.disable_ipv6 = 0
Network parameters successfully changed to requested values
Main Menu
0) Show Current Configuration (scroll with Shift-PgUp/PgDown)
1) Exit this program
2) Default Gateway
3) Hostname
4) DNS
5) Proxy Server
6) IP Address Allocation for eth0
Enter a menu number [0]: 0
Network Configuration for eth0
IPv4 Address: 192.168.110.22
Netmask: 255.255.255.0
IPv6 Address: fd53::22
Prefix: 64
Global Configuration
IPv4 Gateway: 192.168.110.1
IPv6 Gateway: fd53::1
Hostname: vcsa-01a.corp.tanzu
DNS Servers: 127.0.0.1, fd53::11
Domain Name:
Search Path: corp.tanzu
Proxy Server:You’ll also need to make some changes on the system where you will run the tanzu command from so that docker will play nice with IPV6. The main two things are to ensure that your local system does not ignore IPV6 router advertisements (sudo sysctl net.ipv6.conf.eth0.accept_ra=2, replace eth0 as appropriate) and to create a MASQUERADE rule on the local system so that the bootstrap cluster can receive traffic from external resources (sudo ip6tables -t nat -A POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o docker0 -j MASQUERADE, updated IPV6 prefix as necessary).
One thing that I did not have to do is to configure docker to use an IPV6-capable DNS server…mine was already IPV6-capable. If you need to configure a different name server from your local system, you can follow the instructions at Specify DNS servers for Docker.
You’ll need to make sure that you have a DHCP server that is configured to allocate appropriate IPV6 addresses. In my labs, I’m usually using VyOS for networking and this was as simple as configuring an appropriate IPV6 interface (set interfaces ethernet eth1 address fd53::1/64) and then creating an IPV6 DHCP pool (set service dhcpv6-server shared-network-name 'IPV6DHCP' subnet fd53::/64 address-range start fd53::150 stop fd53::199; set service dhcpv6-server shared-network-name 'IPV6DHCP' subnet fd53::/64 name-server fd53::11).
If you will need to access any IPV4 resources from your IPV6-only cluster, you’ll also need a proxy in place.
I has all of this out of the way and thought I was ready to go when I ran into an issue I could not overcome. My labs are typically in a nested vCloud Director environment and I have little control over what happens outside of my own vApps. It turns out that I could not get IPV6 traffic out of the vApp due to a networking constraint at the org/vdc level. This is necessary for pulling down images from projects.registry.vmware.com. With this setback, I went in search of an alternate IPV6 vSphere environment and quickly found one ready to go (and it had IPV6 internet access).
I still had to run the sysctl and ip6tables commands but otherwise there was very little effort involved. The key piece to enabling IPV6 support for TKG 1.4 clusters is the TKG_IP_FAMILY: "ipv6" parameter in your cluster configuration file. With this in place, any tanzu management-cluster create or tanzu cluster create commands will do all of the work of getting your IPV6 cluster created.
What’s happening behind the scenes when the TKG_IP_FAMILY: "ipv6" parameter is set is that the vspheremachinetemplate object for each node is being modified such that it will get its address via dhcp6 instead of dhcp4. You can see this by examining the annotations on the vspheremachinetemplate objects:
kubectl get vspheremachinetemplates tkg-wld-control-plane -o jsonpath='{.metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration}' | jq '.spec.template.spec.network.devices'
[
{
"dhcp6": true,
"networkName": "VM Network"
}
]If this were an IPV4 cluster, dhcp6 would be replaced with dhcp4.
Additionally, the vsphere-cpi-data-values secret has the ipFamily value set to ipv6.
kubectl -n tkg-system get secrets vsphere-cpi-data-values -o jsonpath={.data.values\\.yaml} | base64 -d |grep ipFamily
ipFamily: ipv6The configuration file for my management cluster looked like the following:
VSPHERE_CONTROL_PLANE_ENDPOINT: "fd01:0:106:b:0:a:cccc:9bfd"
CLUSTER_NAME: tkg-mgmt
CLUSTER_PLAN: dev
VSPHERE_SERVER: "sc1-10-182-7-114.eng.vmware.com"
DEPLOY_TKG_ON_VSPHERE7: true
VSPHERE_INSECURE: "true"
VSPHERE_USERNAME: "administrator@vsphere.local"
VSPHERE_PASSWORD: <encoded:QWRtaW4hMjM=>
VSPHERE_DATACENTER: "dc0"
VSPHERE_DATASTORE: "sharedVmfs-0"
VSPHERE_FOLDER: "folder0"
VSPHERE_RESOURCE_POOL: "rp0"
VSPHERE_NETWORK: "VM Network"
VSPHERE_SSH_AUTHORIZED_KEY: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC5KYNeWQgVHrDHaEhBCLF1vIR0OAtUIJwjKYkY4E/5HhEu8fPFvBOIHPFTPrtkX4vzSiMFKE5WheKGQIpW3HHlRbmRPc9oe6nNKlsUfFAaJ7OKF146Gjpb7lWs/C34mjdtxSb1D/YcHSyqK5mxhyHAXPeK8lrxG5MLOJ3X2A3iUvXcBo1NdhRdLRWQmyjs16fnPx6840x9n5NqeiukFYIVhDMFErq42AkeewsWcbZQuwViSLk2cIc09eykAjaXMojCmSbjrj0kC3sbYX+HD2OWbKohTqqO6/UABtjYgTjIS4PqsXWk63dFdcxF6ukuO6ZHaiY7h3xX2rTg9pv1oT8WBR44TYgvyRp0Bhe0u2/n/PUTRfp22cOWTA2wG955g7jOd7RVGhtMHi9gFXeUS2KodO6C4XEXC7Y2qp9p9ARlNvu11QoaDyH3l0h57Me9we+3XQNuteV69TYrJnlgWecMa/x+rcaEkgr7LD61dY9sTuufttLBP2ro4EIWoBY6F1Ozvcp8lcgi/55uUGxwiKDA6gQ+UA/xtrKk60s6MvYMzOxJiUQbWYr3MJ3NSz6PJVXMvlsAac6U+vX4U9eJP6/C1YDyBaiT96cb/B9TkvpLrhPwqMZdYVomVHsdY7YriJB93MRinKaDJor1aIE/HMsMpbgFCNA7mma9x5HS/57Imw== admin@corp.local"
INFRASTRUCTURE_PROVIDER: vsphere
ENABLE_CEIP_PARTICIPATION: "false"
MACHINE_HEALTH_CHECK_ENABLED: "false"
VSPHERE_CONTROL_PLANE_NUM_CPUS: "4"
VSPHERE_CONTROL_PLANE_MEM_MIB: "16000"
VSPHERE_CONTROL_PLANE_DISK_GIB: "80"
VSPHERE_WORKER_NUM_CPUS: "4"
VSPHERE_WORKER_MEM_MIB: "16000"
VSPHERE_WORKER_DISK_GIB: "80"
TKG_IP_FAMILY: "ipv6"
CLUSTER_CIDR: "fd00:100:64::/48"
SERVICE_CIDR: "fd00:100:96::/108"
TKG_HTTP_PROXY: "http://[fd01:0:106:b:0:93ff:fe55:beec]:3128"
TKG_HTTPS_PROXY: "http://[fd01:0:106:b:0:93ff:fe55:beec]:3128"
TKG_NO_PROXY: "localhost,::/0,127.0.0.1,127.0.0.53,172.19.0.1"A few notes about the items in here:
- The value for
VSPHERE_CONTROL_PLANE_ENDPOINThas to be a routable IPV6 address. - The three
PROXYsettings are optional but were required in this environment. - The
CLUSTER_CIDRandSERVICE_CIDRvalues do not need to be set if the defaults are acceptable (the values in this example are the defaults). - As noted previously,
TKG_IP_FAMILYhas to be set toipv6andVSPHERE_INSECUREmust be set totrue(since a proxy is in use).
The command to create the cluster is the same as you would use when creating an IPV4 cluster:
tanzu management-cluster create -f tkg-mgmt.yamlYou should immediately start to see output similar to the following:
Validating the pre-requisites...
vSphere 7.0 Environment Detected.
You have connected to a vSphere 7.0 environment which does not have vSphere with Tanzu enabled. vSphere with Tanzu includes
an integrated Tanzu Kubernetes Grid Service which turns a vSphere cluster into a platform for running Kubernetes workloads in dedicated
resource pools. Configuring Tanzu Kubernetes Grid Service is done through vSphere HTML5 client.
Tanzu Kubernetes Grid Service is the preferred way to consume Tanzu Kubernetes Grid in vSphere 7.0 environments. Alternatively you may
deploy a non-integrated Tanzu Kubernetes Grid instance on vSphere 7.0.
Deploying TKG management cluster on vSphere 7.0 ...
Setting up management cluster...
Validating configuration...
Using infrastructure provider vsphere:v0.7.10
Generating cluster configuration...
Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /home/ubuntu/.kube-tkg/tmp/config_UM02AZZd
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available…You should see a kind container running on your local system (this is the bootstrap cluster):
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b85b9d5d2d projects.registry.vmware.com/tkg/kind/node:v1.21.2_vmware.1 "/usr/local/bin/entrâ¦" 15 seconds ago Up 3 seconds ::1:33821->6443/tcp tkg-kind-c54ehis09c6tfrg5s9b0-control-planeThe location of the kubeconfig for the bootstrap cluster was noted in the tanzu management-cluster create output (/home/ubuntu/.kube-tkg/tmp/config_UM02AZZd). You can use this to see more details on the bootstrap cluster:
kubectl --kubeconfig=/home/ubuntu/.kube-tkg/tmp/config_UM02AZZd get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
tkg-kind-c553ogc6n3g8siht63m0-control-plane Ready control-plane,master 46s v1.21.2+vmware.1-360497810732255795 fc00:f853:ccd:e793::2 <none> Ubuntu 20.04 LTS 4.4.0-142-generic containerd://1.3.3-14-g449e9269You can see that an IPV6 address has been assigned to the control plane node in the bootstrap cluster. If you suspect that the process is not proceeding, you can use this same kubeconfig file to check the status of any pods or to view the logs/events.
After a few minutes, the output from the tanzu management-cluster create command was updated and the process moved on to creating virtual machines:
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Start creating management cluster...
After the control plane VM is provisioned it will be powered on and you can see that it has obtained an IPV6 address from the DHCP server:
As with an IPV4 cluster, the process will continue with creating any additional control plane and worker nodes. In this case, the cluster was only configured for a single control plane node and a single worker node.
The tanzu management-cluster create output continued to progress through the rest of the cluster installation:
Saving management cluster kubeconfig into /home/kubo/.kube/config
Installing providers on management cluster...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available…
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Waiting for the management cluster to get ready for move...
Waiting for addons installation...
Moving all Cluster API objects from bootstrap cluster to management cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Creating objects in the target cluster
Deleting objects from the source cluster
Waiting for additional components to be up and running...
Waiting for packages to be up and running...
Context set for management cluster tkg-mgmt as 'tkg-mgmt-admin@tkg-mgmt'.
Management cluster created!
You can now create your first workload cluster by running the following:
tanzu cluster create [name] -f [file]
Some addons might be getting installed! Check their status by running the following:
kubectl get apps -AIn the vSphere Client you should see both nodes created and running:
You can use the tanzu management-cluster get command to see more details on the new management cluster:
tanzu management-cluster get
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
tkg-mgmt tkg-system running 1/1 1/1 v1.21.2+vmware.1 management
Details:
NAME READY SEVERITY REASON SINCE MESSAGE
/tkg-mgmt True 2m36s
├─ClusterInfrastructure - VSphereCluster/tkg-mgmt True 2m47s
├─ControlPlane - KubeadmControlPlane/tkg-mgmt-control-plane True 2m36s
│ └─Machine/tkg-mgmt-control-plane-6xfr7 True 2m39s
└─Workers
└─MachineDeployment/tkg-mgmt-md-0
└─Machine/tkg-mgmt-md-0-559c48d65d-n7ng5 True 2m44s
Providers:
NAMESPACE NAME TYPE PROVIDERNAME VERSION WATCHNAMESPACE
capi-kubeadm-bootstrap-system bootstrap-kubeadm BootstrapProvider kubeadm v0.3.23
capi-kubeadm-control-plane-system control-plane-kubeadm ControlPlaneProvider kubeadm v0.3.23
capi-system cluster-api CoreProvider cluster-api v0.3.23
capv-system infrastructure-vsphere InfrastructureProvider vsphere v0.7.10And inspecting the nodes in the cluster shows that they both have IPV6 addresses assigned:
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
tkg-mgmt-control-plane-6xfr7 Ready control-plane,master 5m23s v1.21.2+vmware.1 fd01:0:106:b:0:a:cccc:9bfd fd01:0:106:b:0:a:cccc:9bfd VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6
tkg-mgmt-md-0-559c48d65d-n7ng5 Ready <none> 82s v1.21.2+vmware.1 fd01:0:106:b:0:a:0:d67 fd01:0:106:b:0:a:0:d67 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6With the management cluster created, you can now create a workload cluster. The process is the same as creating an IPV4 workload cluster and all you need is an appropriate configuration file, like the following:
VSPHERE_CONTROL_PLANE_ENDPOINT: "fd01:0:106:b:0:a:dddd:9bfd"
CLUSTER_PLAN: dev
CLUSTER_NAME: tkg-wld
VSPHERE_SERVER: "sc1-10-182-7-114.eng.vmware.com"
VSPHERE_INSECURE: "true"
VSPHERE_USERNAME: "administrator@vsphere.local"
VSPHERE_PASSWORD: <encoded:QWRtaW4hMjM=>
VSPHERE_DATACENTER: "dc0"
VSPHERE_DATASTORE: "sharedVmfs-0"
VSPHERE_FOLDER: "folder0"
VSPHERE_RESOURCE_POOL: "rp0"
VSPHERE_NETWORK: "VM Network"
VSPHERE_SSH_AUTHORIZED_KEY: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOjSCcFHRUmi2JF/tjZDbwg1ctA6U2NyWM75nOD0hyP2 kubo"
INFRASTRUCTURE_PROVIDER: vsphere
ENABLE_CEIP_PARTICIPATION: "false"
MACHINE_HEALTH_CHECK_ENABLED: "false"
# only necessary if multiple templates are uploaded
# VSPHERE_TEMPLATE: "ubuntu-2004-kube-v1.21.1"
VSPHERE_CONTROL_PLANE_NUM_CPUS: "4"
VSPHERE_CONTROL_PLANE_MEM_MIB: "16000"
VSPHERE_CONTROL_PLANE_DISK_GIB: "80"
VSPHERE_WORKER_NUM_CPUS: "4"
VSPHERE_WORKER_MEM_MIB: "16000"
VSPHERE_WORKER_DISK_GIB: "80"
TKG_IP_FAMILY: "ipv6"
CLUSTER_CIDR: "fd00:100:64::/48"
SERVICE_CIDR: "fd00:100:96::/108"
TKG_HTTP_PROXY: "http://[fd01:0:106:b:0:93ff:fe55:beec]:3128"
TKG_HTTPS_PROXY: "http://[fd01:0:106:b:0:93ff:fe55:beec]:3128"
TKG_NO_PROXY: "localhost,::/0,127.0.0.1,127.0.0.53,172.19.0.1"This is almost identical to the management cluster configuration and all of the same notes are applicable here. Once your configuration file is ready you can proceed with creating the cluster:
tanzu cluster create -f wc-env-vars.yaml --tkr v1.21.2
Validating configuration...
Warning: Pinniped configuration not found. Skipping pinniped configuration in workload cluster. Please refer to the documentation to check if you can configure pinniped on workload cluster manually
Creating workload cluster 'tkg-wld'...
Waiting for cluster to be initialized...
And the final bit of output from the tanzu cluster create command:
Waiting for cluster nodes to be available...
Waiting for addons installation...
Waiting for packages to be up and running...
Workload cluster 'tkg-wld' createdYou will need to get the kubeconfig for the new cluster to be able to access it:
tanzu cluster list
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN
tkg-wld default running 1/1 1/1 v1.21.2+vmware.1 <none> dev
tanzu cluster kubeconfig get tkg-wld --admin
Credentials of cluster 'tkg-wld' have been saved
You can now access the cluster by running 'kubectl config use-context tkg-wld-admin@tkg-wld'
kubectl config use-context tkg-wld-admin@tkg-wld
Switched to context "tkg-wld-admin@tkg-wld".And just as with the management cluster, you can take a closer look at the new workload cluster:
tanzu cluster get tkg-wld
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
tkg-wld default running 1/1 1/1 v1.21.2+vmware.1 <none>
ℹ
Details:
NAME READY SEVERITY REASON SINCE MESSAGE
/tkg-wld True 16h
├─ClusterInfrastructure - VSphereCluster/tkg-wld True 16h
├─ControlPlane - KubeadmControlPlane/tkg-wld-control-plane True 16h
│ └─Machine/tkg-wld-control-plane-qxwgf True 16h
└─Workers
└─MachineDeployment/tkg-wld-md-0
└─Machine/tkg-wld-md-0-896dc6f7b-m76dw True 16hkubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
tkg-wld-control-plane-qxwgf Ready control-plane,master 9m23s v1.21.2+vmware.1 fd01:0:106:b:0:a:dddd:9bfd fd01:0:106:b:0:a:dddd:9bfd VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6
tkg-wld-md-0-896dc6f7b-m76dw Ready <none> 7m54s v1.21.2+vmware.1 fd01:0:106:b:0:a:0:fe2 fd01:0:106:b:0:a:0:fe2 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6From here you should be able to start deploying workloads.
With this being the first iteration of IPV6 support in TKG, there are some big limitations but we’ll hopefully see those removed in future versions. I’m especially looking forward to dual-stack support as this would allow for a lot more flexibility in the type of environments to which you can deploy TKG.