How to Deploy MetalLB with BGP in a Tanzu Kubernetes Grid 1.1 Cluster

In my last post, I had briefly hinted at the need for LoadBalancer services in TKG on vSphere since the only other externally accessible option (NodePort) will run into issues after an upgrade as your node IP addresses will change. TKG on vSphere does not offer any solution that provides LoadBalancer functionality (yet) but one of the more popular and easy-to-configure options is MetalLB. I’m going to walk through the deployment of MetalLB into a TKG 1.1  cluster running on vSphere and I’m going to use a BGP configuration (vs. a Layer 2 configuration) since the environment where I run all of my labs (vCD) already has a Linux VM running that provides routing and BGP functionality.

If you want to follow these steps largely verbatim, the only prerequisites that you’ll need to fulfill are a TKG cluster running on vSphere (see my first post, Installing TKG 1.1 on vSphere, Step-By-Step, for instructions) and a remote system running BGP that you can peer with. You could also modify these steps fairly easily to create a Layer 2 configuration if that is your preference.

We’ll start out by creating a new namespace, deploying the MetalLB manifest and creating a random secret:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

The next step requires some knowledge of your remote system where BGP is configured. In my example, the Linux VM where I am running BGP is at IP address 192.168.100.1 and the ASN there is 65002. I want to provide a small range of IP addresses for LoadBalancer services so I have chosen 10.40.14.0/27 as my subnet. Lastly, I have chosen 65022 as my local ASN. Once you have this information ready, you can create a ConfigMap that MetalLB will use, per the following example:

kubectl apply -f - << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 192.168.100.1
      peer-asn: 65002
      my-asn: 65022
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 10.40.14.0/27
      bgp-advertisements:
      - aggregation-length: 27
EOF

We’ll need to check the IP addresses assigned to the MetalLB “speaker” pods as they will need to be added to the remote system’s BGP configuration as peer addresses:

kubectl -n metallb-system get po -o wide
 
NAME                          READY   STATUS    RESTARTS   AGE   IP                NODE                         NOMINATED NODE   READINESS GATES
controller-5c9894b5cd-fnph8   1/1     Running   0          12m   100.125.233.126   tsm2-md-0-6cb84c4647-qvxf9   <none>           <none>
speaker-jq2xd                 1/1     Running   0          12m   192.168.100.192   tsm2-control-plane-jf7f9     <none>           <none>
speaker-rn8f2                 1/1     Running   0          12m   192.168.100.194   tsm2-md-0-6cb84c4647-qvxf9   <none>           <none>

In this example, the IP addresses I’ll be working with are 192.168.100.192 and 192.168.100.194.

The next step will vary based on the type of system you’re peering with for BGP. In my example, it’s a Linux VM running quagga and all of the necessary configuration is in the /etc/quagga/bgpd.conf file.

cat /etc/quagga/bgpd.conf
 
!
! Zebra configuration saved from vty
!   2019/12/31 11:13:11
!
hostname bgpd
password VMware1!
enable password VMware1!
log file /var/log/quagga/quagga.log
!
router bgp 65002
 bgp router-id 192.168.100.1
 neighbor 192.168.100.3 remote-as 65001
 neighbor 192.168.100.3 default-originate
 neighbor 192.168.100.4 remote-as 65001
 neighbor 192.168.100.4 default-originate
 neighbor 192.168.200.3 remote-as 65011
 neighbor 192.168.200.3 default-originate
 neighbor 192.168.200.4 remote-as 65011
 neighbor 192.168.200.4 default-originate
 neighbor 192.168.200.5 remote-as 65011
 neighbor 192.168.200.5 default-originate
 neighbor 192.168.210.3 remote-as 65012
 neighbor 192.168.210.3 default-originate
 neighbor 192.168.210.4 remote-as 65012
 neighbor 192.168.210.4 default-originate
 maximum-paths 4
!
line vty
!

We’ll add some lines near the end for the two IP addresses with which we want to peer:

neighbor 192.168.100.192 remote-as 65022
neighbor 192.168.100.192 default-originate
neighbor 192.168.100.194 remote-as 65022
neighbor 192.168.100.194 default-originate

The last step at this end is to restart the quagga service via the service quagga restart command.

With this all in play we can now create  a LoadBalancer service, similar to the following for a WordPress application I had running:

apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
  selector:
    app: wordpress
    tier: frontend
  type: LoadBalancer
  loadBalancerIP: 10.40.14.40

When we check on this service after it’s been created, we can see that it has the external IP address specified (10.40.14.40) from the range that we configured in the MetalLB ConfigMap:

kubectl -n wordpress get svc wordpress
 
NAME              TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
wordpress         LoadBalancer   100.65.202.102   10.40.14.40   80:30347/TCP   40m

And thanks to using BGP, we can see that routing has been configured on my Linux system via BGP (the output below is truncated):

netstat -rn
 
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.40.14.32     192.168.100.192 255.255.255.224 UG        0 0          0 eth1

You can deploy MetalLB into each cluster you create but be sure to choose a different subnet for LoadBalancer IP addresses for each deployment or you could end up with IP address conflicts.

MetalLB might not be the best fit for your TKG on vSphere deployment so be sure to look into other options. In addition to MetalLB, you might consider HAProxy, Avi Networks (now a part of VMware), NGinx, NSX Container Plugin (when it’s supported) and many others. If you are deploying TKG to AWS, you’ll have built-in LoadBalancer support.