How to Configure Fluent Bit and Elasticsearch in Tanzu Kubernetes Grid

Tanzu Kubernetes Grid provides several different Fluent Bit manifest files to help you deploy and configure Fluent Bit for use with Splunk, Elasticsearch, Kafka and a generic HTTP endpoint. In this post, I’ll walk though not only the Fluent Bit configuration which VMware has documented but the deployment of Elasticsearch/Kibana in a TKG cluster. You’ll notice that much of this post is similar to How to Configure Fluent Bit and Splunk in Tanzu Kubernetes Grid and How to Configure Fluent Bit and Kafka in Tanzu Kubernetes Grid since they all share most of the same Fluent Bit configuration.

In addition to the tkg cli utility and the OVA files needed to stand up a TKG cluster, you’ll need to download the extensions file as well from https://my.vmware.com/web/vmware/downloads/details?downloadGroup=TKG-112&productId=988&rPId=46507. This contains the manifest files needed to deploy/configure the authentication, log forwarding and ingress solutions that VMware supports for TKG.

I’ve made use of MetalLB to provide LoadBalancer functionality (per my previous blog post, How to Deploy MetalLB with BGP in a Tanzu Kubernetes Grid 1.1 Cluster) but you could also use a NodePort service (which is the default for the Fluent Bit manifests that VMware ships) or use an Ingress resource.

I have a storage class named k8s-policy, which maps to an NFS volume mounted to all of my ESXi hosts, and accessible to Kubernetes via vSphere CNS.

Once you have a TKG cluster up and running, the first step will be to extract the contents of the extensions bundle. You should see a folder structure similar to the following:

ls tkg-extensions-v1.1.0/
authentication  cert-manager  ingress  logging

We’ll be working in the logging section so you can start to focus on the manifest files in this location and its sub-directories.

The first step to deploying Fluent Bit in a TKG cluster is to create the tanzu-system-logging namespace and the needed RBAC components. These steps are the same regardless of the logging backend.  You can read more about this step in detail at Create Namespace and RBAC Components.  You should inspect the manifest files prior to applying them to make sure that you’re okay with the RBAC objects being created. Note: If you’ve already configured Fluent Bit for Splunk or Kafka,  you can skip applying these four manifests.

kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/00-fluent-bit-namespace.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/01-fluent-bit-service-account.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/02-fluent-bit-role.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/03-fluent-bit-role-binding.yaml

For Elasticsearch/Kibana I decided to use a simple(ish) deployment manifest. A few things to point out from this example:

• I’m deploying everything to the tanzu-system-logging namespace.
• I’m using a LoadBalancer service and specifying the IP addresses to use. I’m using 10.40.14.36 for the elasticsearch service and 10.40.14.37 for the kibana service. I have also created DNS records mapping elasticsearch.corp.local to 10.40.14.36 and kibana.corp.local to 10.40.14.37.
• I’m using PVCs for storage and using a storage policy named k8s-policy. I’v chosen a small size for my PVCs, 10GB, as this is only a POC.

echo 'kind: Service
apiVersion: v1
metadata:
  name: elasticsearchsvc
  namespace: tanzu-system-logging
  labels:
    app: elasticsearch
spec:
  type: LoadBalancer
  loadBalancerIP: 10.40.14.36
  ports:
    - protocol: TCP
      port: 9200
      targetPort: 9200
  selector:
    app: elasticsearch
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: tanzu-system-logging
  name: es-pv
  labels:
    app: elasticsearch
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: k8s-policy
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: tanzu-system-logging
  labels:
    app: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
        ports:
        - containerPort: 9200
        env:
        - name: "discovery.type"
          value: "single-node"
        - name: "node.master"
          value: \"true\"
        - name: "node.data"
          value: \"true\"
        - name: "xpack.security.enabled"
          value: \"false\"
        - name: "bootstrap.memory_lock"
          value: \"true\"
        - name: "ES_JAVA_OPTS"
          value: "-Xms1024m -Xmx1024m"
        volumeMounts:
        - name: es-pv
          mountPath: /usr/share/elasticsearch/data
      securityContext:
        fsGroup: 1000
      volumes:
      - name: es-pv
        persistentVolumeClaim:
          claimName: es-pv
---
kind: Service
apiVersion: v1
metadata:
  name: kibanasvc
  namespace: tanzu-system-logging
  labels:
    app: kibana
spec:
  type: LoadBalancer
  loadBalancerIP: 10.40.14.37
  ports:
    - protocol: TCP
      port: 5601
      targetPort: 5601
  selector:
    app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: tanzu-system-logging
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.7.0
        ports:
        - containerPort: 5601
        env:
        - name: ELASTICSEARCH_HOSTS
          value: http://elasticsearchsvc:9200' > tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/elasticsearch.yaml

kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/elasticsearch.yaml

As with our Splunk and Kafka deployments, you should now have a number of objects created in the tanzu-system-logging namespace:

kubectl -n tanzu-system-logging get po,svc,pvc,pv,secrets,cm
NAME                                  READY   STATUS    RESTARTS   AGE
pod/elasticsearch-8fdd858cf-wm4cr     1/1     Running   0          1d
pod/kibana-6695674d68-qtj9t           1/1     Running   0          1d

NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP               PORT(S)                                                                                                             AGE
service/elasticsearchsvc                LoadBalancer   100.65.139.209   10.40.14.36               9200:32207/TCP                                                                                                      1d
service/kibanasvc                       LoadBalancer   100.65.218.207   10.40.14.37               5601:31937/TCP                                                                                                      1d

NAME                                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES          STORAGECLASS   AGE
persistentvolumeclaim/es-pv                            Bound    pvc-2871a0b6-82f3-4251-8d55-459472ec26cf   10Gi       RWO                   k8s-policy     1d

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                        STORAGECLASS   REASON   AGE
persistentvolume/pvc-2871a0b6-82f3-4251-8d55-459472ec26cf   10Gi       RWO            Delete           Bound    tanzu-system-logging/es-pv                                   k8s-policy              1d

NAME                                        DATA   AGE
configmap/fluent-bit-config-elasticsearch   6      1d

You can run a simple curl command against the elasticsearch service to validate functionality:

curl -k  elasticsearch.corp.local:9200
{
  "name" : "elasticsearch-8fdd858cf-9bpbh",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "iumzFF5KRmC-G58-tW-BxA",
  "version" : {
    "number" : "7.7.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
    "build_date" : "2020-05-12T02:01:37.602180Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

We still have a few more Fluent Bit pieces to configure before we can make use of Elasticsearch and Kibana.

The tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml file has the instance-specific configuration information that we’ll need to provide to allow Fluent Bit to forward logs to our Elasticsearch deployment. In my example, I am changing the following from the default values:

• Setting the Cluster name to vsphere-test (since that’s the name of my TKG workload cluster).
• Setting the Instance name to vsphere (this was an arbitrary choice).
• Setting the Elasticsearch host to elasticsearch.corp.local, the FQDN that maps to the LoadBalancer service IP address for the elasticsearch service.
• Setting the Elasticsearch port to 9200, the default.

sed -i 's/<TKG_CLUSTER_NAME>/vsphere-test/' tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
sed -i 's/<TKG_INSTANCE_NAME>/vsphere/' tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
sed -i 's/<FLUENT_ELASTICSEARCH_HOST>/elasticsearch.corp.local/' tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
sed -i 's/<FLUENT_ELASTICSEARCH_PORT>/9200/' tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml

The only other file we need to be concerned with is tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/05-fluent-bit-ds.yaml but it’s fine in the default configuration.

Now we can deploy Fluent Bit and see what data is getting sent to Elasticsearch:

kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/05-fluent-bit-ds.yaml

Since I have a DNS record for kibana.corp.local, I can point a browser to http://kibana.corp.local:5601 to start looking at the data that Fluent Bit is sending to Elasticsearch (there is no login to Kibana in this configuration):

When you first access Kibana, you’ll need to create a basic index pattern to see the data from your TKG cluster. Set the index patter to logstash-*:

I chose not to use a Time Filter on the next page, just to ensure I wouldn’t be missing any data:

Once  your index pattern is created, you can click on the Discover link on the left to drill down into the received data: