Tanzu Kubernetes Grid provides several different Fluent Bit manifest files to help you deploy and configure Fluent Bit for use with Splunk, Elasticsearch, Kafka and a generic HTTP endpoint. In this post, I’ll walk though not only the Fluent Bit configuration which VMware has documented but the deployment of Elasticsearch/Kibana in a TKG cluster. You’ll notice that much of this post is similar to How to Configure Fluent Bit and Splunk in Tanzu Kubernetes Grid and How to Configure Fluent Bit and Kafka in Tanzu Kubernetes Grid since they all share most of the same Fluent Bit configuration.
In addition to the tkg
cli utility and the OVA files needed to stand up a TKG cluster, you’ll need to download the extensions file as well from https://my.vmware.com/web/vmware/downloads/details?downloadGroup=TKG-112&productId=988&rPId=46507. This contains the manifest files needed to deploy/configure the authentication, log forwarding and ingress solutions that VMware supports for TKG.
I’ve made use of MetalLB to provide LoadBalancer functionality (per my previous blog post, How to Deploy MetalLB with BGP in a Tanzu Kubernetes Grid 1.1 Cluster) but you could also use a NodePort service (which is the default for the Fluent Bit manifests that VMware ships) or use an Ingress resource.
I have a storage class named k8s-policy, which maps to an NFS volume mounted to all of my ESXi hosts, and accessible to Kubernetes via vSphere CNS.
Once you have a TKG cluster up and running, the first step will be to extract the contents of the extensions bundle. You should see a folder structure similar to the following:
ls tkg-extensions-v1.1.0/
authentication cert-manager ingress logging
We’ll be working in the logging section so you can start to focus on the manifest files in this location and its sub-directories.
The first step to deploying Fluent Bit in a TKG cluster is to create the tanzu-system-logging namespace and the needed RBAC components. These steps are the same regardless of the logging backend. You can read more about this step in detail at Create Namespace and RBAC Components. You should inspect the manifest files prior to applying them to make sure that you’re okay with the RBAC objects being created. Note: If you’ve already configured Fluent Bit for Splunk or Kafka, you can skip applying these four manifests.
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/00-fluent-bit-namespace.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/01-fluent-bit-service-account.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/02-fluent-bit-role.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/03-fluent-bit-role-binding.yaml
For Elasticsearch/Kibana I decided to use a simple(ish) deployment manifest. A few things to point out from this example:
- I’m deploying everything to the tanzu-system-logging namespace.
- I’m using a LoadBalancer service and specifying the IP addresses to use. I’m using 10.40.14.36 for the elasticsearch service and 10.40.14.37 for the kibana service. I have also created DNS records mapping elasticsearch.corp.local to 10.40.14.36 and kibana.corp.local to 10.40.14.37.
- I’m using PVCs for storage and using a storage policy named k8s-policy. I’v chosen a small size for my PVCs, 10GB, as this is only a POC.
echo 'kind: Service
apiVersion: v1
metadata:
name: elasticsearchsvc
namespace: tanzu-system-logging
labels:
app: elasticsearch
spec:
type: LoadBalancer
loadBalancerIP: 10.40.14.36
ports:
- protocol: TCP
port: 9200
targetPort: 9200
selector:
app: elasticsearch
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: tanzu-system-logging
name: es-pv
labels:
app: elasticsearch
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 10Gi
storageClassName: k8s-policy
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch
namespace: tanzu-system-logging
labels:
app: elasticsearch
spec:
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
ports:
- containerPort: 9200
env:
- name: "discovery.type"
value: "single-node"
- name: "node.master"
value: \"true\"
- name: "node.data"
value: \"true\"
- name: "xpack.security.enabled"
value: \"false\"
- name: "bootstrap.memory_lock"
value: \"true\"
- name: "ES_JAVA_OPTS"
value: "-Xms1024m -Xmx1024m"
volumeMounts:
- name: es-pv
mountPath: /usr/share/elasticsearch/data
securityContext:
fsGroup: 1000
volumes:
- name: es-pv
persistentVolumeClaim:
claimName: es-pv
---
kind: Service
apiVersion: v1
metadata:
name: kibanasvc
namespace: tanzu-system-logging
labels:
app: kibana
spec:
type: LoadBalancer
loadBalancerIP: 10.40.14.37
ports:
- protocol: TCP
port: 5601
targetPort: 5601
selector:
app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: tanzu-system-logging
labels:
app: kibana
spec:
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.7.0
ports:
- containerPort: 5601
env:
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearchsvc:9200' > tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/elasticsearch.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/elasticsearch.yaml
As with our Splunk and Kafka deployments, you should now have a number of objects created in the tanzu-system-logging namespace:
kubectl -n tanzu-system-logging get po,svc,pvc,pv,secrets,cm
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-8fdd858cf-wm4cr 1/1 Running 0 1d
pod/kibana-6695674d68-qtj9t 1/1 Running 0 1d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearchsvc LoadBalancer 100.65.139.209 10.40.14.36 9200:32207/TCP 1d
service/kibanasvc LoadBalancer 100.65.218.207 10.40.14.37 5601:31937/TCP 1d
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/es-pv Bound pvc-2871a0b6-82f3-4251-8d55-459472ec26cf 10Gi RWO k8s-policy 1d
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-2871a0b6-82f3-4251-8d55-459472ec26cf 10Gi RWO Delete Bound tanzu-system-logging/es-pv k8s-policy 1d
NAME DATA AGE
configmap/fluent-bit-config-elasticsearch 6 1d
You can run a simple curl command against the elasticsearch service to validate functionality:
curl -k elasticsearch.corp.local:9200
{
"name" : "elasticsearch-8fdd858cf-9bpbh",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "iumzFF5KRmC-G58-tW-BxA",
"version" : {
"number" : "7.7.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
"build_date" : "2020-05-12T02:01:37.602180Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
We still have a few more Fluent Bit pieces to configure before we can make use of Elasticsearch and Kibana.
The tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
file has the instance-specific configuration information that we’ll need to provide to allow Fluent Bit to forward logs to our Elasticsearch deployment. In my example, I am changing the following from the default values:
- Setting the Cluster name to vsphere-test (since that’s the name of my TKG workload cluster).
- Setting the Instance name to vsphere (this was an arbitrary choice).
- Setting the Elasticsearch host to elasticsearch.corp.local, the FQDN that maps to the LoadBalancer service IP address for the elasticsearch service.
- Setting the Elasticsearch port to 9200, the default.
sed -i 's/<TKG_CLUSTER_NAME>/vsphere-test/' tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
sed -i 's/<TKG_INSTANCE_NAME>/vsphere/' tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
sed -i 's/<FLUENT_ELASTICSEARCH_HOST>/elasticsearch.corp.local/' tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
sed -i 's/<FLUENT_ELASTICSEARCH_PORT>/9200/' tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
The only other file we need to be concerned with is tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/05-fluent-bit-ds.yaml
but it’s fine in the default configuration.
Now we can deploy Fluent Bit and see what data is getting sent to Elasticsearch:
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/04-fluent-bit-configmap.yaml
kubectl apply -f tkg-extensions-v1.1.0/logging/fluent-bit/vsphere/output/elasticsearch/05-fluent-bit-ds.yaml
Since I have a DNS record for kibana.corp.local, I can point a browser to http://kibana.corp.local:5601 to start looking at the data that Fluent Bit is sending to Elasticsearch (there is no login to Kibana in this configuration):
When you first access Kibana, you’ll need to create a basic index pattern to see the data from your TKG cluster. Set the index patter to logstash-*:
I chose not to use a Time Filter on the next page, just to ensure I wouldn’t be missing any data:
Once your index pattern is created, you can click on the Discover link on the left to drill down into the received data:
Hi
just one question, how can I set up a user and password to send the data to elastic?
Additionally I need to enable TLS and TLS.verify Off how can I add this to the configuration?
Great block by the way.
Thanks Stephan
It’s not something I’ve tried doing myself and the process for configuring fluentbit has changed with the 1.2 version of TKG. You might check out https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.2/vmware-tanzu-kubernetes-grid-12/GUID-extensions-logging-fluentbit.html?hWord=N4IghgNiBcIKYTAZwC4EsDGIC+Q to see if it provides any insight.
Pingback: How to Configure Fluent-Bit to NOT use TLS with Splunk in TKG 1.2 – Little Stuff