Inspections in TMC are using Sonobuoy within the cluster to do their work. You could use Sonobuoy directly and bypass TMC but it would take a little bit of configuration that TMC handles for you…it’s really not much if you’ve got access to TMC, why bother with it? You also wouldn’t get the great UI experience that TMC provides. You can read more about TMC Inspections at Inspecting Clusters.
For these examples, I have a TKG cluster created via TMC. There are no “real” workloads running in it so there should be little to worry about. In a production environment, you might want to consider running an inspection during off-hours to reduce the risk of increased load causing problems for any critical workloads.
There are three master nodes and five worker nodes…this isn’t terribly relevant but likely plays a small part in how long it takes for inspections to complete (the larger the cluster the longer it might take to run).
On the Inspections tab, we can see that none have been run yet.
Let’s start out with one of the quickest inspections, the Lite variety. Simply click on the Run Inspection menu and then select Lite.
We’ll have some progress while this is running but this one should be done in a minute or less.
You can click on the Success link to drill down into it and see the details (of which there are very little for the Lite test).
You can see that there was only one test that was run, checking to make sure that pods can be created/destroyed to validate node conformance. It took only nine seconds to complete and came back successful. This is the simplest test that can be run to get an idea of basic cluster functionality.
I was running a watch on the pods in my cluster while this was happening and you can see what was created and destroyed during this test. The pods that were created test the pod creation/deletion functionality and were put into their own namespace, pods-7254. You’ll see similar behavior in more extensive tests with different types of Kubernetes objects being put into object-specific namespaces:
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 Pending 0 0s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 Pending 0 1s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 ContainerCreating 0 1s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 ContainerCreating 0 1s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 ContainerCreating 0 11s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 ContainerCreating 0 11s
vmware-system-tmc sonobuoy-lite-job-f9a89e2d27114782 0/2 Pending 0 0s
vmware-system-tmc sonobuoy-lite-job-f9a89e2d27114782 0/2 Pending 0 0s
vmware-system-tmc sonobuoy-lite-job-f9a89e2d27114782 0/2 ContainerCreating 0 0s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 2/2 Running 0 12s
vmware-system-tmc sonobuoy-lite-job-f9a89e2d27114782 0/2 ContainerCreating 0 1s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 0/1 Pending 0 0s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 0/1 Pending 0 0s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 0/1 ContainerCreating 0 0s
vmware-system-tmc sonobuoy-lite-job-f9a89e2d27114782 2/2 Running 0 4s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 0/1 ContainerCreating 0 1s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 1/1 Running 0 3s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 1/1 Terminating 0 4s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 0/1 Terminating 0 6s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 2/2 Running 0 20s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 0/1 Terminating 0 15s
pods-7254 pod-submit-remove-5c854143-9fc6-4286-8341-c977e53d0d9a 0/1 Terminating 0 15s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 2/2 Running 0 29s
vmware-system-tmc sonobuoy-lite-job-f9a89e2d27114782 0/2 Completed 0 19s
vmware-system-tmc sonobuoy-lite-job-f9a89e2d27114782 0/2 Terminating 0 20s
vmware-system-tmc sonobuoy-lite-job-f9a89e2d27114782 0/2 Terminating 0 20s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 2/2 Running 0 32s
vmware-system-tmc agentupdater-workload-1598210880-lzr75 0/1 Pending 0 0s
vmware-system-tmc agentupdater-workload-1598210880-lzr75 0/1 Pending 0 0s
vmware-system-tmc agentupdater-workload-1598210880-lzr75 0/1 ContainerCreating 0 0s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 2/2 Running 0 33s
vmware-system-tmc agentupdater-workload-1598210880-lzr75 0/1 ContainerCreating 0 1s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 1/2 NotReady 0 34s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 Completed 0 35s
vmware-system-tmc agentupdater-workload-1598210880-lzr75 1/1 Running 0 2s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 Terminating 0 35s
vmware-system-tmc inspection-bt1c69lc5n14u85qtc00-dh5zq 0/2 Terminating 0 35s
vmware-system-tmc agentupdater-workload-1598210820-p4th7 0/1 Terminating 0 63s
vmware-system-tmc agentupdater-workload-1598210820-p4th7 0/1 Terminating 0 63s
vmware-system-tmc agentupdater-workload-1598210880-lzr75 0/1 Completed 0 4sThere is a pod that should always be running that is needed for inspections to work, inspection-extension. We can review the logs of the this pod to see some of the activity that occurred during the inspection (this output is heavily truncated).
kubectl --kubeconfig=kubeconfig-inspection-test.yml -n vmware-system-tmc logs inspection-extension-94748969d-bsdc2
{"func":"ReconcileInspection.Reconcile","level":"info","msg":"Reconciling for request: vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, PresignUpdateSuccessful, Presign URL in secret was successfully updated, 1598210622)","name":"inspection-extension-94748969d-bsdc2.162dfc17f4bd1178","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, PresignUpdateSuccessful, Presign URL in secret was successfully updated, 1598210622)","name":"inspection-extension-94748969d-bsdc2.162dfc17f4bd1178","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"func":"ReconcileInspection.Reconcile","level":"info","msg":"Reconciling for request: vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, SonobuoyJobProgress, sonobuoy scan job in progress, 1598210622)","name":"inspection-extension-94748969d-bsdc2.162dfc17f3ce7498","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, SonobuoyJobProgress, sonobuoy scan job in progress, 1598210622)","name":"inspection-extension-94748969d-bsdc2.162dfc17f3ce7498","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, SonobuoyJobCompleted, successfully completed Sonobuoy Scan Job, 1598210668)","name":"inspection-extension-94748969d-bsdc2.162dfc22bfb9945e","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, SonobuoyJobCompleted, successfully completed Sonobuoy Scan Job, 1598210668)","name":"inspection-extension-94748969d-bsdc2.162dfc22bfb9945e","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, PresignUpdateSuccessful, Presign URL in secret was successfully updated, 1598210622)","name":"inspection-extension-94748969d-bsdc2.162dfc17f4bd1178","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, PresignUpdateSuccessful, Presign URL in secret was successfully updated, 1598210622)","name":"inspection-extension-94748969d-bsdc2.162dfc17f4bd1178","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"func":"ReconcileInspect.Reconcile","level":"info","msg":"Reconciling for request: vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0","time":"2020-08-23T19:29:59Z"}
{"error":"Inspect.intents.tmc.cloud.vmware.com \"inspection-bt1c4esr4eemg7v574j0\" not found","func":"ReconcileInspect.Reconcile","level":"error","msg":"r.Get: object not found","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, SonobuoyJobCompleted, successfully completed Sonobuoy Scan Job, 1598210668)","name":"inspection-extension-94748969d-bsdc2.162dfc22bfb9945e","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"level":"info","msg":"Sending reconcile request for the inspect CRD with namespacedName vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0 - (Normal, SonobuoyJobCompleted, successfully completed Sonobuoy Scan Job, 1598210668)","name":"inspection-extension-94748969d-bsdc2.162dfc22bfb9945e","namespace":"vmware-system-tmc","time":"2020-08-23T19:29:59Z"}
{"func":"ReconcileInspect.Reconcile","level":"info","msg":"Reconciling for request: vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0","time":"2020-08-23T19:29:59Z"}
{"error":"Inspect.intents.tmc.cloud.vmware.com \"inspection-bt1c4esr4eemg7v574j0\" not found","func":"ReconcileInspect.Reconcile","level":"error","msg":"r.Get: object not found","time":"2020-08-23T19:29:59Z"}
{"func":"ReconcileInspection.Reconcile","level":"info","msg":"Reconciling for request: vmware-system-tmc/inspection-bt1c4esr4eemg7v574j0","time":"2020-08-23T19:29:59Z"}Let’s try running a Conformance inspection. This one can take considerably longer as it tests nearly all aspects of the cluster. Right off the bat we can see that there are going to be a significant number of tests run:
And just a few minutes in we see loads more activity in the cluster:
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 0/2 Pending 0 0s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 0/2 Pending 0 0s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 0/2 ContainerCreating 0 0s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 0/2 ContainerCreating 0 0s
vmware-system-tmc agentupdater-workload-1598211420-dhdhn 0/1 Pending 0 0s
vmware-system-tmc agentupdater-workload-1598211420-dhdhn 0/1 Pending 0 0s
vmware-system-tmc agentupdater-workload-1598211420-dhdhn 0/1 ContainerCreating 0 0s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 0/2 ContainerCreating 0 19s
vmware-system-tmc agentupdater-workload-1598211420-dhdhn 0/1 ContainerCreating 0 0s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 2/2 Running 0 19s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 2/2 Running 0 19s
vmware-system-tmc sonobuoy-e2e-job-dac14561d3c649de 0/2 Pending 0 0s
vmware-system-tmc sonobuoy-e2e-job-dac14561d3c649de 0/2 Pending 0 0s
vmware-system-tmc sonobuoy-e2e-job-dac14561d3c649de 0/2 ContainerCreating 0 0s
vmware-system-tmc sonobuoy-e2e-job-dac14561d3c649de 0/2 ContainerCreating 0 1s
vmware-system-tmc agentupdater-workload-1598211420-dhdhn 1/1 Running 0 1s
vmware-system-tmc agentupdater-workload-1598211360-dxctk 0/1 Terminating 0 63s
vmware-system-tmc agentupdater-workload-1598211360-dxctk 0/1 Terminating 0 63s
vmware-system-tmc agentupdater-workload-1598211420-dhdhn 0/1 Completed 0 4s
vmware-system-tmc sonobuoy-e2e-job-dac14561d3c649de 2/2 Running 0 19s
kubelet-test-4509 bin-false31dd3f4a-bfc7-4b34-b4a7-d94fafd09753 0/1 Pending 0 0s
kubelet-test-4509 bin-false31dd3f4a-bfc7-4b34-b4a7-d94fafd09753 0/1 Pending 0 0s
kubelet-test-4509 bin-false31dd3f4a-bfc7-4b34-b4a7-d94fafd09753 0/1 ContainerCreating 0 0s
kubelet-test-4509 bin-false31dd3f4a-bfc7-4b34-b4a7-d94fafd09753 0/1 ContainerCreating 0 0s
kubelet-test-4509 bin-false31dd3f4a-bfc7-4b34-b4a7-d94fafd09753 0/1 Error 0 1s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 2/2 Running 0 55s
services-2195 pod1 0/1 Pending 0 0s
services-2195 pod1 0/1 Pending 0 0s
services-2195 pod1 0/1 ContainerCreating 0 0s
services-2195 pod1 0/1 ContainerCreating 0 0s
kubelet-test-4509 bin-false31dd3f4a-bfc7-4b34-b4a7-d94fafd09753 0/1 Terminating 0 9s
kubelet-test-4509 bin-false31dd3f4a-bfc7-4b34-b4a7-d94fafd09753 0/1 Terminating 0 9s
services-2195 pod1 1/1 Running 0 4s
services-2195 pod2 0/1 Pending 0 0s
services-2195 pod2 0/1 Pending 0 0s
services-2195 pod2 0/1 ContainerCreating 0 0s
services-2195 pod2 0/1 ContainerCreating 0 1s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 2/2 Running 0 63s
services-2195 pod2 1/1 Running 0 4s
services-2195 pod1 1/1 Terminating 0 10s
services-2195 pod1 0/1 Terminating 0 10s
services-2195 pod2 1/1 Terminating 0 6s
security-context-test-1467 busybox-user-65534-1ef6222d-2150-47bc-a678-73d77497e141 0/1 Pending 0 0s
services-2195 pod2 0/1 Terminating 0 7s
security-context-test-1467 busybox-user-65534-1ef6222d-2150-47bc-a678-73d77497e141 0/1 Pending 0 0s
security-context-test-1467 busybox-user-65534-1ef6222d-2150-47bc-a678-73d77497e141 0/1 ContainerCreating 0 0s
security-context-test-1467 busybox-user-65534-1ef6222d-2150-47bc-a678-73d77497e141 0/1 ContainerCreating 0 1s
security-context-test-1467 busybox-user-65534-1ef6222d-2150-47bc-a678-73d77497e141 0/1 Completed 0 1s
statefulset-6532 ss2-0 0/1 Pending 0 0s
statefulset-6532 ss2-0 0/1 Pending 0 0s
statefulset-6532 ss2-0 0/1 ContainerCreating 0 0s
vmware-system-tmc inspection-bt1cai5le0sd36r03nvg-tdspt 2/2 Running 0 72s
statefulset-6532 ss2-0 0/1 ContainerCreating 0 1s
services-2195 pod2 0/1 Terminating 0 11s
services-2195 pod2 0/1 Terminating 0 11s
services-2195 pod1 0/1 Terminating 0 18s
services-2195 pod1 0/1 Terminating 0 18s
statefulset-6532 ss2-0 0/1 Running 0 5s
security-context-test-1467 busybox-user-65534-1ef6222d-2150-47bc-a678-73d77497e141 0/1 Terminating 0 7s
security-context-test-1467 busybox-user-65534-1ef6222d-2150-47bc-a678-73d77497e141 0/1 Terminating 0 8s
statefulset-6532 ss2-0 1/1 Running 0 6s
statefulset-6532 ss2-1 0/1 Pending 0 0s
statefulset-6532 ss2-1 0/1 Pending 0 0s
statefulset-6532 ss2-1 0/1 ContainerCreating 0 0s
statefulset-6532 ss2-1 0/1 ContainerCreating 0 1s
vmware-system-tmc agentupdater-workload-1598211480-b96x5 0/1 Pending 0 0s
vmware-system-tmc agentupdater-workload-1598211480-b96x5 0/1 Pending 0 0s
vmware-system-tmc agentupdater-workload-1598211480-b96x5 0/1 ContainerCreating 0 0s
vmware-system-tmc agentupdater-workload-1598211480-b96x5 0/1 ContainerCreating 0 0s
vmware-system-tmc agentupdater-workload-1598211480-b96x5 1/1 Running 0 1s
vmware-system-tmc agentupdater-workload-1598211420-dhdhn 0/1 Terminating 0 63s
vmware-system-tmc agentupdater-workload-1598211420-dhdhn 0/1 Terminating 0 63s
vmware-system-tmc agentupdater-workload-1598211480-b96x5 0/1 Completed 0 4s
statefulset-6532 ss2-1 0/1 Running 0 7s
statefulset-6532 ss2-1 1/1 Running 0 8s
statefulset-6532 ss2-2 0/1 Pending 0 0s
statefulset-6532 ss2-2 0/1 Pending 0 0s
statefulset-6532 ss2-2 0/1 ContainerCreating 0 0s
statefulset-6532 ss2-2 0/1 ContainerCreating 0 1s
statefulset-6532 ss2-2 0/1 Running 0 5s
statefulset-6532 ss2-2 1/1 Running 0 5s
statefulset-6532 ss2-2 1/1 Terminating 0 16s
statefulset-6532 ss2-2 0/1 Terminating 0 17s
statefulset-6532 ss2-2 0/1 Terminating 0 18s
statefulset-6532 ss2-2 0/1 Terminating 0 18s
statefulset-6532 ss2-2 0/1 Pending 0 0s
statefulset-6532 ss2-2 0/1 Pending 0 0s
statefulset-6532 ss2-2 0/1 ContainerCreating 0 0s
statefulset-6532 ss2-2 0/1 ContainerCreating 0 1s
statefulset-6532 ss2-2 0/1 Running 0 5s
statefulset-6532 ss2-2 1/1 Running 0 5s
statefulset-6532 ss2-0 1/1 Terminating 0 40s
statefulset-6532 ss2-0 1/1 Terminating 0 40s
statefulset-6532 ss2-2 1/1 Terminating 0 8s
statefulset-6532 ss2-0 0/1 Pending 0 0s
statefulset-6532 ss2-2 1/1 Terminating 0 8s
statefulset-6532 ss2-0 0/1 Pending 0 0s
statefulset-6532 ss2-0 0/1 ContainerCreating 0 0s
statefulset-6532 ss2-0 0/1 ContainerCreating 0 1s
statefulset-6532 ss2-0 0/1 Running 0 2sThis same level of activity kept up for most of the inspection run. You can now see what was hinted at earlier also, that there are numerous namespaces created for the different Kubernetes object types being tested.
And if you’re tailing the logs from the inspection pod you’ll see that they’re flying by.
When it’s done we should see the same successful completion as was noted for the Lite test.
You can also drill down into the results of the Conformance test to see the details and if anything failed (this a truncated screenshot).
We can see that this test took just about one hour and 11 minutes to complete. This cluster had no issues so there’s really nothing left to do here at this point.
I tried this same inspection against a cluster that I fully expected to fail and you can see that my suspicion was not unwarranted.
This was running in a nested environment, which would explain the two hours and 35 minutes it took to complete. You’ll see that when you have tests that failed, you can expand each failed test to get a better idea of what happened.
In nearly every one of these failed tests, the reason for the failure was a timeout or lack of response. Most of my labs run in severely resource-constrained environments so this is just about what I was expecting to see.
The last kind of inspection we can run is a CIS Benchmark inspection, which validates that the cluster is properly secured. This one should complete fairly quickly.
Per the Note on this page, it’s not unexpected for some items to fail based on how the cluster was instantiated and the limitations of some of the testing methods. You can expand any of the failed or warning items to see details about the condition and possible remediation steps:
