I’ve stepped away from many of the Tanzu products for the last few months but was recently asked by one of my coworkers who supports Tanzu Observability if we could get an instance of Tanzu Application Service (TAS) running in a vSphere environment with NSX-T networking. I had never used this particular product before but a quick bit of research showed that it installed similarly to Tanzu Kubernetes Grid Integrated Edition (TGKI) so I felt it shouldn’t be too far outside of my comfort zone to get it up and running. Check out the results below.
If you’re not familiar with TAS, in a nutshell, it provides the ability to quickly and reliably spin up applications on the cloud of your choice. It was formerly called Pivotal Application Service (PAS) prior to VMware acquiring Pivotal and moving most of their products under the Tanzu portfolio. You can read more about TAS at https://tanzu.vmware.com/application-service.
vSphere
This installation was done in a nested vApp running under vCloud Director and started as a simple vSphere 7.0 U3 deployment with a few hundred GB of NFS storage. My first step was to create a few resource pools to house the various TAS components that would be deployed.
To make full use of TAS, there are four organizational categories to consider:
- Infrastructure – bosh and Opsman
- Deployment – TAS components
- Services – marketplace and on-demand services
- Isolation – isolated deployment workloads
With this in mind, I created four resources pools: TAS-Infrastructure, TAS-Deployment, TAS-Services, TAS-Isolation.
NSX-T
The basic NSX-T configuration is the same as noted in previous posts I’ve done related to TKGI (see http://little-stuff.com/2020/09/30/tgki-1-9-with-windows-workers/#Configure_NSX-T_Resources for a detailed walkthrough of configuring NSX-T objects).
The main things that should be in place prior to starting are a T0 logical router (t0-tas in my setup)
T1 logical routers for each of the four organizational categories noted earlier( T1-Router-TAS-Infrastructure, T1-Router-TAS-Deployment, T1-Router-TAS-Services, T1-Router-TAS-Isolation in my setup)
Defined subnets for each of the four T1 routers (172.31.0.0/24, 172.31.1.0/24, 172.31.2.0/24, 172.31.3.0/24 respectively in my setup)
Logical switches for each of four organizational categories noted earlier (ls-tas-infrastructure, ls-tas-deployment, ls-tas-services, ls-tas-isolation respectively in my setup) and an uplink logical switch (uplink-vlan-ls in my setup)
NAT rules for inbound communication to the bosh and opsman VM and outbound for anything on the infrastructure network (172.31.0.0/24)
IP Block to be used for container networking
External IP Pool used for NAT
A load balancer configuration to allow traffic to access the deployed applications. This was the main unique difference between the NSX-T setup for TAS vs the same setup for TKGI, and it was not trivial to get it configured properly. My deployment network is 172.31.1.0/24 and my external network is 10.40.14.0/24. I decided to use 10.40.14.65 as the external, load-balanced address for accessing resources on the deployment network.
Create Active Health Monitors (health checks) for use by the virtual server:
In the NSX-T Manager UI, navigate to Networking > Network Services > Load Balancing > Monitors > Active Health Monitors.
Create the health monitor for web load balancing:
Click +ADD.
Enter Monitor Properties:
Name: tas-web-monitor
Health Check Protocol: LbHttpMonitor
Monitoring Port: 8080
Click Next.
Enter Health Check Parameters:
HTTP Method: GET
HTTP Request URL: /health
HTTP Response Code: 200
Click Finish.
Create the health monitor for TCP load balancing:
Click +ADD.
Enter Monitor Properties:
Name: tas-tcp-monitor
Health Check Protocol: LbHttpMonitor
Monitoring Port: 80
Click Next.
Enter Health Check Parameters:
HTTP Method: GET
HTTP Request URL: /health
HTTP Response Code: 200
Click Finish.
Create the health monitor for SSH load balancing:
Click +ADD.
Enter Monitor Properties:
Name: tas-ssh-monitor
Health Check Protocol: LbTcpMonitor
Monitoring Port: 2222
Click Next, then click Finish.
Create server pools (collections of VMs which handle traffic) for use by the virtual server:
In the NSX-T Manager UI, navigate to Networking > Network Services > Load Balancing > Server Pools.
Create the server pool for web load balancing:
Click +ADD to add a new pool.
Enter General Properties:
Name: tas-web-pool
Click Next.
Enter SNAT Translation:
Translation Mode: IP List
IP Address: 172.31.1.65-172.31.1.128
Click Next.
Enter Pool Members:
Membership Type: Static
Click Next.
Enter Health Monitors:
Active Health Monitor: tas-web-monitor
Click Finish.
Create the server pool for TCP load balancing:
Click +ADD to add new pool.
Enter General Properties:
Name: tas-tcp-pool
Click Next.
Enter SNAT Translation:
Translation Mode: Transparent
Click Next.
Enter Pool Members:
Membership Type: Static
Click Next.
Enter Health Monitors:
Active Health Monitor: tas-tcp-monitor
Click Finish.
Create the server pool for SSH load balancing:
Click +ADD to add new pool.
Enter General Properties:
Name: tas-ssh-pool
Click Next.
Enter SNAT Translation:
Translation Mode: Transparent
Click Next.
Enter Pool Members:
Membership Type: Static
Click Next.
Enter Health Monitors:
Active Health Monitor: tas-ssh-monitor
Click Finish.
Create virtual servers:
In the NSX-T Manager UI, navigate to Networking > Network Services > Load Balancing > Virtual Servers.
Create the virtual servers which forward HTTP and HTTPS traffic to apps in the foundation:
Click +ADD.
Enter General Properties:
Name: tas-web-vs
Application Type: Layer 4 (TCP)
Application Profile: nsx-default-lb-fast-tcp-profile
Click Next.
Enter Virtual Server Identifiers:
IP Address: 10.40.14.65
Port: 80
Enter Server Pool and Rules:
Default Server Pool: tas-web-pool
Click Next several times, then click Finish.
Repeat but name the virtual server tas-https-vs and set the port to 443.
Create the virtual server which forwards traffic to apps with custom ports to the foundation:
Click +ADD to add a new virtual server.
Enter General Properties:
Name: tas-tcp-vs
Application Type: Layer 4 (TCP)
Application Profile: nsx-default-lb-fast-tcp-profile
Click Next.
Enter Virtual Server Identifiers:
IP Address: 10.40.14.65
Port: 1024-1123,5900
Click Next.
Enter Server Pool and Rules:
Default Server Pool: tas-tcp-pool
Click Next, then click Finish.
Create the virtual server which forwards SSH traffic to the foundation:
Click +ADD to add a new virtual server.
Enter General Properties:
Name: tas-ssh-vs
Application Type: Layer 4 (TCP)
Application Profile: nsx-default-lb-fast-tcp-profile
Click Next.
Enter Virtual Server Identifiers:
IP Address: 10.40.14.65
Port: 2222
Click Next.
Enter Server Pool and Rules:
Default Server Pool: tas-ssh-pool
Click Next, then click Finish.
Create the load balancer:
In the NSX-T Manager UI, navigate to Networking > Network Services > Load Balancing > Load Balancers.
Click +ADD.
Enter the fields:
Name: tas-lb
Load Balancer Size: small
Click OK.
Select tas-lb.
Click Actions > Attach to a Virtual Server, and then select tas-web-vs. Repeat this procedure for the Virtual Servers tas-tcp-vs and tas-ssh-vs.
Click Actions > Attach to a Logical Router, and then select T1-Router-TAS for VMs-Deployment.
DNS
There are some unique DNS requirements that need to be in place for the deployed applications to be accessible.
- Create three sub-domains under your primary domain named tas-system, tas-apps and tas-ssh.
- In each sub-domain, create a wildcard DNS record (*.tas-system.corp.vmw for example) that maps to the external IP address used when creating the load balancer in NSX-T (10.40.14.64 in my setup).
TAS Certificate
A wildcard certificate must be created for the TAS components and there are a few different addresses (Subject Alternative Names, or SANs) that need to be included. The following is an example of the configuration file I used to create a CSR for my Microsoft Certificate Authority to generate a valid certificate:
[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[ req_distinguished_name ]
countryName = US
stateOrProvinceName = California
localityName = Palo Alto
organizationName = VMware
organizationalUnitName = VMware
commonName = corp.vmw
[ v3_req ]
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *corp.vmw
DNS.2 = *.tas-system.corp.vmw
DNS.3 = *.tas-app.corp.vmw
DNS.4 = *.tas-ssh.corp.vmw
DNS.5 = uaa.tas-system.corp.vmw
DNS.6 = login.tas-system.corp.vmw
DNS.7 = api.tas-system.corp.vmwBe sure to save the certificate and key files as they will be used later.
Opsman
Opsman (Tanzu Operations Manager) and bosh are largely deployed the same as noted in my previous post, http://little-stuff.com/2020/09/30/tgki-1-9-with-windows-workers/#Deploying_Tanzu_Operations_Manager. The main difference between the TAS and TKGI configurations is that there were four availability zones (AZs) and four networks created for TAS, one for each of the organizational categories noted previously (infrastructure, deployment, services, isolation). Each AZ uses the corresponding resource pool in vSphere and each network uses the corresponding logical switch from NSX-T. The following shows the deployment AZ and network as an example:
Ensure that the network settings are appropriately configured for each network category. This is largely going to come down to setting the appropriate subnet information for each network. In this example, the 172.31.1.0/24 network is used but it would differ for the other network types (173.31.0.0/24 for the infrastructure network for example).
On the Assign AZs and Networks page, the Singleton AZ and Network are both set to TAS-Infrastructure:
If you’re satisfied with the bosh configuration, you can navigate to the Review Pending Changes page and click the Apply Changes button to deploy the bosh VM.
NSX-T Tile
TAS will be using the NSX-T Container Plugin so we’ll need to install and configure it in Opsman. You can download the NSX-T tile from PivNet.
Click the Import a Product button in Opsman and choose the NSX-T tile file. When the tile is imported, click the + button next to it on the left side of the Opsman UI to move it over to the installation dashboard.
Click on the NSX-T tile and configure the NSX Manager page as appropriate to your environment.
Click the Save button and move on to the NCP page.
The PAS Foundation Name value is arbitrary but the other items are dependent on how you have configured NSX-T.
Click the Add button next to IP Blocks of Container Networks. If you have already created an IP Block in NSX-T, you only need to enter it’s name here and can leave the CIDR field blank.
Click the Add button next to IP Pools used to provide External (NAT) IP Addresses to Org Networks. As with the last step, you only need to enter the name if it’s already created in NSX-T.
You can deploy the NSX-T tile now (Review Pending Changes > Apply Changes) or wait until after the TAS tile is configured and knock them both out at the same time.
TAS Tile
As with the NSX-T tile, you can download the TAS tile from PivNet and load it into Opsman. You’ll see that there are three different download options here:
Tanzu Application Service is the primary TAS tile, but if you are in a resource constrained environment (like I am), you may want to try the Small Footprint TAS option. It deploys fewer VMs as several of the VMs pull double duty in their functionality. You will also want to download the cf CLI as it will be used for deploying applications later.
Once the TAS tile is imported, click the + button next to it on the left side of the Opsman UI to move it over to the installation dashboard.
Click on the TAS tile. You should be dropped into the Assign AZs and Networks page where you can select the appropriate options from what was configured on the bosh tile (TAS-Deployment in my setup).
On the Domains page, enter the system and apps DNS sub-domains created earlier (tas-system.corp.vmw and tas-app.corp.vmw in my setup)
On the Networking page, click the Add button next to Certificates and private keys for the Gorouter and HAProxy. The name is arbitrary but the Certificate and private key data should be the TAS certificate created earlier.
Set the TLS termination point to Gorouter
Set HAProxy forwards all requests to the Gorouter over TLS to Disable
Lastly, set Container network interface plugin to External (since we’re using NSX-T)
On the App Security Groups page, type X in the box as acknowledgement
On the UAA page, paste the TAS certificate and key in the SAML service provider credentials section.
On the CredHub page, click the Add button next to Internal encryption provider keys and enter a Name (arbitrary) and key (20 characters long). Check the Primary box.
On the Internal MySQL page, enter an email address for notifications in the Email address field.
You can disable any errands you might not need on the Errands page but this is purely your preference based on how you’ll be using TAS.
On the Resource Config page, you can adjust the sizing of any of the jobs to better suit your needs. On the same page you will need to make a change to the Router and Control jobs. Expand the Router job and paste json content similar to the following in the Logical Load Balancer field.
{
"server_pools": [
{
"name": "tas-web-pool",
"port": "80"
},
{
"name": "tas-https-pool",
"port": "443"
}
]
}This json corresponds to the names of the load balancer server pools created in NSX-T (tas-web-pool and tas-https-pool).
Expand the Control job and paste json content similar to the following in the Logical Load Balancer field.
{
"server_pools": [
{
"name": "tas-ssh-pool",
"port": "2222"
}
]
}This json corresponds to the name of the load balancer server pools created in NSX-T (tas-ssh-pool).
At this point the TAS tile should be configured. You can navigate back to the Installation Dashboard.
Click on the Review Pending Changes button and then on the Apply Changes button to deploy the TAS and NSX-T tiles.
You can follow the progress in the Opsman UI or via the bosh task command. You will see a lot of output over a long period of time.
You’ll see several VMs get deployed in the vSphere Client under the TAS-Deployment resource pool
You can get a little bit of a better idea of what these VMs are via the bosh vms command.
bosh vms
Using environment '172.31.0.11' as client 'ops_manager'
Task 118. Done
Deployment 'cf-59b72eed9fa560870c04'
Instance Process State AZ IPs VM CID VM Type Active Stemcell
blobstore/60d0f8b6-4d83-4fda-904b-6764a2544515 running TAS-Deployment 172.31.1.12 vm-d32fee29-90e9-48ff-b181-e076b25a9934 medium true bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.251
compute/62181f19-5582-4900-9707-2984f0d940a3 running TAS-Deployment 172.31.1.14 vm-d70b7061-24af-4208-8a51-7deb3ecc0c71 xlarge.disk true bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.251
control/b645ed45-1610-432f-b7db-fd81fa1a9480 running TAS-Deployment 172.31.1.13 vm-f62c7d41-2aff-4117-8063-1bf6076ccb41 xlarge.disk true bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.251
database/058f8df4-ccfa-475d-9e55-3eed3144fe4e running TAS-Deployment 172.31.1.11 vm-b2c952d1-8200-4aa3-8cad-14b420d442f9 large.disk true bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.251
router/2a9890d6-a946-4680-9470-cda7c334b96c running TAS-Deployment 172.31.1.15 vm-4b954902-945a-413c-8be6-45916da52d72 micro.ram true bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.251
5 vms
Succeeded
In NSX-T you should see additional objects as well.
Two new T1 logical routers:
Two new logical switches:
Two new SNAT rules:
And when the process is done, you should be notified in the Opsman UI.
TAS UI
You can access the TAS UI in a web browser at login.<system sub-domain>.<domain name>. In my setup this is https://login.tas-system.corp.vmw.
To get the login credentials, head back to Opsman and click on the TAS tile. Navigate to Credentials and scroll down the UAA Job section. Click on the Link to Credential link next to the Admin Credentials item.
You should see a page similar to the following:
{"credential":{"type":"simple_credentials","value":{"identity":"admin","password":"B7AXCycBkAJaGmQ0iC1tqsf25mBRV-o9"}}}Use the noted password (7AXCycBkAJaGmQ0iC1tqsf25mBRV-o9 in this example) along with the admin user name to login to the TAS UI.
If you see a page similar to the following, click on the Apps Manager icon to continue to the TAS UI.
You might have noticed that the URL changes to apps.tas-system.corp.vmw…this is where those wildcard DNS records come in handy.
Before deploying an app we need to create an org and a space as we don’t want to deploy to the default (system) org. Click the Create Org button.
Enter an org name and then click the Create Org button.
On the page for your new org, click the Create New Space button.
Enter a name for the space and click the Create button.
You can see from this page that there are no apps deployed yet.
Deploy an application to TAS
One of the components downloaded earlier was the cf CLI. Be sure to extract this on a system in your environment that will have access to TAS.
On the system where cf is extracted run a command similar to the following to login to TAS (the username and password are the same as were used to login to the TAS UI):
cf login -a api.tas-system.corp.vmw -u admin -p B7AXCycBkAJaGmQ0iC1tqsf25mBRV-o9
API endpoint: api.tas-system.corp.vmw
Authenticating...
OK
Select an org:
1. system
2. tas-org
Org (enter to skip): 2
Targeted org tas-org.
Targeted space tas-space.
API endpoint: https://api.tas-system.corp.vmw
API version: 3.115.0
user: admin
org: tas-org
space: tas-space
You can see in the output that I selected the tas-org org but you can skip this and select it manually in a separate step.
cf target -o tas-org -s tas-space
API endpoint: https://api.tas-system.corp.vmw
API version: 3.115.0
user: admin
org: tas-org
space: tas-spaceIf you’re not building your own app, there are several samples you can deploy that are available at https://github.com/cloudfoundry-samples/. I’m going to deploy the test-app app and start off with cloning it locally.
git clone https://github.com/cloudfoundry-samples/test-app.git
Cloning into 'test-app'...
remote: Enumerating objects: 305, done.
remote: Total 305 (delta 0), reused 0 (delta 0), pack-reused 305
Receiving objects: 100% (305/305), 85.88 KiB | 514.00 KiB/s, done.
Resolving deltas: 100% (116/116), done.From here you will only need to cd into the newly created test-app directory and cf push the application.
cf push
Pushing app test-app to org tas-org / space tas-space as admin...
Applying manifest file /home/ubuntu/test-app/manifest.yml...
Updating with these attributes...
---
applications:
+ - name: test-app
+ instances: 1
memory: 256M
random-route: true
Manifest applied
Packaging files to upload...
Uploading files...
36.87 KiB / 36.87 KiB [====================================================================================] 100.00% 1s
Waiting for API to complete processing files...
Staging app and tracing logs...
Downloading binary_buildpack...
Downloading java_buildpack_offline...
Downloading go_buildpack...
Downloading nodejs_buildpack...
Downloading python_buildpack...
Downloaded nodejs_buildpack
Downloading r_buildpack...
Downloaded binary_buildpack (5.6M)
Downloading staticfile_buildpack...
Downloaded staticfile_buildpack
Downloading php_buildpack...
Downloading java_buildpack_offline failed
Downloading ruby_buildpack...
Cell 62181f19-5582-4900-9707-2984f0d940a3 destroying container for instance 005fa221-c39f-429e-ad9f-58d8569b4f83
Cell 62181f19-5582-4900-9707-2984f0d940a3 successfully destroyed container for instance 005fa221-c39f-429e-ad9f-58d8569b4f83
Downloaded ruby_buildpack
Downloading dotnet_core_buildpack...
Downloaded dotnet_core_buildpack (240.2M)
Downloading nginx_buildpack...
Downloaded php_buildpack (544.1M)
Downloading binary_buildpack...
Downloaded python_buildpack (571.7M)
Downloaded go_buildpack (582.6M)
Downloading staticfile_buildpack...
Downloaded r_buildpack (948M)
Downloading java_buildpack_offline...
Downloading ruby_buildpack...
Downloaded staticfile_buildpack
Downloading nginx_buildpack...
Downloaded binary_buildpack
Downloading nodejs_buildpack...
Downloaded nginx_buildpack (51.3M)
Downloading go_buildpack...
Downloaded ruby_buildpack
Downloading r_buildpack...
Downloaded nginx_buildpack
Downloading python_buildpack...
Downloaded nodejs_buildpack
Downloading php_buildpack...
Downloaded go_buildpack
Downloading dotnet_core_buildpack...
Downloaded dotnet_core_buildpack
Downloaded python_buildpack
Downloaded php_buildpack
Downloaded r_buildpack
Downloaded java_buildpack_offline (604.3M)
Cell 62181f19-5582-4900-9707-2984f0d940a3 creating container for instance 005fa221-c39f-429e-ad9f-58d8569b4f83
Cell 62181f19-5582-4900-9707-2984f0d940a3 successfully created container for instance 005fa221-c39f-429e-ad9f-58d8569b4f83
Downloading app package...
Downloaded app package (36.9K)
-----> Go Buildpack version 1.9.46
-----> Installing godep 80
Copy [/tmp/buildpacks/842ad584ef83b1381d372507d31fe31d/dependencies/52a892f00e80ca4fdcf27d9828c7aba1/godep-v80-linux-x64-cflinuxfs3-b60ac947.tgz]
-----> Installing glide 0.13.3
Copy [/tmp/buildpacks/842ad584ef83b1381d372507d31fe31d/dependencies/f5e4affa54f8cf8e22cf524de5165a6e/glide-v0.13.3-linux-x64-cflinuxfs3-ef07acb5.tgz]
-----> Installing dep 0.5.4
Copy [/tmp/buildpacks/842ad584ef83b1381d372507d31fe31d/dependencies/f1900fcb2de60a12ea6743ecf05e14d2/dep-v0.5.4-linux-x64-cflinuxfs3-79b3ab9e.tgz]
-----> Installing go 1.17.9
Copy [/tmp/buildpacks/842ad584ef83b1381d372507d31fe31d/dependencies/c6a24d72065a75e1ba17d24093721ddd/go_1.17.9_linux_x64_cflinuxfs3_a620909b.tgz]
**WARNING** Installing package '.' (default)
-----> Running: go install -tags cloudfoundry -buildmode pie .
Exit status 0
Uploading droplet, build artifacts cache...
Uploading droplet...
Uploading build artifacts cache...
Uploaded build artifacts cache (13.1M)
Uploaded droplet (4.4M)
Uploading complete
Cell 62181f19-5582-4900-9707-2984f0d940a3 stopping instance 005fa221-c39f-429e-ad9f-58d8569b4f83
Cell 62181f19-5582-4900-9707-2984f0d940a3 destroying container for instance 005fa221-c39f-429e-ad9f-58d8569b4f83
Waiting for app test-app to start...
Instances starting...
Instances starting...
Instances starting...
Instances starting...
name: test-app
requested state: started
routes: test-app-quick-impala-tj.tas-app.corp.vmw
last uploaded: Thu 30 Jun 12:27:56 PDT 2022
stack: cflinuxfs3
buildpacks:
name version detect output buildpack name
go_buildpack 1.9.46 go go
type: web
sidecars:
instances: 1/1
memory usage: 256M
start command: test-app
state since cpu memory disk details
#0 running 2022-06-30T19:28:13Z 0.0% 0 of 0 0 of 0
Near the end of the output of the cf push command is a URL (noted as routes: in the output). This is where your new app will be accessible (test-app-quick-impala-tj.tas-app.corp.vmw in this example).
Back in the TAS UI, you should see that there is an app deployed under tas-space.
If you drill down into tas-space, you’ll see more information about the deployed app and how to get to it.
You can click on the test-app link here to see even more details about the app, make changes and see event history.
You can click the View App link here to launch a new browser tab and see that that application is up and running.
One last note, you can see that the URL for the app starts out with the app name (test-app), followed by some arbitrary text (quick-impala), followed by the app subdomain (tas-app.corp.vmw).
I love this guide, will you do a 1.5 update and how to user windows workers for windows based containers?
Hello John. I am not currently planning on doing an update as TAS is not something I work with very regularly. I might have a post in the future covering the Wavefront / Tanzu Observability integration with TAS though.
Pingback: Installing TAS Integration for Tanzu Observability on vSphere – Little Stuff