VMware Blockchain 1.6.0.1 Install on vSphere 7U3

I am very new to the VMware Blockchain product but have deployed it a number of times since the 1.4 version. It is definitely not a “click next” type of installation but it’s not too difficult either. The process takes a little over an hour of time if you have a functional vSphere environment already.

To give a quick overview of the VMware Blockchain product, I’ll defer to the VMware Blockchain Documentation page as it is stated much more eloquently there than I could do it justice:

VMware Blockchain is an enterprise-grade blockchain platform that enables multi-party workflows. It uses advanced techniques such as byzantine fault-tolerant state machine replication, authenticated data structures, and integration with smart contract execution engines to enable customers to build and run decentralized multi-party applications.
https://docs.vmware.com/en/VMware-Blockchain/index.html#vmware-blockchain-overview-0

I would highly recommend reading up on the Blockchain product at the official documentation page. There is loads of information there as well as numerous architectural diagrams to help understand the flow of data.

VMware Blockchain is capable of running under two different deployment methodologies. You can run your Blockchain installation on AWS or on an on-prem vSphere installation. For the pupose of this pose, I chose to install VMware Blockchain on a vSphere 7.0 U3 installation.

Table of Contents

vSphere

There are a few logical constructs that need to be created in vSphere before moving on to the Blockchain installation.

Create resource pools and virtual machine folders named Blockchain, Blockchain-Replica and Blockchain-Client. The Blockchain Orchestrator appliance will be deployed to the Blockchain resource pool and folder, the Blockchain replica nodes will be deployed to the Blockchain-Replica resource pool and folder and the Blockchain client nodes will be deployed to the Blockchain-Client resource pool and folder.

Create a distributed portgroup named Blockchain. The Orchestrator appliance and all Blockchain nodes will use this network.

Create a role named Blockchain with the following privileges:

Resource Type	Privileges
Content Library	Read Storage
Datastore	Allocate Space
Network	all
Resource	Assign vApp to resource pool
vApp	all
Virtual Machine	all

Create an SSO user named blockchain. This user will be used by the Orchestrator appliance to provision the virtual machines that will become the replica and client nodes.

Assign the blockchain user the Blockchain role at the vCenter Server level and propagate to children.

Navigate to Administration > Access Control > Global Permissions. Assign the blockchain user the Read-only Global Permission and propagate to children.

Create a content library named blockchain-templates.

Configure the content library to sync to a URL similar to http://vmbco-01a.corp.vmw:8083/lib.json. This location should correspond to the FQDN or IP address that the Orchestrator appliance will have when it is deployed.

You could choose to create the content library after the Orchestrator appliance is deployed if desired.

Monitoring/Observability

I’m sending logs and metrics to a locally installed ELK installation as well as to Tanzu Observability.

ELK can be installed and configured largely per the documentation with one small caveat. There is no ability to provide private TLS certificate data when making a connection from the Blockchain nodes to the elasticsearch service. With this in mind, you will need to ensure that you are either using a certificate signed by a public CA for your elasticsearch service or you have disabled http SSL traffic for the elasticsearch service.

xpack.security.http.ssl.enabled: false

Making this change to the elasticsearch service will also mean that Kibana and Logstash will also need to be modified to use the proper protocol (http instead of https).

elasticsearch.hosts: ["http://elk.corp.vmw:9200"]

output {
  elasticsearch {
    hosts => ["http://elk.corp.vmw:9200"]
    user => 'logstash_writer'
    password => 'VMware1!'
  }
}

To send metrics to Tanzu Observability, you will need to know your organization’s URL (like https://vmware.wavefront.com) and have an API Token.

You can get your API token by clicking on the gear icon at the top right and then clicking on your user name.

Click on the API Access tab and make note of the token value.

In this example, the API token value is cf391585-ca50-4ca3-a681-8e539dddd59c.

Local Registry

The container images that are used to run VMware Blockchain are available at https://vmwaresaas.jfrog.io/vmwblockchain. If you are installing in an air-gapped environment or have poor bandwidth that may result in timeouts during installation, you can deploy a local registry and copy the needed images over.

I used the instructions I put together at How to install Harbor 2.0 on a Photon OS VM to stand up a local Harbor instance (running Harbor 2.5.3). I created a new project named vmwblockchain to better mimic the default installation process.

Once you have access to both https://vmwaresaas.jfrog.io/vmwblockchain and your local registry, you can issue commands similar to the following to copy the images over:

docker login harbor.corp.vmw -u blockchain -p VMware1!
docker login -u '<username>' -p '<password>' https://vmwaresaas.jfrog.io/vmwblockchain 

docker pull vmwaresaas.jfrog.io/vmwblockchain/agent:1.6.0.1.266
docker pull vmwaresaas.jfrog.io/vmwblockchain/concord-core:1.6.0.1.266
docker pull vmwaresaas.jfrog.io/vmwblockchain/operator:1.6.0.1.266
docker pull vmwaresaas.jfrog.io/vmwblockchain/daml-execution-engine:1.6.0.1.266
docker pull vmwaresaas.jfrog.io/vmwblockchain/daml-index-db:1.6.0.1.266
docker pull vmwaresaas.jfrog.io/vmwblockchain/daml-ledger-api:1.6.0.1.266
docker pull vmwaresaas.jfrog.io/vmwblockchain/fluentd:1.1
docker pull vmwaresaas.jfrog.io/vmwblockchain/wavefront-proxy:10.12
docker pull vmwaresaas.jfrog.io/vmwblockchain/telegraf:1.18.3
docker pull vmwaresaas.jfrog.io/vmwblockchain/jaeger-agent:1.22
docker pull vmwaresaas.jfrog.io/vmwblockchain/ethrpc:1.6.0.1.266
docker pull vmwaresaas.jfrog.io/vmwblockchain/cre:1.6.0.1.266
docker pull vmwaresaas.jfrog.io/vmwblockchain/clientservice:1.6.0.1.266

docker tag vmwaresaas.jfrog.io/vmwblockchain/agent:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/agent:1.6.0.1.266
docker tag vmwaresaas.jfrog.io/vmwblockchain/concord-core:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/concord-core:1.6.0.1.266
docker tag vmwaresaas.jfrog.io/vmwblockchain/operator:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/operator:1.6.0.1.266
docker tag vmwaresaas.jfrog.io/vmwblockchain/daml-execution-engine:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/daml-execution-engine:1.6.0.1.266
docker tag vmwaresaas.jfrog.io/vmwblockchain/daml-index-db:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/daml-index-db:1.6.0.1.266
docker tag vmwaresaas.jfrog.io/vmwblockchain/daml-ledger-api:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/daml-ledger-api:1.6.0.1.266
docker tag vmwaresaas.jfrog.io/vmwblockchain/fluentd:1.1 harbor.corp.vmw/vmwblockchain/fluentd:1.1
docker tag vmwaresaas.jfrog.io/vmwblockchain/wavefront-proxy:10.12 harbor.corp.vmw/vmwblockchain/wavefront-proxy:10.12
docker tag vmwaresaas.jfrog.io/vmwblockchain/telegraf:1.18.3 harbor.corp.vmw/vmwblockchain/telegraf:1.18.3
docker tag vmwaresaas.jfrog.io/vmwblockchain/jaeger-agent:1.22 harbor.corp.vmw/vmwblockchain/jaeger-agent:1.22
docker tag vmwaresaas.jfrog.io/vmwblockchain/ethrpc:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/ethrpc:1.6.0.1.266
docker tag vmwaresaas.jfrog.io/vmwblockchain/cre:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/cre:1.6.0.1.266
docker tag vmwaresaas.jfrog.io/vmwblockchain/clientservice:1.6.0.1.266 harbor.corp.vmw/vmwblockchain/clientservice:1.6.0.1.266

docker push harbor.corp.vmw/vmwblockchain/agent:1.6.0.1.266
docker push harbor.corp.vmw/vmwblockchain/concord-core:1.6.0.1.266
docker push harbor.corp.vmw/vmwblockchain/operator:1.6.0.1.266
docker push harbor.corp.vmw/vmwblockchain/daml-execution-engine:1.6.0.1.266
docker push harbor.corp.vmw/vmwblockchain/daml-index-db:1.6.0.1.266
docker push harbor.corp.vmw/vmwblockchain/daml-ledger-api:1.6.0.1.266
docker push harbor.corp.vmw/vmwblockchain/fluentd:1.1
docker push harbor.corp.vmw/vmwblockchain/wavefront-proxy:10.12
docker push harbor.corp.vmw/vmwblockchain/telegraf:1.18.3
docker push harbor.corp.vmw/vmwblockchain/jaeger-agent:1.22
docker push harbor.corp.vmw/vmwblockchain/ethrpc:1.6.0.1.266
docker push harbor.corp.vmw/vmwblockchain/cre:1.6.0.1.266
docker push harbor.corp.vmw/vmwblockchain/clientservice:1.6.0.1.266

The tag value of 1.6.0.1.266 is important to note as that is the specific tag needed for the Blockchain 1.6.0.1 version. You would need to update this appropriately if you were to install a different version.

You can see from the Harbor UI that the total space consumed by the VMware Blockchain images is 5.3GB.

Deploy the Orchestrator VM

The Orchestrator is a virtual appliance that you can download from https://customerconnect.vmware.com/downloads/details?downloadGroup=VMBC-1601-2&productId=1298&rPId=89996 and install into your vSphere environment. The installation process is fairly simple if you’ve ever deployed an OVA before.

The following are the parameters that I configured during deployment:

Item	Value
Name	vmbco-01a
Folder	Blockchain
Resource Pool	Blockchain
Storage	vol1
Network	Blockchain
Root Password	VMware1!VMware1! (a longer password is required)
Blockchain User Password	VMware1!VMware1! (a longer password is required)
Hostname	vmbco-01a.corp.vmw
IP Address	192.168.110.80
Prefix	24
Gateway	192.168.110.1
DNS	192.168.110.10

Once the deployment was complete, the vmbco-01a VM was powered on.

You can ssh to the Orchestrator VM as the blockchain user with the password that was configured during deployment.

Most commands will need to be run as the blockchain user but if you ever find that you need to do something as the root user, you can su to the root account (sudo is not configured).

The first time you ssh to the Orchestrator appliance, you will need to accept the EULA before you’ll be able to do anything else. You’ll need to accept the EULA again if you ever log in as the root user.

You can validate that the Orchestrator should be functional by checking the running containers:

docker ps
 
CONTAINER ID   IMAGE                                                                                                COMMAND                  CREATED          STATUS          PORTS                                                               NAMES
4bc6daceb6ae   athena-docker-local.artifactory.eng.vmware.com/persephone-provisioning:cl-nginx-blockchain-1.6.0.1   "nginx -g 'daemon of…"   14 minutes ago   Up 14 minutes   0.0.0.0:8083->80/tcp, :::8083->80/tcp                               orchestrator-runtime_blockchain-content-library-server_1
1623951a9a1e   athena-docker-local.artifactory.eng.vmware.com/vtpm-tools:1.6.0.1.266                                "java -Dspring.confi…"   14 minutes ago   Up 14 minutes   0.0.0.0:9797->9797/tcp, :::9797->9797/tcp                           orchestrator-runtime_vtpm-tools_1
4f767604fdba   athena-docker-local.artifactory.eng.vmware.com/persephone-provisioning:1.6.0.1.266                   "./wait-for-it.sh vt…"   14 minutes ago   Up 14 minutes   8000/tcp, 0.0.0.0:9002->9002/tcp, :::9002->9002/tcp                 orchestrator-runtime_persephone-provisioning_1
f225bca24919   athena-docker-local.artifactory.eng.vmware.com/persephone-configuration:1.6.0.1.266                  "java -Dspring.confi…"   14 minutes ago   Up 14 minutes   0.0.0.0:9003->9003/tcp, :::9003->9003/tcp, 0.0.0.0:8000->9023/tcp   orchestrator-runtime_config-service_1

If you see fewer containers than these, you will need to check that they were ever created (docker ps -a) and inspect the logs of any failed containers (docker logs <container id>).

In the /home/blockchain folder, there should only be one sub-folder present named orchestrator-runtime. This folder contains the scripts and docker-compose configuration files that will be called when the Blockchain nodes are deployed. We’ll need to create two more folders at the same level as orchestrator-runtime.

mkdir /home/blockchain/descriptors
mkdir /home/blockchain/output

Before moving into the bulk of the Blockchain configuration, I made one small change to the /home/blockchain/orchestrator-runtime/docker-compose-orchestrator.yml file. There is a deployment timeout of 30 minutes that is not configurable by default. In my nested environment, 30 minutes was sometimes not long enough for the deployment to finish. Making this change was necessary to help avoid this scenario.

version: '3.8'
services:
  castor:
    image: "${castor_repo}:${castor_tag}"
    volumes:
      - "${ORCHESTRATOR_DESCRIPTORS_DIR:?ORCHESTRATOR_DESCRIPTORS_DIR is required}:/descriptors:ro"
      - type: bind
        source: "${ORCHESTRATOR_OUTPUT_DIR:?ORCHESTRATOR_OUTPUT_DIR is required}"
        target: /output
        read_only: false
    environment:
      - "castor_deployment_type=${ORCHESTRATOR_DEPLOYMENT_TYPE:-PROVISION}"
      - "deployment_platform=${ORCHESTRATOR_DEPLOYMENT_PLATFORM:-VSPHERE}"
      - "castor_infrastructure_descriptor_location=/descriptors/${INFRA_DESC_FILENAME:?INFRA_DESC_FILENAME is required}"
      - "castor_deployment_descriptor_location=/descriptors/${DEPLOY_DESC_FILENAME:?DEPLOY_DESC_FILENAME is required}"
      - "castor_output_directory_location=/output"
    entrypoint: ['./wait-for-it.sh', 'persephone-provisioning:9002', '-t', '60', '--',
             "java", "-Dspring.config.location=/config/app/profiles/,./",
             "-jar", "castor.jar"]

The one line that needs to be added (to the environment: section) is:

- "castor.deployment.timeout.minutes=${CASTOR_DEPLOYMENT_TIMEOUT_MINUTES}"

With this change in place, you would be able to provide a CASTOR_DEPLOYMENT_TIMEOUT_MINUTES=### parameter to the docker-compose command that brings up the Blockchain.

Descriptor Files

Deployment Descriptor

The deployment descriptor file must contain the information necessary to configure the client and replica nodes.

Configuring the Deployment Descriptor Parameters on vSphere provides detailed descriptions of the parameters used in the deployment descriptor file.

{
  "replicas": [
    {
      "zoneName": "test-zone-replica",
        "providedIp": "192.168.100.31"
    },
    {
      "zoneName": "test-zone-replica",
        "providedIp": "192.168.100.32"
    },
    {
      "zoneName": "test-zone-replica",
        "providedIp": "192.168.100.33"
    },
    {
      "zoneName": "test-zone-replica",
        "providedIp": "192.168.100.34"
    }
  ],
  "replicaNodeSpec": {
    "cpuCount": 8,
    "memoryGb": 24,
    "diskSizeGb": 64
  },
  "fullCopyClients": [
    {
      "providedIp": "192.168.100.36",
      "zoneName": "test-zone-replica",
      "accessKey": "minio",
      "bucketName": "blockchain",
      "protocol": "HTTP",
      "secretKey": "minio123",
      "url": "192.168.110.60:9000"
    }
  ],
    "fullCopyClientNodeSpec": {
      "cpuCount": 8,
      "memoryGb": 24,
      "diskSizeGb": 64
    },
  "clients": [
    {
      "zoneName": "test-zone-client",
      "providedIp": "192.168.100.35",
      "groupName": "Group1"
    }
  ],
 "clientNodeSpec": {
    "cpuCount": 8,
    "memoryGb": 24,
    "diskSizeGb": 64
  },
"operatorSpecifications": {
  "operatorPublicKey": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEp8KvgIfJsiyG0ttxuGuHYu0k+E6y\nx3sJdgawvdEGlUpGKmZVO64LgWKKlkdUWyb+VOylaIwkpycyaxWZrwz5/w==\n-----END PUBLIC KEY-----\n"
},
  "blockchain": {
    "consortiumName": "EPG-blockchain-deployment",
    "blockchainType": "DAML"
  }
}

replicas:zoneName – this is arbitrary but should be the same for all replicas.
replicas:providedIP – this will be dependent on your network infrastructure, but they should all be in the same subnet/VLAN.
replicaNodeSpec – this will determine the size of the deployed replica nodes. There are four in this example but more could be configured. Ensure that you have enough compute and storage capacity to accommodate all of the nodes. My deployment is using about the smallest compute and storage profile that will work.

fullCopyClients – these are an optional set of parameters that will result in a “Full Copy Client” node being deployed. This node will keep a read-only copy of the entire blockchain on S3 storage. You can read more about this functionality at VMware Blockchain Data Archiving Implementation. An important thing to note is that you are not able to provide private TLS certificate data for the S3 storage so it either use a certificate signed by a public CA or you must configure the full copy client to not use TLS to communicate with the S3 storage (as I have done…"protocol": "HTTP").
fullCopyClientNodeSpec – this will determine the size of the deployed full copy client nodes.

clients:zoneName – this is arbitrary but should be the same for all replicas.
clients:providedIP – this will be dependent on your network infrastructure, but they should all be in the same subnet/VLAN.
clientNodeSpec – this will determine the size of the deployed client nodes. There is only one in this example but more could be configured. Ensure that you have enough compute and storage capacity to accommodate all of the nodes. My deployment is using about the smallest compute and storage profile that will work.

operatorSpecifications:operatorPublicKey – the operator container is a special container that you can run on one of the client nodes (it is not deployed by default) that is needed for adding/removing nodes (scaling) and performing maintenance on the Blockchain nodes. This parameter is a single-line version of the public key that is used to access the operator container.

You can create the private/public key for the operator container via the following steps (from a system with openssl installed):

openssl ecparam -name prime256v1 -genkey -noout -out /home/blockchain/blockchain-private-key.pem
openssl ec -in /home/blockchain/blockchain-private-key.pem -pubout -out /home/blockchain/blockchain-public-key.pem

You should see output similar to the following:

read EC key
writing EC key

The following command will get the public key onto one line for use in the deployment descriptor file:

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' /home/blockchain/blockchain-public-key.pem

You should see output similar to the following:

-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEp8KvgIfJsiyG0ttxuGuHYu0k+E6y\nx3sJdgawvdEGlUpGKmZVO64LgWKKlkdUWyb+VOylaIwkpycyaxWZrwz5/w==\n-----END PUBLIC KEY-----\n

blockchain:consortiumName – this is arbitrary
blockchain:blockchainType – this will determine what type of ledger is used on the replica nodes in the Blockchain. I’m using DAML (Digital Asset) for this example.

Infrastructure Descriptor

The infrastructure descriptor file must contain the vSphere information needed by the Orchestrator appliance to deploy VMs. Additionally, you can supply registry, logging, metrics and limited Blockchain configuration data in this file.

Configuring the Infrastructure Descriptor Parameters on vSphere provides detailed descriptions of the parameters used in the infrastructure descriptor file.

{
  "organization": {
    "damlSdk": "2.0.1",
    "dockerImage": "1.6.0.1.266",
    "enableBftClient": true,
    "generatePassword": false
  },
  "zones": [
    {
      "name": "test-zone-replica",
      "vCenter": {
        "url": "https://vcsa-01a.corp.vmw/",
        "userName": "blockchain@vsphere.local",
        "password": "VMware1!",
        "resourcePool": "Blockchain-Replica",
        "storage": "vol1",
        "folder": "Blockchain-Replica",
        "tlsCertificateData": "-----BEGIN CERTIFICATE-----\nMIIFczCCA1ugAwIBAgIQTYJITQ3SZ4BBS9UzXfJIuTANBgkqhkiG9w0BAQsFADBM\nMRMwEQYKCZImiZPyLGQBGRYDdm13MRQwEgYKCZImiZPyLGQBGRYEY29ycDEfMB0G\nA1UEAxMWY29udHJvbGNlbnRlci5jb3JwLnZtdzAeFw0yMjAzMjExOTE3MjhaFw0z\nNzAzMjExOTI3MjNaMEwxEzARBgoJkiaJk/IsZAEZFgN2bXcxFDASBgoJkiaJk/Is\nZAEZFgRjb3JwMR8wHQYDVQQDExZjb250cm9sY2VudGVyLmNvcnAudm13MIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA2OYKxckOjhgufWu1YEnatvJ1M127\ngwPbFNj11/dICXaPe+mjN1Hce0PiS2QaaeAe8kH+mOKRa2JjaGdXr6rOiB80KZOR\nuw0GzSJyL5w7ewR+NJf31YO62BD/mt3sHeMnCXmSBxOQvb0nGkhTr1y+rDpvxJ87\nzNczgfN54to6S379wjOsC4bkHLnMJ5EtJG78pPqX1+1wcVOURNJ6y9BcejLnoy/y\nCFpXKOVxKHzy2nnsitAuBb+hD+Jxw8/jFQUhxH0VlgyfXCQdegasSA9RHtZtfpVs\nhshisjkSlvQmbsEknBZrAfBVIYidwt3w050jVhiUs5Ql6vDotY6Gqtzzgq0obv6P\n7E9NPej3BzhPSIUyqnpf57UWI4zUiRJvbSu/J2MCBKHwYfzke1cnvLA7viDEdB9+\n/Htk9aG9/1B6ddDfafrcSOWtkTfHWYLv21o3Uwoh9W5OpK9JikZu/PqnpZkUi+2C\nL+WCww/BS1yhQwVif6PqUMeSLz3jtq3w6R/ruUMlO+0E5//bskDT6QGxBgcvMF9n\nDl+u0uqHKOdiUvOXBtF139HKUrZsq0m3WPoel2/p+cVVJYsyJG/rRpeh1g/X0cB3\n9EuTjX6vnrT+IS8ZfAaoHzpmgh1vGu2r2xgPq2E8x4ji9FGV8YTjAs60Nw7YxKUW\nWgj+YNpxP2SxFqUCAwEAAaNRME8wCwYDVR0PBAQDAgGGMA8GA1UdEwEB/wQFMAMB\nAf8wHQYDVR0OBBYEFMaD85ZQCWnnZTKtlo2FgeBfJBnHMBAGCSsGAQQBgjcVAQQD\nAgEAMA0GCSqGSIb3DQEBCwUAA4ICAQAutXwOtsmYcbj/bs3Mydx0Di9m+6UVTEZd\nORRrTus/BL/TNryO7zo2beczGPK26MwqhmUZiaF61jRb36kxmFPVx2uV2np4LbQj\n5MrxtPzf2XXy4b7ADqQpLgu4rR3mZiXGmzUoV17hmAhyfSU1qm4FssXGK2ypWsQs\nBwsKX4DsIijJJZbXwKFaauq0LtnkgeGWdoEFFWAH0yJWPbz9h+ovlCxq0DBiG00l\nbrnY90sqpoiWTxMKNCXDDhNjvtxO3kQIDQVvbNMCEbmYG+RrWQHtvufw97RK/cTL\n9dKFSblIIizMINVwM/gqtlVVvWP1EFaUy0xG5bvOO+SCe+TlA7rz4/RORqqE5Ugg\n7F8fWz+o6BM/qf/Kwh+WN42dyR1rOsFqEVNamZLjrAzgwjQ/nquRRMl2cK6yg6Fq\nd0O42wwYPpLUEFv4xe4a3kpRvvhshNkzR4IacbmaUlnzmlewoFXVueEblviBHJoV\n1OUC6qfLkCjfCEv470Kr5vDe5Y/l/7j8EYj7a/wa2++kq+7xd+bj/DDed85fm3Yk\ndhfp7bGXKm4KbPLzkSpiYWbE+EbArLtIk62exjcJvJPdoxMTxgbdelzl/snPLrdg\nw0oGuTTBfxSMKs767N3G1q5tz0mwFpIqIQtXUSmaJ+9p7IkpWcThLnyYYo1IpWm/\nZHtjzZMQVA==\n-----END CERTIFICATE-----\n"
      },
      "network": {
        "name": "Blockchain",
        "gateway": "192.168.100.1",
        "subnet": 24,
        "nameServers": [
          "192.168.110.10"
        ]
      },
      "containerRegistry": {
        "url": "https://harbor.corp.vmw/vmwblockchain",
        "userName": "blockchain",
        "password": "VMware1!",
        "tlsCertificateData": "-----BEGIN CERTIFICATE-----\nMIIFczCCA1ugAwIBAgIQTYJITQ3SZ4BBS9UzXfJIuTANBgkqhkiG9w0BAQsFADBM\nMRMwEQYKCZImiZPyLGQBGRYDdm13MRQwEgYKCZImiZPyLGQBGRYEY29ycDEfMB0G\nA1UEAxMWY29udHJvbGNlbnRlci5jb3JwLnZtdzAeFw0yMjAzMjExOTE3MjhaFw0z\nNzAzMjExOTI3MjNaMEwxEzARBgoJkiaJk/IsZAEZFgN2bXcxFDASBgoJkiaJk/Is\nZAEZFgRjb3JwMR8wHQYDVQQDExZjb250cm9sY2VudGVyLmNvcnAudm13MIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA2OYKxckOjhgufWu1YEnatvJ1M127\ngwPbFNj11/dICXaPe+mjN1Hce0PiS2QaaeAe8kH+mOKRa2JjaGdXr6rOiB80KZOR\nuw0GzSJyL5w7ewR+NJf31YO62BD/mt3sHeMnCXmSBxOQvb0nGkhTr1y+rDpvxJ87\nzNczgfN54to6S379wjOsC4bkHLnMJ5EtJG78pPqX1+1wcVOURNJ6y9BcejLnoy/y\nCFpXKOVxKHzy2nnsitAuBb+hD+Jxw8/jFQUhxH0VlgyfXCQdegasSA9RHtZtfpVs\nhshisjkSlvQmbsEknBZrAfBVIYidwt3w050jVhiUs5Ql6vDotY6Gqtzzgq0obv6P\n7E9NPej3BzhPSIUyqnpf57UWI4zUiRJvbSu/J2MCBKHwYfzke1cnvLA7viDEdB9+\n/Htk9aG9/1B6ddDfafrcSOWtkTfHWYLv21o3Uwoh9W5OpK9JikZu/PqnpZkUi+2C\nL+WCww/BS1yhQwVif6PqUMeSLz3jtq3w6R/ruUMlO+0E5//bskDT6QGxBgcvMF9n\nDl+u0uqHKOdiUvOXBtF139HKUrZsq0m3WPoel2/p+cVVJYsyJG/rRpeh1g/X0cB3\n9EuTjX6vnrT+IS8ZfAaoHzpmgh1vGu2r2xgPq2E8x4ji9FGV8YTjAs60Nw7YxKUW\nWgj+YNpxP2SxFqUCAwEAAaNRME8wCwYDVR0PBAQDAgGGMA8GA1UdEwEB/wQFMAMB\nAf8wHQYDVR0OBBYEFMaD85ZQCWnnZTKtlo2FgeBfJBnHMBAGCSsGAQQBgjcVAQQD\nAgEAMA0GCSqGSIb3DQEBCwUAA4ICAQAutXwOtsmYcbj/bs3Mydx0Di9m+6UVTEZd\nORRrTus/BL/TNryO7zo2beczGPK26MwqhmUZiaF61jRb36kxmFPVx2uV2np4LbQj\n5MrxtPzf2XXy4b7ADqQpLgu4rR3mZiXGmzUoV17hmAhyfSU1qm4FssXGK2ypWsQs\nBwsKX4DsIijJJZbXwKFaauq0LtnkgeGWdoEFFWAH0yJWPbz9h+ovlCxq0DBiG00l\nbrnY90sqpoiWTxMKNCXDDhNjvtxO3kQIDQVvbNMCEbmYG+RrWQHtvufw97RK/cTL\n9dKFSblIIizMINVwM/gqtlVVvWP1EFaUy0xG5bvOO+SCe+TlA7rz4/RORqqE5Ugg\n7F8fWz+o6BM/qf/Kwh+WN42dyR1rOsFqEVNamZLjrAzgwjQ/nquRRMl2cK6yg6Fq\nd0O42wwYPpLUEFv4xe4a3kpRvvhshNkzR4IacbmaUlnzmlewoFXVueEblviBHJoV\n1OUC6qfLkCjfCEv470Kr5vDe5Y/l/7j8EYj7a/wa2++kq+7xd+bj/DDed85fm3Yk\ndhfp7bGXKm4KbPLzkSpiYWbE+EbArLtIk62exjcJvJPdoxMTxgbdelzl/snPLrdg\nw0oGuTTBfxSMKs767N3G1q5tz0mwFpIqIQtXUSmaJ+9p7IkpWcThLnyYYo1IpWm/\nZHtjzZMQVA==\n-----END CERTIFICATE-----\n"
      },
      "elasticSearch": {
        "url": "http://elk.corp.vmw:9200",
        "userName": "elastic",
        "password": "VMware1!"
      },
      "wavefront": {
        "url": "https://vmware.wavefront.com",
        "token": "cf391585-ca50-4ca3-a681-8e539dddd59c"
      },
      "logManagement": [
        {
          "type": "HTTP",
          "address": "https://elk.corp.vmw:8443",
          "tlsCertificateData": "-----BEGIN CERTIFICATE-----\nMIIFczCCA1ugAwIBAgIQTYJITQ3SZ4BBS9UzXfJIuTANBgkqhkiG9w0BAQsFADBM\nMRMwEQYKCZImiZPyLGQBGRYDdm13MRQwEgYKCZImiZPyLGQBGRYEY29ycDEfMB0G\nA1UEAxMWY29udHJvbGNlbnRlci5jb3JwLnZtdzAeFw0yMjAzMjExOTE3MjhaFw0z\nNzAzMjExOTI3MjNaMEwxEzARBgoJkiaJk/IsZAEZFgN2bXcxFDASBgoJkiaJk/Is\nZAEZFgRjb3JwMR8wHQYDVQQDExZjb250cm9sY2VudGVyLmNvcnAudm13MIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA2OYKxckOjhgufWu1YEnatvJ1M127\ngwPbFNj11/dICXaPe+mjN1Hce0PiS2QaaeAe8kH+mOKRa2JjaGdXr6rOiB80KZOR\nuw0GzSJyL5w7ewR+NJf31YO62BD/mt3sHeMnCXmSBxOQvb0nGkhTr1y+rDpvxJ87\nzNczgfN54to6S379wjOsC4bkHLnMJ5EtJG78pPqX1+1wcVOURNJ6y9BcejLnoy/y\nCFpXKOVxKHzy2nnsitAuBb+hD+Jxw8/jFQUhxH0VlgyfXCQdegasSA9RHtZtfpVs\nhshisjkSlvQmbsEknBZrAfBVIYidwt3w050jVhiUs5Ql6vDotY6Gqtzzgq0obv6P\n7E9NPej3BzhPSIUyqnpf57UWI4zUiRJvbSu/J2MCBKHwYfzke1cnvLA7viDEdB9+\n/Htk9aG9/1B6ddDfafrcSOWtkTfHWYLv21o3Uwoh9W5OpK9JikZu/PqnpZkUi+2C\nL+WCww/BS1yhQwVif6PqUMeSLz3jtq3w6R/ruUMlO+0E5//bskDT6QGxBgcvMF9n\nDl+u0uqHKOdiUvOXBtF139HKUrZsq0m3WPoel2/p+cVVJYsyJG/rRpeh1g/X0cB3\n9EuTjX6vnrT+IS8ZfAaoHzpmgh1vGu2r2xgPq2E8x4ji9FGV8YTjAs60Nw7YxKUW\nWgj+YNpxP2SxFqUCAwEAAaNRME8wCwYDVR0PBAQDAgGGMA8GA1UdEwEB/wQFMAMB\nAf8wHQYDVR0OBBYEFMaD85ZQCWnnZTKtlo2FgeBfJBnHMBAGCSsGAQQBgjcVAQQD\nAgEAMA0GCSqGSIb3DQEBCwUAA4ICAQAutXwOtsmYcbj/bs3Mydx0Di9m+6UVTEZd\nORRrTus/BL/TNryO7zo2beczGPK26MwqhmUZiaF61jRb36kxmFPVx2uV2np4LbQj\n5MrxtPzf2XXy4b7ADqQpLgu4rR3mZiXGmzUoV17hmAhyfSU1qm4FssXGK2ypWsQs\nBwsKX4DsIijJJZbXwKFaauq0LtnkgeGWdoEFFWAH0yJWPbz9h+ovlCxq0DBiG00l\nbrnY90sqpoiWTxMKNCXDDhNjvtxO3kQIDQVvbNMCEbmYG+RrWQHtvufw97RK/cTL\n9dKFSblIIizMINVwM/gqtlVVvWP1EFaUy0xG5bvOO+SCe+TlA7rz4/RORqqE5Ugg\n7F8fWz+o6BM/qf/Kwh+WN42dyR1rOsFqEVNamZLjrAzgwjQ/nquRRMl2cK6yg6Fq\nd0O42wwYPpLUEFv4xe4a3kpRvvhshNkzR4IacbmaUlnzmlewoFXVueEblviBHJoV\n1OUC6qfLkCjfCEv470Kr5vDe5Y/l/7j8EYj7a/wa2++kq+7xd+bj/DDed85fm3Yk\ndhfp7bGXKm4KbPLzkSpiYWbE+EbArLtIk62exjcJvJPdoxMTxgbdelzl/snPLrdg\nw0oGuTTBfxSMKs767N3G1q5tz0mwFpIqIQtXUSmaJ+9p7IkpWcThLnyYYo1IpWm/\nZHtjzZMQVA==\n-----END CERTIFICATE-----\n"
        }
      ]
    },
    {
      "name": "test-zone-client",
      "vCenter": {
        "url": "https://vcsa-01a.corp.vmw/",
        "userName": "blockchain@vsphere.local",
        "password": "VMware1!",
        "resourcePool": "Blockchain-Client",
        "storage": "vol1",
        "folder": "Blockchain-Client",
        "tlsCertificateData": "-----BEGIN CERTIFICATE-----\nMIIFczCCA1ugAwIBAgIQTYJITQ3SZ4BBS9UzXfJIuTANBgkqhkiG9w0BAQsFADBM\nMRMwEQYKCZImiZPyLGQBGRYDdm13MRQwEgYKCZImiZPyLGQBGRYEY29ycDEfMB0G\nA1UEAxMWY29udHJvbGNlbnRlci5jb3JwLnZtdzAeFw0yMjAzMjExOTE3MjhaFw0z\nNzAzMjExOTI3MjNaMEwxEzARBgoJkiaJk/IsZAEZFgN2bXcxFDASBgoJkiaJk/Is\nZAEZFgRjb3JwMR8wHQYDVQQDExZjb250cm9sY2VudGVyLmNvcnAudm13MIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA2OYKxckOjhgufWu1YEnatvJ1M127\ngwPbFNj11/dICXaPe+mjN1Hce0PiS2QaaeAe8kH+mOKRa2JjaGdXr6rOiB80KZOR\nuw0GzSJyL5w7ewR+NJf31YO62BD/mt3sHeMnCXmSBxOQvb0nGkhTr1y+rDpvxJ87\nzNczgfN54to6S379wjOsC4bkHLnMJ5EtJG78pPqX1+1wcVOURNJ6y9BcejLnoy/y\nCFpXKOVxKHzy2nnsitAuBb+hD+Jxw8/jFQUhxH0VlgyfXCQdegasSA9RHtZtfpVs\nhshisjkSlvQmbsEknBZrAfBVIYidwt3w050jVhiUs5Ql6vDotY6Gqtzzgq0obv6P\n7E9NPej3BzhPSIUyqnpf57UWI4zUiRJvbSu/J2MCBKHwYfzke1cnvLA7viDEdB9+\n/Htk9aG9/1B6ddDfafrcSOWtkTfHWYLv21o3Uwoh9W5OpK9JikZu/PqnpZkUi+2C\nL+WCww/BS1yhQwVif6PqUMeSLz3jtq3w6R/ruUMlO+0E5//bskDT6QGxBgcvMF9n\nDl+u0uqHKOdiUvOXBtF139HKUrZsq0m3WPoel2/p+cVVJYsyJG/rRpeh1g/X0cB3\n9EuTjX6vnrT+IS8ZfAaoHzpmgh1vGu2r2xgPq2E8x4ji9FGV8YTjAs60Nw7YxKUW\nWgj+YNpxP2SxFqUCAwEAAaNRME8wCwYDVR0PBAQDAgGGMA8GA1UdEwEB/wQFMAMB\nAf8wHQYDVR0OBBYEFMaD85ZQCWnnZTKtlo2FgeBfJBnHMBAGCSsGAQQBgjcVAQQD\nAgEAMA0GCSqGSIb3DQEBCwUAA4ICAQAutXwOtsmYcbj/bs3Mydx0Di9m+6UVTEZd\nORRrTus/BL/TNryO7zo2beczGPK26MwqhmUZiaF61jRb36kxmFPVx2uV2np4LbQj\n5MrxtPzf2XXy4b7ADqQpLgu4rR3mZiXGmzUoV17hmAhyfSU1qm4FssXGK2ypWsQs\nBwsKX4DsIijJJZbXwKFaauq0LtnkgeGWdoEFFWAH0yJWPbz9h+ovlCxq0DBiG00l\nbrnY90sqpoiWTxMKNCXDDhNjvtxO3kQIDQVvbNMCEbmYG+RrWQHtvufw97RK/cTL\n9dKFSblIIizMINVwM/gqtlVVvWP1EFaUy0xG5bvOO+SCe+TlA7rz4/RORqqE5Ugg\n7F8fWz+o6BM/qf/Kwh+WN42dyR1rOsFqEVNamZLjrAzgwjQ/nquRRMl2cK6yg6Fq\nd0O42wwYPpLUEFv4xe4a3kpRvvhshNkzR4IacbmaUlnzmlewoFXVueEblviBHJoV\n1OUC6qfLkCjfCEv470Kr5vDe5Y/l/7j8EYj7a/wa2++kq+7xd+bj/DDed85fm3Yk\ndhfp7bGXKm4KbPLzkSpiYWbE+EbArLtIk62exjcJvJPdoxMTxgbdelzl/snPLrdg\nw0oGuTTBfxSMKs767N3G1q5tz0mwFpIqIQtXUSmaJ+9p7IkpWcThLnyYYo1IpWm/\nZHtjzZMQVA==\n-----END CERTIFICATE-----\n"
      },
      "network": {
        "name": "Blockchain",
        "gateway": "192.168.100.1",
        "subnet": 24,
        "nameServers": [
          "192.168.110.10"
        ]
      },
      "containerRegistry": {
        "url": "https://harbor.corp.vmw/vmwblockchain",
        "userName": "blockchain",
        "password": "VMware1!",
        "tlsCertificateData": "-----BEGIN CERTIFICATE-----\nMIIFczCCA1ugAwIBAgIQTYJITQ3SZ4BBS9UzXfJIuTANBgkqhkiG9w0BAQsFADBM\nMRMwEQYKCZImiZPyLGQBGRYDdm13MRQwEgYKCZImiZPyLGQBGRYEY29ycDEfMB0G\nA1UEAxMWY29udHJvbGNlbnRlci5jb3JwLnZtdzAeFw0yMjAzMjExOTE3MjhaFw0z\nNzAzMjExOTI3MjNaMEwxEzARBgoJkiaJk/IsZAEZFgN2bXcxFDASBgoJkiaJk/Is\nZAEZFgRjb3JwMR8wHQYDVQQDExZjb250cm9sY2VudGVyLmNvcnAudm13MIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA2OYKxckOjhgufWu1YEnatvJ1M127\ngwPbFNj11/dICXaPe+mjN1Hce0PiS2QaaeAe8kH+mOKRa2JjaGdXr6rOiB80KZOR\nuw0GzSJyL5w7ewR+NJf31YO62BD/mt3sHeMnCXmSBxOQvb0nGkhTr1y+rDpvxJ87\nzNczgfN54to6S379wjOsC4bkHLnMJ5EtJG78pPqX1+1wcVOURNJ6y9BcejLnoy/y\nCFpXKOVxKHzy2nnsitAuBb+hD+Jxw8/jFQUhxH0VlgyfXCQdegasSA9RHtZtfpVs\nhshisjkSlvQmbsEknBZrAfBVIYidwt3w050jVhiUs5Ql6vDotY6Gqtzzgq0obv6P\n7E9NPej3BzhPSIUyqnpf57UWI4zUiRJvbSu/J2MCBKHwYfzke1cnvLA7viDEdB9+\n/Htk9aG9/1B6ddDfafrcSOWtkTfHWYLv21o3Uwoh9W5OpK9JikZu/PqnpZkUi+2C\nL+WCww/BS1yhQwVif6PqUMeSLz3jtq3w6R/ruUMlO+0E5//bskDT6QGxBgcvMF9n\nDl+u0uqHKOdiUvOXBtF139HKUrZsq0m3WPoel2/p+cVVJYsyJG/rRpeh1g/X0cB3\n9EuTjX6vnrT+IS8ZfAaoHzpmgh1vGu2r2xgPq2E8x4ji9FGV8YTjAs60Nw7YxKUW\nWgj+YNpxP2SxFqUCAwEAAaNRME8wCwYDVR0PBAQDAgGGMA8GA1UdEwEB/wQFMAMB\nAf8wHQYDVR0OBBYEFMaD85ZQCWnnZTKtlo2FgeBfJBnHMBAGCSsGAQQBgjcVAQQD\nAgEAMA0GCSqGSIb3DQEBCwUAA4ICAQAutXwOtsmYcbj/bs3Mydx0Di9m+6UVTEZd\nORRrTus/BL/TNryO7zo2beczGPK26MwqhmUZiaF61jRb36kxmFPVx2uV2np4LbQj\n5MrxtPzf2XXy4b7ADqQpLgu4rR3mZiXGmzUoV17hmAhyfSU1qm4FssXGK2ypWsQs\nBwsKX4DsIijJJZbXwKFaauq0LtnkgeGWdoEFFWAH0yJWPbz9h+ovlCxq0DBiG00l\nbrnY90sqpoiWTxMKNCXDDhNjvtxO3kQIDQVvbNMCEbmYG+RrWQHtvufw97RK/cTL\n9dKFSblIIizMINVwM/gqtlVVvWP1EFaUy0xG5bvOO+SCe+TlA7rz4/RORqqE5Ugg\n7F8fWz+o6BM/qf/Kwh+WN42dyR1rOsFqEVNamZLjrAzgwjQ/nquRRMl2cK6yg6Fq\nd0O42wwYPpLUEFv4xe4a3kpRvvhshNkzR4IacbmaUlnzmlewoFXVueEblviBHJoV\n1OUC6qfLkCjfCEv470Kr5vDe5Y/l/7j8EYj7a/wa2++kq+7xd+bj/DDed85fm3Yk\ndhfp7bGXKm4KbPLzkSpiYWbE+EbArLtIk62exjcJvJPdoxMTxgbdelzl/snPLrdg\nw0oGuTTBfxSMKs767N3G1q5tz0mwFpIqIQtXUSmaJ+9p7IkpWcThLnyYYo1IpWm/\nZHtjzZMQVA==\n-----END CERTIFICATE-----\n"
      },
      "elasticSearch": {
        "url": "http://elk.corp.vmw:9200",
        "userName": "elastic",
        "password": "VMware1!"
      },
      "wavefront": {
        "url": "https://try.wavefront.com",
        "token": "cf391585-ca50-4ca3-a681-8e539dddd59c"
      },
      "logManagement": [
        {
          "type": "HTTP",
          "address": "https://elk.corp.vmw:8443",
          "tlsCertificateData": "-----BEGIN CERTIFICATE-----\nMIIFczCCA1ugAwIBAgIQTYJITQ3SZ4BBS9UzXfJIuTANBgkqhkiG9w0BAQsFADBM\nMRMwEQYKCZImiZPyLGQBGRYDdm13MRQwEgYKCZImiZPyLGQBGRYEY29ycDEfMB0G\nA1UEAxMWY29udHJvbGNlbnRlci5jb3JwLnZtdzAeFw0yMjAzMjExOTE3MjhaFw0z\nNzAzMjExOTI3MjNaMEwxEzARBgoJkiaJk/IsZAEZFgN2bXcxFDASBgoJkiaJk/Is\nZAEZFgRjb3JwMR8wHQYDVQQDExZjb250cm9sY2VudGVyLmNvcnAudm13MIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA2OYKxckOjhgufWu1YEnatvJ1M127\ngwPbFNj11/dICXaPe+mjN1Hce0PiS2QaaeAe8kH+mOKRa2JjaGdXr6rOiB80KZOR\nuw0GzSJyL5w7ewR+NJf31YO62BD/mt3sHeMnCXmSBxOQvb0nGkhTr1y+rDpvxJ87\nzNczgfN54to6S379wjOsC4bkHLnMJ5EtJG78pPqX1+1wcVOURNJ6y9BcejLnoy/y\nCFpXKOVxKHzy2nnsitAuBb+hD+Jxw8/jFQUhxH0VlgyfXCQdegasSA9RHtZtfpVs\nhshisjkSlvQmbsEknBZrAfBVIYidwt3w050jVhiUs5Ql6vDotY6Gqtzzgq0obv6P\n7E9NPej3BzhPSIUyqnpf57UWI4zUiRJvbSu/J2MCBKHwYfzke1cnvLA7viDEdB9+\n/Htk9aG9/1B6ddDfafrcSOWtkTfHWYLv21o3Uwoh9W5OpK9JikZu/PqnpZkUi+2C\nL+WCww/BS1yhQwVif6PqUMeSLz3jtq3w6R/ruUMlO+0E5//bskDT6QGxBgcvMF9n\nDl+u0uqHKOdiUvOXBtF139HKUrZsq0m3WPoel2/p+cVVJYsyJG/rRpeh1g/X0cB3\n9EuTjX6vnrT+IS8ZfAaoHzpmgh1vGu2r2xgPq2E8x4ji9FGV8YTjAs60Nw7YxKUW\nWgj+YNpxP2SxFqUCAwEAAaNRME8wCwYDVR0PBAQDAgGGMA8GA1UdEwEB/wQFMAMB\nAf8wHQYDVR0OBBYEFMaD85ZQCWnnZTKtlo2FgeBfJBnHMBAGCSsGAQQBgjcVAQQD\nAgEAMA0GCSqGSIb3DQEBCwUAA4ICAQAutXwOtsmYcbj/bs3Mydx0Di9m+6UVTEZd\nORRrTus/BL/TNryO7zo2beczGPK26MwqhmUZiaF61jRb36kxmFPVx2uV2np4LbQj\n5MrxtPzf2XXy4b7ADqQpLgu4rR3mZiXGmzUoV17hmAhyfSU1qm4FssXGK2ypWsQs\nBwsKX4DsIijJJZbXwKFaauq0LtnkgeGWdoEFFWAH0yJWPbz9h+ovlCxq0DBiG00l\nbrnY90sqpoiWTxMKNCXDDhNjvtxO3kQIDQVvbNMCEbmYG+RrWQHtvufw97RK/cTL\n9dKFSblIIizMINVwM/gqtlVVvWP1EFaUy0xG5bvOO+SCe+TlA7rz4/RORqqE5Ugg\n7F8fWz+o6BM/qf/Kwh+WN42dyR1rOsFqEVNamZLjrAzgwjQ/nquRRMl2cK6yg6Fq\nd0O42wwYPpLUEFv4xe4a3kpRvvhshNkzR4IacbmaUlnzmlewoFXVueEblviBHJoV\n1OUC6qfLkCjfCEv470Kr5vDe5Y/l/7j8EYj7a/wa2++kq+7xd+bj/DDed85fm3Yk\ndhfp7bGXKm4KbPLzkSpiYWbE+EbArLtIk62exjcJvJPdoxMTxgbdelzl/snPLrdg\nw0oGuTTBfxSMKs767N3G1q5tz0mwFpIqIQtXUSmaJ+9p7IkpWcThLnyYYo1IpWm/\nZHtjzZMQVA==\n-----END CERTIFICATE-----\n"
        }
      ]
    }
  ]
}

organization:damlSdk – this needs to be set to 2.0.1 for the 1.6.0.1 version of VMware Blockchain but would be different for other versions.
organization:dockerImage – this is very specific to the VMware Blockchain version and will determine the version of all container images used.
organization:generatePassword – setting this to false allows the default password to be used for the vmbc account on replica and client nodes. Setting it to true will result in a random password being generated.

zones:*:name – this is arbitrary but you will need at least one zone for replica nodes and one zone for client nodes. These zone names are references in the deployment descriptor file.
zones:*:vCenter:* – these parameters dictate the vCenter Server to be used for deploy8ing Blockchain nodes and the logical constructs to be used for VM placement. If your vCenter Server is using a certificate signed by a public CA, the tlsCertificateData parameter is not necessary. In my example, all components are using a certificate signed by my internal CA so it is a required field. This certificate data must be on one line so you can use the same command noted earlier (awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' <filename>) to get an internal CA certificate on to one line.
zones:*:network:* – these parameters are used to define the logical network (and it’s IP configuration) to be used by the replica and client nodes
zones:*:containerRegistry:* – in my example, I am referencing my locally installed Harbor registry where I have copied all of the needed images. The tlsCertificateData field is needed as my Harbor installation is using a certificate signed by my internal CA. If you are using the publicly available registry for deployment, the url value would be https://vmwaresaas.jfrog.io/vmwblockchain and you would not need the tlsCertificateData field.
zones:*:elasticSearch:* – these fields are only necessary if you are using ELK. As noted earlier, there is no ability to provide a tlsCertifcateData field for this set of parameters so your elasticsearch service must not have http ssl enabled or user a certificate signed by a public CA.
zones:*:wavefront:* – these fields are optional and dictate the Tanzu Observability URL and API token to be used when sending metrics to Tanzu Observability.
zones:*:logManagement:* – these are also optional fields and will dictate where logs from the Blockchain nodes are sent. In my example, I am sending logs to the logstash component of my ELK installation. Logging data could also be sent to vRealize Log Insight (cloud or on-prem), or other log ingestion services that use an HTTP protocol.

Deploy the Blockchain Nodes

From the Orchestrator VM, a docker-compose command similar to the following is used to deploy the replica and client nodes:

cd /home/blockchain/orchestrator-runtime

ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors INFRA_DESC_FILENAME=infrastructure_descriptor.json DEPLOY_DESC_FILENAME=deployment_descriptor.json ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output  CASTOR_DEPLOYMENT_TIMEOUT_MINUTES=120 docker-compose -f docker-compose-orchestrator.yml up

ORCHESTRATOR_DESCRIPTORS_DIR – tells docker-compose where your descriptor files are saved
INFRA_DESC_FILENAME – tells docker-compose the name of the infrastructure descriptor file
DEPLOY_DESC_FILENAME – tells docker-compose the name of the deployment descriptor file
ORCHESTRATOR_OUTPUT_DIR – tells docker-compose where to save the log and configuration files from the deployment
CASTOR_DEPLOYMENT_TIMEOUT_MINUTES – tells docker-compose to increase the default deployment timeout from 30 minutes to 120 minutes (this is only needed if you made the change to the /home/blockchain/orchestrator-runtime/docker-compose-orchestrator.yml noted earlier

Once the command is initiated, you should see output similar to the following:

Creating orchestrator-runtime_castor_1 ... done
Attaching to orchestrator-runtime_castor_1
castor_1  | wait-for-it.sh: waiting 60 seconds for persephone-provisioning:9002
castor_1  | wait-for-it.sh: persephone-provisioning:9002 is available after 0 seconds
castor_1  | **************************************************
castor_1  | VMware Blockchain Orchestrator(c) Vmware Inc. 2020
castor_1  | **************************************************
castor_1  |
castor_1  | [INFO ] [2022-08-02 16:59:52.879] [thread=main] [OpID=] [user=] [org=] [function=CastorApplication] [message=Starting CastorApplication using Java 11.0.11 on a6cc6cff585f with PID 1 (/castor/castor.jar started by blockchain in /castor)]
castor_1  | [INFO ] [2022-08-14 16:59:52.729] [thread=background-preinit] [OpID=] [user=] [org=] [function=Version] [message=HV000001: Hibernate Validator 6.2.3.Final]

You should immediately see a new container running on the Orchestrator appliance:

CONTAINER ID        IMAGE                                                                                                COMMAND                  CREATED             STATUS              PORTS                                            NAMES
b774150f826b        athena-docker-local.artifactory.eng.vmware.com/castor:1.4.0.0.91                                     "./wait-for-it.sh peâ¦"   2 minutes ago       Up 2 minutes                                                         orchestrator-runtime_castor_1

This container will use the descriptor files created earlier to orchestrate the replica and client node deployment and configuration.

You will also see that the content library created has synced against the Orchestrator appliance:

The replica nodes will be the first VMs to be created:

You will see separate OVF deployment tasks for each replica node. All Blockchain VM names start with the auto-generated blockchain ID (b4af2e91-7930-4ed9-9476-b9a21c4cbd7e in this example).

If you suspect that the deployment is not proceeding as expected, you can run docker logs -f against the orchestrator-runtime_persephone-provisioning_1 and orchestrator-runtime_config-service_1 containers on the Orchestrator appliance to get more details on the progress.

Once all four replicas are deployed, they should be reconfigured and powered on. The full copy client node deployment will start at the same time (to the same Resource Pool and Virtual Machine folder as the replicas).

The client node VM(s) will be deployed once the full copy client node VM is powered on.

As with the replica and full copy client nodes, the client node(s) will be reconfigured and powered on.

Very shortly after the client node(s) is powered on, you should see the docker-compose command complete successfully.

castor_1  | blockchain_id: "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e"
castor_1  | consortium_id: "827bd2dc-4dbb-4025-81f5-7a39426b0655"
castor_1  | ]
castor_1  | [INFO ] [2022-08-02 17:35:40.059] [thread=grpc-default-executor-1] [OpID=] [user=] [org=] [function=DeploymentExecutionEventResponseObserver] [message=onNext event received for requestId: 94d1199f-bfb9-4859-9105-b660a827de3b, event: session_id: "19a9e8d8-5e5a-494b-860d-2e379f5c198b"
castor_1  | type: COMPLETED
castor_1  | status: SUCCESS
castor_1  | blockchain_id: "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e"
castor_1  | consortium_id: "827bd2dc-4dbb-4025-81f5-7a39426b0655"
castor_1  | blockchain_version: "1.6.0.1.266"
castor_1  | ]
castor_1  | [INFO ] [2022-08-02 17:35:40.200] [thread=grpc-default-executor-1] [OpID=] [user=] [org=] [function=DeploymentExecutionEventResponseObserver] [message=Deployment with requestId: 94d1199f-bfb9-4859-9105-b660a827de3b succeeded]
castor_1  | [INFO ] [2022-08-02 17:35:40.221] [thread=main] [OpID=] [user=] [org=] [function=DeployerServiceImpl] [message=Deployment completed with status: SUCCESS]
orchestrator-runtime_castor_1 exited with code 0

There will be two files created under the /home/blockchain/output folder related to the installation. The filenames will start with the blockchain:consortiumName specified in the deployment descriptor file (EPG_blockchain-deployment in my example) and end with a timestamp.

ls /home/blockchain/output/

EPG-blockchain-deployment_2022-08-02T16:59:52.25028
EPG-blockchain-deployment_2022-08-02T16:59:52.25028.json

EPG-blockchain-deployment_2022-08-02T16:59:52.25028 is a log of the installation while EPG-blockchain-deployment_2022-08-02T16:59:52.25028.json contains all of the deployment metadata. The files largely have the same information in them but the .json version should be easier to parse. You will need to examine the .json file to get password for the vmbc user on each Blockchain node if you need to log in to any of them. This file also contains several other parameters that are needed during day-2 operations so it is a good idea to save a copy of it somewhere safe.

EPG-blockchain-deployment_2022-08-02T16:59:52.25028

Starting deployment type: PROVISION, at 2022-08-02T16:59:52.256089
Consortium Name: EPG-blockchain-deployment, Consortium Id: 827bd2dc-4dbb-4025-81f5-7a39426b0655
Blockchain Version: 1.6.0.1.266
Deployment Request Id: 94d1199f-bfb9-4859-9105-b660a827de3b
DAML message type: null
Blockchain Id: b4af2e91-7930-4ed9-9476-b9a21c4cbd7e
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: /0bedb2f1-8aa1-4642-a922-69d5f23edeb7, key: PRIVATE_IP, value: 192.168.100.31
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: DAML_SDK_VERSION, value: 2.0.1
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: DataDeviceId, value: 2001
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: ENCRYPT_SECRETS, value: True
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: NODE_LOGIN, value: Bl0ckch@!n
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: NODE_USER, value: vmbc
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: PRIVATE_IP, value: 192.168.100.31
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: VM_IP, value: 192.168.100.31
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: vmId, value: b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-0bedb2f1-8aa1-4642-a922-69d5f23edeb7
Node Id: 0bedb2f1-8aa1-4642-a922-69d5f23edeb7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004, key: vmInstance, value: vm-10004
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: /8c7528a4-de07-4ad8-8857-0decb8cf00e0, key: PRIVATE_IP, value: 192.168.100.36
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: DAML_SDK_VERSION, value: 2.0.1
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: DataDeviceId, value: 2001
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: ENCRYPT_SECRETS, value: True
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: NODE_LOGIN, value: Bl0ckch@!n
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: NODE_USER, value: vmbc
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: OBJECT_STORE_ACCESS_KEY, value: minio
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: OBJECT_STORE_BUCKET_NAME, value: blockchain
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: OBJECT_STORE_PROTOCOL, value: HTTP
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: OBJECT_STORE_SECRET_KEY, value: minio123
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: OBJECT_STORE_URL, value: 192.168.110.60:9000
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: PRIVATE_IP, value: 192.168.100.36
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: VM_IP, value: 192.168.100.36
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: vmId, value: b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-8c7528a4-de07-4ad8-8857-0decb8cf00e0
Node Id: 8c7528a4-de07-4ad8-8857-0decb8cf00e0, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005, key: vmInstance, value: vm-10005
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: /9014a45f-6fe0-44be-8eae-6260763e7daa, key: PRIVATE_IP, value: 192.168.100.33
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: DAML_SDK_VERSION, value: 2.0.1
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: DataDeviceId, value: 2001
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: ENCRYPT_SECRETS, value: True
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: NODE_LOGIN, value: Bl0ckch@!n
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: NODE_USER, value: vmbc
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: PRIVATE_IP, value: 192.168.100.33
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: VM_IP, value: 192.168.100.33
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: vmId, value: b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-9014a45f-6fe0-44be-8eae-6260763e7daa
Node Id: 9014a45f-6fe0-44be-8eae-6260763e7daa, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003, key: vmInstance, value: vm-10003
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: /a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, key: PRIVATE_IP, value: 192.168.100.35
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: CLIENT_BACKUP_ENABLED, value: false
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: CLIENT_CONFIG_PROPERTIES, value: {
}
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: CLIENT_ENDPOINT, value: https://192.168.100.35:6865
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: CLIENT_GROUP_ID, value: 08ad38df-02fe-448a-9210-e56f1ca8d814
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: CLIENT_GROUP_NAME, value: Group1
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: CLIENT_POSTGRES_PROPERTIES, value: {
  "values": {
    "max_connections": "300",
    "shared_buffers": "4GB",
    "effective_cache_size": "12GB",
    "maintenance_work_mem": "1GB",
    "checkpoint_completion_target": "0.9",
    "wal_buffers": "16MB",
    "default_statistics_target": "100",
    "random_page_cost": "4",
    "effective_io_concurrency": "2",
    "work_mem": "40MB",
    "min_wal_size": "2GB",
    "max_wal_size": "8GB",
    "max_worker_processes": "4",
    "max_parallel_workers_per_gather": "2",
    "max_parallel_workers": "4",
    "max_parallel_maintenance_workers": "2"
  }
}
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: DAML_DB_PASSWORD, value: b1o_N4-sU6rtS8S
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: DAML_SDK_VERSION, value: 2.0.1
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: DataDeviceId, value: 2001
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: ENCRYPT_SECRETS, value: True
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: NODE_LOGIN, value: Bl0ckch@!n
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: NODE_USER, value: vmbc
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: PRIVATE_IP, value: 192.168.100.35
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: VM_IP, value: 192.168.100.35
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: vmId, value: b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-a1b9dae0-a16a-472b-bfd3-fb76c241ffdf
Node Id: a1b9dae0-a16a-472b-bfd3-fb76c241ffdf, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006, key: vmInstance, value: vm-10006
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: /ad662820-f24e-451d-ac9a-e653af51e3d7, key: PRIVATE_IP, value: 192.168.100.32
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: DAML_SDK_VERSION, value: 2.0.1
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: DataDeviceId, value: 2001
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: ENCRYPT_SECRETS, value: True
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: NODE_LOGIN, value: Bl0ckch@!n
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: NODE_USER, value: vmbc
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: PRIVATE_IP, value: 192.168.100.32
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: VM_IP, value: 192.168.100.32
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: vmId, value: b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-ad662820-f24e-451d-ac9a-e653af51e3d7
Node Id: ad662820-f24e-451d-ac9a-e653af51e3d7, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002, key: vmInstance, value: vm-10002
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: /f69f6599-5cc5-4013-87c4-01f8cd620433, key: PRIVATE_IP, value: 192.168.100.34
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: DAML_SDK_VERSION, value: 2.0.1
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: DataDeviceId, value: 2001
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: ENCRYPT_SECRETS, value: True
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: NODE_LOGIN, value: Bl0ckch@!n
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: NODE_USER, value: vmbc
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: PRIVATE_IP, value: 192.168.100.34
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: VM_IP, value: 192.168.100.34
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: vmId, value: b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-f69f6599-5cc5-4013-87c4-01f8cd620433
Node Id: f69f6599-5cc5-4013-87c4-01f8cd620433, name: https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001, key: vmInstance, value: vm-10001
Blockchain Id: b4af2e91-7930-4ed9-9476-b9a21c4cbd7e, completion status: SUCCESS
RequestId: 94d1199f-bfb9-4859-9105-b660a827de3b succeeded
Deployment finished at 2022-08-02T17:33:07.518337 with status SUCCESS

EPG-blockchain-deployment_2022-08-02T16:59:52.25028.json

{
  "nodes": [
    {
      "nodeId": "f69f6599-5cc5-4013-87c4-01f8cd620433",
      "names": [
        "/f69f6599-5cc5-4013-87c4-01f8cd620433",
        "https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10001"
      ],
      "properties": {
        "NODE_LOGIN": "Bl0ckch@!n",
        "DAML_SDK_VERSION": "2.0.1",
        "SIGNATURE_SAVE": "DISABLE",
        "vmInstance": "vm-10001",
        "vmId": "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-f69f6599-5cc5-4013-87c4-01f8cd620433",
        "ENCRYPT_SECRETS": "True",
        "NODE_USER": "vmbc",
        "PRIVATE_IP": "192.168.100.34",
        "VM_IP": "192.168.100.34",
        "DataDeviceId": "2001"
      }
    },
    {
      "nodeId": "9014a45f-6fe0-44be-8eae-6260763e7daa",
      "names": [
        "/9014a45f-6fe0-44be-8eae-6260763e7daa",
        "https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10003"
      ],
      "properties": {
        "NODE_LOGIN": "Bl0ckch@!n",
        "DAML_SDK_VERSION": "2.0.1",
        "SIGNATURE_SAVE": "DISABLE",
        "vmInstance": "vm-10003",
        "vmId": "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-9014a45f-6fe0-44be-8eae-6260763e7daa",
        "ENCRYPT_SECRETS": "True",
        "NODE_USER": "vmbc",
        "PRIVATE_IP": "192.168.100.33",
        "VM_IP": "192.168.100.33",
        "DataDeviceId": "2001"
      }
    },
    {
      "nodeId": "8c7528a4-de07-4ad8-8857-0decb8cf00e0",
      "names": [
        "/8c7528a4-de07-4ad8-8857-0decb8cf00e0",
        "https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10005"
      ],
      "properties": {
        "NODE_LOGIN": "Bl0ckch@!n",
        "OBJECT_STORE_URL": "192.168.110.60:9000",
        "vmInstance": "vm-10005",
        "OBJECT_STORE_SECRET_KEY": "minio123",
        "vmId": "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-8c7528a4-de07-4ad8-8857-0decb8cf00e0",
        "OBJECT_STORE_PROTOCOL": "HTTP",
        "VM_IP": "192.168.100.36",
        "DAML_SDK_VERSION": "2.0.1",
        "SIGNATURE_SAVE": "DISABLE",
        "ENCRYPT_SECRETS": "True",
        "NODE_USER": "vmbc",
        "OBJECT_STORE_ACCESS_KEY": "minio",
        "OBJECT_STORE_BUCKET_NAME": "blockchain",
        "PRIVATE_IP": "192.168.100.36",
        "DataDeviceId": "2001"
      }
    },
    {
      "nodeId": "ad662820-f24e-451d-ac9a-e653af51e3d7",
      "names": [
        "/ad662820-f24e-451d-ac9a-e653af51e3d7",
        "https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10002"
      ],
      "properties": {
        "NODE_LOGIN": "Bl0ckch@!n",
        "DAML_SDK_VERSION": "2.0.1",
        "SIGNATURE_SAVE": "DISABLE",
        "vmInstance": "vm-10002",
        "vmId": "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-ad662820-f24e-451d-ac9a-e653af51e3d7",
        "ENCRYPT_SECRETS": "True",
        "NODE_USER": "vmbc",
        "PRIVATE_IP": "192.168.100.32",
        "VM_IP": "192.168.100.32",
        "DataDeviceId": "2001"
      }
    },
    {
      "nodeId": "0bedb2f1-8aa1-4642-a922-69d5f23edeb7",
      "names": [
        "/0bedb2f1-8aa1-4642-a922-69d5f23edeb7",
        "https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10004"
      ],
      "properties": {
        "NODE_LOGIN": "Bl0ckch@!n",
        "DAML_SDK_VERSION": "2.0.1",
        "SIGNATURE_SAVE": "DISABLE",
        "vmInstance": "vm-10004",
        "vmId": "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-0bedb2f1-8aa1-4642-a922-69d5f23edeb7",
        "ENCRYPT_SECRETS": "True",
        "NODE_USER": "vmbc",
        "PRIVATE_IP": "192.168.100.31",
        "VM_IP": "192.168.100.31",
        "DataDeviceId": "2001"
      }
    },
    {
      "nodeId": "a1b9dae0-a16a-472b-bfd3-fb76c241ffdf",
      "names": [
        "/a1b9dae0-a16a-472b-bfd3-fb76c241ffdf",
        "https://vcsa-01a.corp.vmw//rest/vcenter/vm/vm-10006"
      ],
      "properties": {
        "NODE_LOGIN": "Bl0ckch@!n",
        "vmInstance": "vm-10006",
        "vmId": "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e-a1b9dae0-a16a-472b-bfd3-fb76c241ffdf",
        "CLIENT_GROUP_ID": "08ad38df-02fe-448a-9210-e56f1ca8d814",
        "VM_IP": "192.168.100.35",
        "DAML_SDK_VERSION": "2.0.1",
        "CLIENT_CONFIG_PROPERTIES": "{\n}",
        "CLIENT_GROUP_NAME": "Group1",
        "CLIENT_ENDPOINT": "https://192.168.100.35:6865",
        "ENCRYPT_SECRETS": "True",
        "NODE_USER": "vmbc",
        "CLIENT_POSTGRES_PROPERTIES": "{\n  \"values\": {\n    \"max_connections\": \"300\",\n    \"shared_buffers\": \"4GB\",\n    \"effective_cache_size\": \"12GB\",\n    \"maintenance_work_mem\": \"1GB\",\n    \"checkpoint_completion_target\": \"0.9\",\n    \"wal_buffers\": \"16MB\",\n    \"default_statistics_target\": \"100\",\n    \"random_page_cost\": \"4\",\n    \"effective_io_concurrency\": \"2\",\n    \"work_mem\": \"40MB\",\n    \"min_wal_size\": \"2GB\",\n    \"max_wal_size\": \"8GB\",\n    \"max_worker_processes\": \"4\",\n    \"max_parallel_workers_per_gather\": \"2\",\n    \"max_parallel_workers\": \"4\",\n    \"max_parallel_maintenance_workers\": \"2\"\n  }\n}",
        "DAML_DB_PASSWORD": "b1o_N4-sU6rtS8S",
        "PRIVATE_IP": "192.168.100.35",
        "CLIENT_BACKUP_ENABLED": "false",
        "DataDeviceId": "2001"
      }
    }
  ],
  "metadata": {
    "deploymentType": "PROVISION",
    "consortiumName": "EPG-blockchain-deployment",
    "startingTime": "2022-08-02T16:59:52.256089",
    "deployment type": "PROVISION",
    "blockchainVersion": "1.6.0.1.266",
    "finishingTime": "2022-08-02T17:33:07.518337",
    "deploymentRequestId": "94d1199f-bfb9-4859-9105-b660a827de3b",
    "blockchainId": "b4af2e91-7930-4ed9-9476-b9a21c4cbd7e",
    "consortiumId": "827bd2dc-4dbb-4025-81f5-7a39426b0655",
    "status": "SUCCESS"
  }

Validate the Installation

Once the deployment is complete, you can examine the metadata file (/home/blockchain/output/EPG-blockchain-deployment_2022-08-02T16:59:52.25028.json in this example) to get the password for the vmbc user account (look for the NODE_LOGIN parameter for each node). You can ssh to each node to check that all needed containers are up and running. Be sure to preface all commands with sudo as the vmbc user has few privileges on the Blockchain nodes.

On replica nodes, you should see the following containers running:

sudo docker ps
 
CONTAINER ID   IMAGE                                             COMMAND                  CREATED      STATUS                          PORTS                                                                                                      NAMES
55b5e3c63f12   96e1e024f557                                      "/go/bin/agent-linux"    5 days ago   Up 12 minutes (healthy)         0.0.0.0:5775->5775/udp, 0.0.0.0:6831-6832->6831-6832/udp, 0.0.0.0:5778->5778/tcp                           jaeger-agent
1328c3b7af1e   def4a8d834f5                                      "/concord/concord-en…"   5 days ago   Up 12 minutes (healthy)         0.0.0.0:3501->3501/tcp, 3501-3505/udp, 0.0.0.0:50051->50051/tcp, 3502-3505/tcp, 127.0.0.1:5458->5458/tcp   concord
4896943fb83b   6b3f9670fd94                                      "/bin/bash /opt/wave…"   5 days ago   Up 12 minutes                   2878/tcp, 3878/tcp, 4242/tcp                                                                               wavefront-proxy
abd1f875147b   4697884441e0                                      "/doc/daml/entrypoin…"   5 days ago   Up 12 minutes (healthy)         0.0.0.0:55000->55000/tcp                                                                                   daml_execution_engine
32a61957f4f0   cb15bf57b0aa                                      "tini -- /bin/entryp…"   5 days ago   Up 12 minutes (healthy)         5140/tcp, 24224/tcp                                                                                        fluentd
5dce10507743   harbor.corp.vmw/vmwblockchain/agent:1.6.0.1.266   "java -jar node-agen…"   5 days ago   Up 12 minutes                   0.0.0.0:8546->8546/tcp                                                                                     agent
070ae96c04e2   870ee38129f8                                      "/entrypoint.sh tele…"   5 days ago   Up 12 minutes (healthy)         8092/udp, 8125/udp, 8094/tcp, 0.0.0.0:9273->9273/tcp                                                       telegraf

On full copy client nodes, you should see the following containers running:

sudo docker ps
 
CONTAINER ID   IMAGE                                             COMMAND                  CREATED      STATUS                          PORTS                                                                                                      NAMES
f79ad9d8eba8   870ee38129f8                                      "/entrypoint.sh tele…"   5 days ago   Up 11 minutes (healthy)         8092/udp, 8125/udp, 8094/tcp, 0.0.0.0:9273->9273/tcp                                                       telegraf
cfae4b504f81   96e1e024f557                                      "/go/bin/agent-linux"    5 days ago   Up 12 minutes (healthy)         0.0.0.0:5775->5775/udp, 0.0.0.0:6831-6832->6831-6832/udp, 0.0.0.0:5778->5778/tcp                           jaeger-agent
6f224aa1fa62   6b3f9670fd94                                      "/bin/bash /opt/wave…"   5 days ago   Up 12 minutes                   2878/tcp, 3878/tcp, 4242/tcp                                                                               wavefront-proxy
8282e38571fd   def4a8d834f5                                      "/concord/concord-en…"   5 days ago   Up 12 minutes (healthy)         0.0.0.0:3501->3501/tcp, 3501-3505/udp, 0.0.0.0:50051->50051/tcp, 3502-3505/tcp, 127.0.0.1:5458->5458/tcp   concord
fdc660633485   cb15bf57b0aa                                      "tini -- /bin/entryp…"   5 days ago   Up 12 minutes (healthy)         5140/tcp, 24224/tcp                                                                                        fluentd
5e4cb07ebfe4   harbor.corp.vmw/vmwblockchain/agent:1.6.0.1.266   "java -jar node-agen…"   5 days ago   Up 12 minutes                   0.0.0.0:8546->8546/tcp                                                                                     agent

On client nodes, you should see the following containers running:

sudo docker ps
 
CONTAINER ID   IMAGE                                             COMMAND                  CREATED      STATUS                          PORTS                                                                              NAMES
41077d06d4cf   3382e600c110                                      "/clientservice/clie…"   5 days ago   Up 11 minutes (healthy)         0.0.0.0:50505->50505/tcp                                                           clientservice
f09352787a2f   870ee38129f8                                      "/entrypoint.sh tele…"   5 days ago   Up 11 minutes (healthy)         8092/udp, 8125/udp, 8094/tcp, 0.0.0.0:9273->9273/tcp                               telegraf
0b4ace21eb4f   f1edf3cb8810                                      "/cre/cre_server"        5 days ago   Up 11 minutes                                                                                                      cre
cb9ae083f8b4   96e1e024f557                                      "/go/bin/agent-linux"    5 days ago   Up 11 minutes (healthy)         0.0.0.0:5775->5775/udp, 0.0.0.0:6831-6832->6831-6832/udp, 0.0.0.0:5778->5778/tcp   jaeger-agent
79b054c4aa96   418c1f4894c2                                      "/bin/sh -c '/doc/da…"   5 days ago   Up 11 minutes (healthy)         0.0.0.0:6865->6865/tcp                                                             daml_ledger_api
d95e81a1b4aa   6b3f9670fd94                                      "/bin/bash /opt/wave…"   5 days ago   Up 11 minutes                   2878/tcp, 3878/tcp, 4242/tcp                                                       wavefront-proxy
16aa15d0f24f   01e47563f112                                      "/doc/daml/scripts/d…"   5 days ago   Up 11 minutes (healthy)         5432/tcp                                                                           daml_index_db
c9bdc55ed1d0   cb15bf57b0aa                                      "tini -- /bin/entryp…"   5 days ago   Up 11 minutes (healthy)         5140/tcp, 24224/tcp                                                                fluentd
6898c4ba6f18   harbor.corp.vmw/vmwblockchain/agent:1.6.0.1.266   "java -jar node-agen…"   5 days ago   Up 11 minutes                   0.0.0.0:8546->8546/tcp                                                             agent

If you did not configure Tanzu Observability, you will not see a wavefront-proxy container running.

If you do not see the same containers running, you can run docker ps -a to see if one or more has crashed and then docker logs <container id> to see what might have gone wrong.

If you see no containers running, something might have gone wrong with the firstboot process and you should examine the /var/log/cloud-init-output.log log on the node to get a better idea of what happened.

At the end of the firstboot process, the agent container will be running. The agent container will then communicate with the Orchestrator appliance to pull down the configuration and bring up all other containers. If only the agent container is running, you should examine it’s logs (docker logs agent) to better determine what didn’t work quite right.

View Blockchain Metrics and Logs

As noted earlier, I configured my Blockchain deployment to send logs and metrics to my local ELK installation and also send metrics to Tanzu Observability.

If you are using ELK and configured it as an endpoint for metrics, you should see a new index with the same name as the blockchain id (b4af2e91-7930-4ed9-9476-b9a21c4cbd7e in this example).

And if you also configured ELK as an endpoint for logs, you should see a new data stream similar to the following (coming from logstash):

You can create a Data View for either of these and then view logs and/or metrics from the Discover page.

I went so far as to create a dashboard using some common metrics that our SRE team uses to monitor the health of Blockchain deployments in Tanzu Observability.

If you have configured your Blockchain deployment to send metrics to Tanzu Observability, you can make use of the Blockchain integration there to see several dashboards that will give you insight into the health of your Blockchain deployment.

In Tanzu Observability, navigate to Integrations and type Blockchain in the search bar.

Select VMware Blockchain

Click on VMware Blockchain and then click on the Dashboards tab.

From here you can select any of the available dashboards to dig deeper into the health of your Blockchain deployment.

This is the Concord Metrics dashboard and the couple of blips of activity you can see on a few of the visuals are representative of me submitting a few transactions using my Blockchain deployment.

If the pre-configured dashboards don’t have what you’re looking for, you can view individual metrics from the Metrics tab.

I plan on doing a few more posts in the coming weeks/months related to deploying a sample DAML application, scaling out the Blockchain nodes and backup and recovery.