Run Algorand Indexer on Azure with Azure CLI

The Algorand blockchain, like other popular blockchains, generates a huge amount of data. A new block is created every 4 to 5 seconds with a number of transactions ranging from a few dozen to over a hundred transactions in each block. Reporting on those data is crucial for any business application connected to the blockchain and this requires a powerful search engine, capable for example of retrieving the so called “whale transactions”, or to track the balance of a specific account.

The Algorand Indexer provides powerful search capabilities for Algorand blockchain. You can query the indexer for transactions, assets, blocks, accounts, providing various criteria. The Indexer requires a PostgreSQL database to store the ledger data and a connexion to an Algorand archive node to read the data. There is already a complete guide on how to install an Indexer on the Algorand developer portal, but this tutorial will focus on doing it on Microsoft Azure, the cloud service provided by Microsoft. We will first use Azure Cli which is a command line interface to interact with Azure, and then Azure ARM Template which allow using json template for deployment, thus allowing for deployment automation.

Complete resources can be found in the github repository

1. Azure Cli or Azure ARM Template?

When you want to automate resources deployment on Azure, you will generally use ARM Template to define your resources in JSON files and put them under version control, with Git for example. The main issue with ARM Template is that when you have complex deployments with resources linked to each other, it can become difficult. You need to know some configuration details that are hidden when you use Azure portal or Azure Cli.

For this tutorial, we will therefore first use Azure Cli to deploy our resources, then we will export all the created resources as an azure template. And after some necessary cleaning in the exported files, we will redeploy our stack using the template.

2. Install Azure CLI

Azure CLI is the command line interface provided by Microsoft to provision resources on Azure. It is also your friend when you want to integrate azure deployment with an automated workflow. Run below commands to install it.

 #Install dependencies
sudo apt-get update
sudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg

#Install Microsoft signing key
curl -sL https://packages.microsoft.com/keys/microsoft.asc |
    gpg --dearmor |
    sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg > /dev/null

#Add the Azure CLI software repository
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $(lsb_release -cs) main" |
    sudo tee /etc/apt/sources.list.d/azure-cli.list

#Install
sudo apt-get update
sudo apt-get install azure-cli

3. Login

Once the installation is complete, you need to login.

az login

If you run this on a system with a GUI, it will open a web page in your browser for you to authenticate. Otherwise, it will display a code that you need to enter on another device on which you have a web browser. Go to https://aka.ms/devicelogin and enter the code when prompted. If like me you run on a system with no browser available, run az login --use-device-code directly.

4. Setup your workspace.

Before creating any resource in azure, the first choice you have to make is the location of your resources. Azure provides regions and availability zones that define where your resources will physically be located. Not all resources and services are available in all regions. Except for a very short testing purpose, you should not select a random location for your resources.

Azure resource group is a management feature that allows you to group related resources and apply global setting on them. All the resources that we will create for this tutorial will belong to the same resource group. This is very useful as deleting the resource group at the end of the tutorial will delete all resources created in the group.

export ALGORAND_RESOURCE_GROUP=algorand-resource-group
export ALGORAND_LOCATION=WestEurope
export ALGORAND_VNET=algorand-vnet
export ALGORAND_DB_SUBNET=algorand-subnet-db
export ALGORAND_VM_SUBNET=algorand-subnet-vm

#Set the default location to apply to all resources
az configure --defaults location=$ALGORAND_LOCATION
#create resource group
az group create --name $ALGORAND_RESOURCE_GROUP
#Set default resource group
az configure --defaults group=$ALGORAND_RESOURCE_GROUP

5. Create the Indexer Database

As of the time of writing this tutorial, the last version of Algorand Indexer is 2.9.0 and was tested with Postgres 13. There are two different services provided by Azure to deploy a managed PostgreSQL server: Azure single server and Azure flexible server.

One of the main differences between both is that the later targets mission-critical workload. Azure single server would be enough for our tutorial, but only Azure flexible server supports PostgreSQL 13 as of the time of writing this tutorial.
Use az postgres flexible-server create commands to create your database

export DB_USER=algorandindexer
export DB_PASSWORD=xxxxxxxx
export DB_NAME=algorand-indexer-db

az postgres flexible-server create --name algorand-indexer-flexible-server \
  --vnet $ALGORAND_VNET --subnet $ALGORAND_DB_SUBNET --address-prefixes 172.0.0.0/16 --subnet-prefixes 172.0.0.0/24\
  --admin-user $DB_USER --admin-password $DB_PASSWORD \
  --sku-name Standard_B1ms --tier Burstable --storage-size 1024  --tags "Billing=algorand" --version 13 \
  --database-name $DB_NAME --yes

When you create a flebible server, a Vnet (virtual network) and a subnet are required. You can use an existing Vnet but a dedicated subnet is required and will be used only for the resources created in the flexible server. You cannot create other type of resources in that subnet. If the Vnet and/or the subnet do not exist, azure-cli can create them for you automatically before creating the database server. This is why we provided --address-prefixes and --subnet-prefixes. We could also create the Vnet and the subnet using az network vnet create and az network vnet subnet create commands of Azure virtual network and then pass the vnet and subnet names to the database command.

The parameter --yes is mandatory if you use the command with an automated workflow. It automatically confirms the creation of the Vnet and the subnet. If you don’t use it, azure-cli will prompt you to confirm the creation.

Regarding --sku-name, --tier and --storage-size, these parameters are about the computation power of your database server. You should select values that correspond to your need and check the pricing. Note however that the --storage-size unit is GB, so we have 1 To here. When using a free tier account, since the storage size is limited, you will probably need to reduce this value. Unfortunately, the indexer may therefore not have enough space to complete its initialization.

The output of the command will display the private DNS name of the database in the host field of the json output. You need to keep it for later use when running the indexer binaries. See Figure 5-1 for the DNS name generated in our example.

Figure 5-1 Output command display showing private DNS name

6. Create Azure VM to install and run Indexer binaries.

Azure VM is used to create virtual machines. We will use one to install and run the Algorand indexer binaries.
When you create a virtual machine, you need to setup how you will log into your server. A best practice is to use SSH keys and avoid a simple user/password authentication. Azure also allows you to create and manage SSH keys.

#First create a subnet for all your vms
az network vnet subnet create --vnet-name $ALGORAND_VNET --address-prefixes 172.0.1.0/24 --name $ALGORAND_VM_SUBNET

#create ssh key for the vm. the private key will be saved on 
#the local computer under ~/.ssh/
az sshkey create --name "algorand-key"

#create vm
az vm create --name algorand-indexer-vm --image Canonical:UbuntuServer:18.04-LTS:latest \
  --admin-username algoranduser --ssh-key-name algorand-key \
  --size Standard_B2s --public-ip-address-allocation static \
  --subnet $ALGORAND_VM_SUBNET --vnet-name $ALGORAND_VNET \
  --custom-data ./install-indexer-binaries.yml

The second command will generate a public/private key pair and store them in ~/.ssh/ on your computer (check the command output for exact location). Make sure to run chmod 600 on the private key before using it otherwise your ssh client will complain. Azure CLI will upload the public key to Azure and you can reference it by name when creating your vm. Doing this, the public key will be deployed to the ~/.ssh/authorized_keys of you azure vm.

--public-ip-address-allocation static will create a static ip address for your vm. --size will define the computation power of your vm. You can run az vm list-skus to display available values and also check this page for a description of available configurations.

--image allows you to select the operating system that will be installed on your vm. Check this page for additional information on available images.

--custom-data allows you to pass a cloud init initialization file that will run once your vm has been created. We use it here to download Algorand Indexer binaries and install them. Here is the content of the yaml file referenced in the command.

#cloud-config

runcmd:
  - mkdir /run/algorand
  - wget https://github.com/algorand/indexer/releases/download/2.9.0/algorand-indexer_linux_amd64_2.9.0.tar.bz2 -O /run/algorand/algorand-indexer_linux_amd64_2.9.0.tar.bz2
  - wget https://raw.githubusercontent.com/algorand/go-algorand/master/installer/genesis/betanet/genesis.json -O /run/algorand/betanet_genesis.json
  - mv /run/algorand/algorand-indexer_linux_amd64_2.9.0.tar.bz2 /home/algoranduser/
  - mv /run/algorand/betanet_genesis.json /home/algoranduser/
  - chown algoranduser /home/algoranduser/algorand-indexer_linux_amd64_2.9.0.tar.bz2
  - chown algoranduser /home/algoranduser/betanet_genesis.json
  - runuser -u algoranduser mkdir /home/algoranduser/indexer
  - runuser -c "tar -xf /home/algoranduser/algorand-indexer_linux_amd64_2.9.0.tar.bz2 -C /home/algoranduser/indexer" - algoranduser
  - runuser -c "mv /home/algoranduser/betanet_genesis.json /home/algoranduser/indexer" - algoranduser

Note that the initialization script runs as root, so you need to run the appropriate commands (i.e runuser and/or chown) to use the user created for your vm above. Here we just download the binaries archive and the genesis file for Algorand Betanet (for example if we plan to run an Indexer on Betanet). We will later connect to the vm to run the indexer. But before that, we need to make sure that our vm can connect to the database.

7. Test you database connection

When you create a Vm in azure, some defaults networking rules are applied to your resources. For example, all resources created in a Vnet can reach each other, you don’t have to add specific rules. The AllowVnetInbound rule is automatically attached to the network interface of your vm. This means that our Indexer vm can already reach the database.

#connect to the indexer vm from the computer on which you installed azure-cli
ssh -i /home/<user>/.ssh/<generated_ssh_private_key_path> algoranduser@<vm_ip_static_adress>

az postgres flexible-server connect --admin-user algorandindexer --name algorand-indexer-flexible-server --interactive

Enter the database password when prompted and you should get a command line in your database server form which you can send postgresql commands. Type exit to go back to the indexer vm.

8. Run indexer

Now you need to use the binaries downloaded in the cloud init script above to run the indexer

cd /home/algoranduser/indexer
./algorand-indexer daemon -P "host=algorand-indexer-flexible-server.postgres.database.azure.com port=5432 user=algorandindexer password= dbname=postgres sslmode=require" --algod-net="archive_node_url" --algod-token="archive_node_token_access" --genesis [path_to_genesis_file]

Note that the process will run for a few days to complete if you run it on BetaNet, TestNet or MainNet.

9. Export resources to create ARM template

It is now time to create a template corresponding to all our indexer resources. It is also time for you to see how many resources where automatically created for you when using Azure Cli.

az group export --name algorand-indexer-deployment > algorand_indexer_deployment.json

It is possible to filter the resources that are exported to the deployment, but since we used a specific resources group for this tutorial, we will export everything.
Now you need to clean the template because it contains many static values that are only valid for your current deployment. In order to easily reuse it, you have to:

Update the VM resource.

Locate the VM resource, which has type: "Microsoft.Compute/virtualMachines" and remove fields properties.storageProfile.osDisk.managedDisk.id, properties.osProfile.requireGuestProvisionSignal. Add a new field properties.osProfile.customData which must contain the base64 encoded version of the cloud init script that we used on step 6.

Update the database server.

Locate the database server resource which has type: Microsoft.DBforPostgreSQL/flexibleServers and add field properties.administratorLoginPassword. Since it is a password, you should make sure to not use a clear string as value. Please, refer to the template I provided in the Github Repository for an example.

More cleanup.

There are a lot of resources of type Microsoft.DBforPostgreSQL/flexibleServers/configurations. I didn’t include them in the template of the Github repository for simplicity. They contain defaults configurations that will be created automatically. You can remove them.

If you want to test the template, run az deployment group create --resource-group $ALGORAND_RESOURCE_GROUP --parameters @full_deployment_parameters.json --template-file ./full_deployment_template.json. The full_deployment_parameters.json file contains parameters required for the deployment. Please check the Github repository for an example.

10. Summary

In this tutorial, we covered the steps required to setup and run an Algorand Indexer on Microsoft Azure cloud, first using Azure CLI, the command line interface of Azure, second using Azure ARM templates created by exporting all the resources created with Azure CLI. The second option enables the versioning of your resource.

Note that topics like high availability were not covered in this tutorial which is provided for learning purpose and is not intended to run production ready infrastructure.