Change ARM template to use a supported Ubuntu LTS version

[andrewrimmer]

Currently, the ARM template deploys only to Ubuntu 16.04-LTS VMs

Ubuntu 16.04 is over 5 years old and standard support for it expired in April 2021. Any releases using the ARM template will now be deploying clusters with an unsupported OS.

There have been 2 subsequent LTS releases and it would make sense to switch to using 20.04 LTS. As mentioned in a separate issue, it would be helpful if the Ubuntu version could be provided as a parameter.

[jwcw]

I asked this the other day, no response yet. #387

EDIT: Removed my erroneous suggestion to change the SKU in the template (see my follow-up comment for what we actually did)

[russcam]

Thanks for opening @andrewrimmer and @jwcw. I'll close this issue in order to consolidate the discussion on #387

[josh-koslosky-dsg]

@jwcw - Are you able to share how you updated the OS in the ARM template? I've tried changing the sku in the ubuntuSettings.json file, but it's still creating at v16.04.

@russcam - Maybe this info could be added to one of the readmes?

[jwcw]

@josh-koslosky-dsg sorry, it turns out I am mistaken. I went back to see how we did it, it turns out we didn't update the elastic/azure-marketplace templates, but instead exported ARM templates from the running 16.04 machines in the cluster that were created with elastic/azure-marketplace. With the exported ARM templates we changed the OS images and then created new nodes for the cluster using those. We migrated the indices to the new nodes and then decommissioned the 16.04 machines.

This is probably not a practical procedure for a new cluster, in our case our cluster had been running and contained data.

[josh-koslosky-dsg]

@jwcw - Thank you for clarifying. I was hoping I could create a cluster with the new version of Ubuntu, but there doesn't look to be a straightforward way to do this right now. Disk swaps seem to be the Azure recommended way for upgrades.

@russcam - Would this kind of change be an option in the future or is managed Elastic via Azure the way forward?

[jwcw]

You could also do a release upgrade in Ubuntu immediately after creating the cluster. We've done this in clusters where we had not other other OS image requirements.

[josh-koslosky-dsg]

So I figured out how to deploy an Elastic cluster via this ARM template with a different OS version.

The steps to do this are below:

1.) Fork the repo
2.) Update the ubuntuSettings.json file to whichever image you want (Note that Canonical changed the naming convention for 20.04)
3.) Either change the _artifactsLocation.defaultValue in the mainTempalte.json or set this value in your own json parameters file. It should point to your forked repos src directory.
4.) Make sure to also point to your forked version of the template when you run the group create command via the CLI.

[russcam]

@josh-koslosky-dsg There's some details on https://github.com/elastic/azure-marketplace/blob/master/build/README.md#development for forking the repository and updating links in the template to point to your fork repository. Running the command

npm run links -- --repo <username>/<repo> --branch <branch>

will update all the links in the template to point to your github fork at <username>/<repo> (<branch> will default to the current branch).

Once this is done, push your changes to your fork, and you'll be able to deploy it similar to how the Elastic ARM template is deployed. Taking the Azure CLI 2.0 example, the differences would be

template_base_uri=https://raw.githubusercontent.com/<username>/<repo>
template_version=<branch>
esVersion=7.11.1

az group deployment create \
  --resource-group "<name>" \
  --template-uri $template_base_uri/$template_version/src/mainTemplate.json \
  --parameters _artifactsLocation=$template_base_uri/$template_version/src/ \
               esVersion=$esVersion esClusterName=elasticsearch \
               vmDataDiskCount=1 dataNodesAreMasterEligible=Yes \
               adminUsername=russ adminPassword=Password1234 \
               securityBootstrapPassword=bootstrapPassword123 \
               securityAdminPassword=adminPassword123 \
               securityKibanaPassword=kibanaPassword123 \
               securityLogstashPassword=logstashPassword123 \
               securityBeatsPassword=beatsPassword123 \
               securityApmPassword=apmPassword123 \
               securityRemoteMonitoringPassword=remoteMonitoringPassword123

esVersion will need to be a valid version in the template. template_base_uri is assumed to be accessible from where the command is invoked.