Create a hardened Ubuntu Pro 18.04 LTS shared image with Azure Image Builder

In this tutorial, we will use Azure Image Builder to create a hardened Ubuntu Pro 18.04 LTS “golden” image in an Azure Shared Image Gallery.

The resulting images will have CIS hardening applied to them, which helps meet security best practice, CIS-specific requirements and also improves compliance with the Azure Linux Security Baseline policy.

What you’ll learn

• How to set up your Azure environment with a Shared Image Gallery and Azure resources you need to distribute the image within your environment
• How to create an image definition for Ubuntu Pro 18.04 LTS and customise the image build JSON to apply the CIS hardening and add any other applications we may want in every Ubuntu VM
• How to create the image version in this Azure Image Builder service
• How to create a VM from the image in the Shared Image Gallery

We will be using Ubuntu Pro as the starting point for our Ubuntu images. These are normally the best choice for production workloads on Azure and they include access to CIS hardening scripts.

What you’ll need

• A Microsoft Azure account
• Azure Command-Line Interface/az cli

Credits

This tutorial is based on the article Preview: Create a Linux image and distribute it to a Shared Image Gallery by using Azure CLI in the Microsoft documentation and an earlier version of this tutorial created by David Coronel here.

We will be using some pieces of information repeatedly, so we will create some variables to store that information. Then we will create the identity, image definition and gallery that we will need for the image.

Set variables for use throughout the tutorial

We will create the resource group below, so the name should be one you are not already using. We will delete the resource groups that we have created at the end of the tutorial.

# Resource group name - we are using ibUbuntuProGalleryRG in this example
sigResourceGroup=ibUbuntuProGalleryRG
# Datacenter location - we are using North Europe in this example
location=northeurope
# Additional region to replicate the image to - we are using West Europe in this example
additionalregion=westeurope

Now we will set variables for the Gallery Name and Image Definition Name. The image will be displayed in the Azure Portal as sigName/imageDefName.

# name of the shared image gallery - in this example we are using myGallery
sigName=myUbuntuProGallery
# name of the image definition to be created
imageDefName=UbuntuPro1804CIS
# image distribution metadata reference name
runOutputName=aibLinuxSIG

Create a variable for your subscription ID. If you only have one, this will be the output of:
az account show -o json | grep id
If you have more than one, you can use
az account list -o table
and take the value from the SubscriptionId column for the subscription you would like to use for this tutorial.

Then set this to the variable as follows:

subscriptionID=<Subscription ID>

Now we are going to set variables for the Ubuntu Pro plan we are going to use in the tutorial. If you have an Ubuntu Pro private offer with Canonical, for example including 24x7 Technical Support with SLAs, you will have a custom Offer and Sku we can enter these here instead. If not, we will use the plan name and product for the public Ubuntu Pro from the Azure Marketplace.

# ProPlanPublisher the 'Publisher' field for the Marketplace VM Offer we want to start from
ProPlanPublisher=canonical
# ProPlanOffer the 'Offer' field for the Marketplace VM Offer we want to start from
ProPlanOffer=0001-com-ubuntu-pro-bionic
# ProPlanSku the 'Sku' field for the Marketplace VM Offer we want to start from
ProPlanSku=pro-18_04-lts

Create required resources, identities and permissions

Create the resource group:

az group create -n $sigResourceGroup -l $location --subscription $subscriptionID

Image Builder will use the user-identity provided to inject the image into the Azure Shared Image Gallery (SIG). In this example, you will create an Azure role definition that has the granular actions to perform distributing the image to the SIG. The role definition will then be assigned to the user-identity.

# create user assigned identity for image builder to access the storage account where the script is located
identityName=aibBuiUserId$(date +'%s')
az identity create -g $sigResourceGroup -n $identityName --subscription $subscriptionID

# get identity id
imgBuilderCliId=$(az identity show -g $sigResourceGroup -n $identityName --subscription $subscriptionID -o json | grep "clientId" | cut -c16- | tr -d '",')

# get the user identity URI, needed for the template
imgBuilderId=/subscriptions/$subscriptionID/resourcegroups/$sigResourceGroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/$identityName

# this command will download an Azure role definition template, and update the template with the parameters specified earlier.
curl https://raw.githubusercontent.com/Azure/azvmimagebuilder/master/solutions/12_Creating_AIB_Security_Roles/aibRoleImageCreation.json -o aibRoleImageCreation.json

imageRoleDefName="Azure Image Builder Image Def"$(date +'%s')

# update the definition
sed -i -e "s/<subscriptionID>/$subscriptionID/g" aibRoleImageCreation.json
sed -i -e "s/<rgName>/$sigResourceGroup/g" aibRoleImageCreation.json
sed -i -e "s/Azure Image Builder Service Image Creation Role/$imageRoleDefName/g" aibRoleImageCreation.json

# create role definitions
az role definition create --role-definition ./aibRoleImageCreation.json

# need to wait a bit here
sleep 45

# grant role definition to the user assigned identity
# If this gives an error, wait a bit longer and try again
az role assignment create \
    --assignee $imgBuilderCliId \
    --role "$imageRoleDefName" \
    --scope /subscriptions/$subscriptionID/resourceGroups/$sigResourceGroup

Create an image definition and gallery

To use Image Builder with a shared image gallery, you need to have an existing image gallery and image definition. Image Builder will not create the image gallery and image definition for you.

We will start by creating a gallery and image definition. First, we will create the Gallery:

az sig create \
    -g $sigResourceGroup \
    --gallery-name $sigName \
    --subscription $subscriptionID

Then, create an image definition. This uses the variables we set earlier.

az sig image-definition create \
   -g $sigResourceGroup \
   --gallery-name $sigName \
   --gallery-image-definition $imageDefName \
   --publisher $ProPlanPublisher \
   --offer $ProPlanOffer \
   --sku $ProPlanSku \
   --os-type Linux \
   --plan-name $ProPlanSku \
   --plan-product $ProPlanOffer \
   --plan-publisher $ProPlanPublisher \
   --subscription $subscriptionID

Now we are going to create a template that contains the build instructions for the standard, “golden” image we want to create.

We can download a starting point for this from here

wget https://gist.githubusercontent.com/Hooloovoo/3e544681a12121a36b5dbda684465b8d/raw/85a7c4461e704b722b901ffc981e23db9a14ce82/BasicCISUbuntuPro1804SIGTemplate.json

Then we can customise it to use the values we have set above. The sed commands below simply replace the <variable> placeholders in BasicCISUbuntuPro1804SIGTemplate.json with the values for the parameters that we set earlier:

sed -i -e "s/<subscriptionID>/$subscriptionID/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<rgName>/$sigResourceGroup/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<imageDefName>/$imageDefName/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<sharedImageGalName>/$sigName/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<region1>/$location/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<region2>/$additionalregion/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<runOutputName>/$runOutputName/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s%<imgBuilderId>%$imgBuilderId%g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<ProPlanPublisher>/$ProPlanPublisher/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<ProPlanOffer>/$ProPlanOffer/g" BasicCISUbuntuPro1804SIGTemplate.json
sed -i -e "s/<ProPlanSku>/$ProPlanSku/g" BasicCISUbuntuPro1804SIGTemplate.json

Let’s review some of the sections of the BasicCISUbuntuPro1804SIGTemplate.json we have just updated. It is worth reading over all of this (short) file, but below we will look at particular sections in more detail. We should not need to change any of these values.

Note the below section:

        "source": {
            "type": "PlatformImage",
                "publisher": "canonical",
                "offer": "0001-com-ubuntu-pro-bionic",
                "sku": "pro-18_04-lts",
                "version": "latest",
        "planInfo": {
                    "planName": "pro-18_04-lts",
                    "planProduct": "0001-com-ubuntu-pro-bionic",
                    "planPublisher": "canonical"
                }
        },

This will show the plan details for the Marketplace or Private Offer VM image you are using as a starting point for your golden image.

The customize section allows us to run commands as part of the image building process. The following waits until Ubuntu’s ua client has attached to its subscription and then enables access to the CIS hardening scripts:

        "customize": [
            {
            "type": "Shell",
            "name": "WaitForUAtokenAutoAttach",
            "inline": [
                "sudo ua status --wait"
            ]
        },

        {
            "type": "Shell",
            "name": "EnableCISfeature",
            "inline": [
            	"sudo ua enable cis"
            ]
        },

In a real deployment, we would at this point in the customisation script modify /usr/share/ubuntu-scap-security-guides/cis-hardening/ruleset-params.conf to set values appropriate for our environment, before running the CIS hardening script. See this documentation for more information. See also the rest of the CIS documentation here. To keep things simple, however, we will skip that step here.

The following runs the CIS hardening script and then removes one of these rules, as it conflicts with how Azure provides provisioning information to VMs:

        {
            "type": "Shell",
            "name": "RunCIShardening - see https://ubuntu.com/security/certifications/docs/cis-compliance",
            "inline": [
                "sudo /usr/share/ubuntu-scap-security-guides/cis-hardening/Canonical_Ubuntu_18.04_CIS-harden.sh lvl1_server"
            ]
        },

        {
            "type": "Shell",
            "name": "UDFworkaroundForAzureVMbooting - UDF is required for Azure image provisioning",
            "inline": [
                "sudo rm -f /etc/modprobe.d/Canonical_Ubuntu_CIS_rule-1.1.1.7.conf"
            ]
        },

The following is a placeholder for any custom commands we want to run. We will come back to this in the next step.

        {
            "type": "Shell",
            "name": "Placeholder for custom commands required in each Ubuntu VM",
            "inline": [
                "echo 'Replace me!'"
            ]
        },

The below runs some commands that deregisters the golden image from Ubuntu Pro and removes the machine-id. This ensures that VMs generated from the golden image will generate their own unique IDs.

	    {
            "type": "Shell",
            "name": "DetachUA -- images created from this will auto attach themselves with new credentials",
            "inline": [
                "sudo ua detach --assume-yes && sudo rm -rf /var/log/ubuntu-advantage.log"
            ]
     	},

        {
            "type": "Shell",
            "name": "Replace /etc/machine-id with empty file to ensure UA client does not see clones as duplicates",
            "inline": [
                "sudo rm -f /etc/machine-id && sudo touch /etc/machine-id"
            ]
        }

Custom customisation: Make the VMs auto-update

As an example of how we can customise an image building script for our requirements, we will now change one of these customize steps.

Find the following section in BasicCISUbuntuPro1804SIGTemplate.json:

        {
            "type": "Shell",
            "name": "Placeholder for custom commands required in each Ubuntu VM",
            "inline": [
                "echo 'Replace me!'"
            ]
        },

Change this to the below, which enables unattended upgrades on the machine:

        {
            "type": "Shell",
            "name": "Install upgrades automatically",
            "inline": [
                "sudo apt install unattended-upgrades"
            ]
        },

We will now create the image version in the gallery.

First, we submit the image configuration to the Azure Image Builder service:

az resource create \
    --resource-group $sigResourceGroup \
    --subscription $subscriptionID \
    --properties @BasicCISUbuntuPro1804SIGTemplate.json \
    --is-full-object \
    --resource-type Microsoft.VirtualMachineImages/imageTemplates \
    -n BasicCISUbuntuPro1804SIG01

In the next step we will start the image build. This step can take many minutes (25-30 mins on my testing), as Azure will actually launch a VM and run the steps we have defined. We need to wait for this to complete before we can create a VM.

az resource invoke-action \
     --resource-group $sigResourceGroup \
     --subscription $subscriptionID \
     --resource-type  Microsoft.VirtualMachineImages/imageTemplates \
     -n BasicCISUbuntuPro1804SIG01 \
     --action Run

While we are waiting, however, we can see the logs of the AIB build process by going to the storage account inside the resource group created by AIB (ie. Azure Portal > Resource groups > [IT_ibUbuntuProGalleryRG_BasicCISUbuntuPro18_randomID > Random ID of the storage account > Containers > packerlogs > Random ID of the container > customization.log > Download.

You should be able to see traces of the CIS hardening like this:

[...]
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm: Execute rule 1.1.1.2
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm:
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm: Ensure mounting of freevxfs filesytems is disabled
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm: Execute rule 1.1.1.3
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm:
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm: Ensure mounting of jffs2 filesytems is disabled
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm: Execute rule 1.1.1.4
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm:
[4c6d22ac-7c0f-466f-a1c2-b30672274e89] PACKER OUT     azure-arm: Ensure mounting of hfs filesytems is disabled
[...]

If required: Accept Marketplace terms and retry

If you have never launched an Ubuntu Pro 18.04 LTS image before, this command may fail with something like:

Deployment failed. Correlation ID: eba17586-86f5-495a-85f9-3f9191065447. During the image build a failure has occurred. Please review the build log to identify which build/customization step failed. For more troubleshooting steps go to https://aka.ms/azvmimagebuilderts. Image build log location: https://armecifcu32rwz7jlos8zurq.blob.core.windows.net/packerlogs/bf091959-ba35-4db2-8888-c2b3179f119e/customization.log. OperationId: 1e853ba4-7d46-493f-890e-7e039fa44f9f. Use this operationId to search packer logs.

and if you look at the logs it mentions, you can see something like:

{\"code\":\"MarketplacePurchaseEligibilityFailed\",\"message\":\"Marketplace purchase eligibilty check returned errors. See inner errors for details. \",\"details\":[{\"code\":\"BadRequest\",\"message\":\"Offer with PublisherId: 'canonical', OfferId: '0001-com-ubuntu-pro-bionic' cannot be purchased due to validation errors. For more information see details. [...] 
You have not accepted the legal terms on this subscription: '[subscription ID]' for this plan. Before the subscription can be used, you need to accept the legal terms of the image. To read and accept legal terms, use the Azure CLI commands described at https://go.microsoft.com/fwlink/?[REDACTED] or the PowerShell commands available at https://go.microsoft.com/fwlink/?[REDACTED] Alternatively, deploying via the Azure portal provides a UI experience for reading and accepting the legal terms. Offer details: publisher='canonical' offer = '0001-com-ubuntu-pro-bionic', sku = 'pro-18_04-lts', Correlation Id: '7c302144-1c69-4642-be15-10e75049d0f7'.

In this case, we will need to accept the terms and retry the above command. You can see the terms on the Ubuntu Pro 18.04 Pro LTS marketplace listing here or for your specific Private Offer.

To accept, you can either launch a single VM through the Portal for a UI experience or you can accept from the commandline with:

az vm image terms accept --plan $ProPlanSku --offer $ProPlanOffer --publisher $ProPlanPublisher --subscription $subscriptionID

Then we need to re-run:

az resource invoke-action \
     --resource-group $sigResourceGroup \
     --subscription $subscriptionID \
     --resource-type  Microsoft.VirtualMachineImages/imageTemplates \
     -n BasicCISUbuntuPro1804SIG01 \
     --action Run

(If it does not work immediately, wait a few minutes and try again.)

Completing the build process

Once it has completed, it will change from “Running” to show something like the following:

{
  "endTime": "2021-11-03T17:42:21.7582048Z",
  "name": "67B28A17-AD58-4BDD-8CFB-1777906F8626",
  "startTime": "2021-11-03T17:11:38.9933333Z",
  "status": "Succeeded"
}

Create a VM from the Azure Portal

We are now going to create a VM from the Azure Portal.

Once you log in, click “Virtual machines”:

Click “Create” > “Virtual machine”

In the “Create a virtual machine” screen, you will need to click “See all images” below the “Image” drop down:

Select “Shared Images” on the left-hand side:

You should see our image in the “My Items”:

You can click this and it will select that as the image type.

Complete the remaining fields as you wish and click “Review + Create” to create a VM (or simply proceed to the next step if you prefer, as we will not be using the VM from this step):

Create a VM from the commandline

We are now going to create a VM from the Shared Image Gallery. As we created the image with plan information, we also need to specify that information when launching the instance.

If you already have an SSH key that you want to use, then use the following to launch the VM:

SSHPublicKeyPath=<path to your id_rsa.pub>

az vm create \
  --resource-group $sigResourceGroup \
  --subscription $subscriptionID \
  --name myAibGalleryVM \
  --admin-username aibuser \
  --location $location \
  --image "/subscriptions/$subscriptionID/resourceGroups/$sigResourceGroup/providers/Microsoft.Compute/galleries/$sigName/images/$imageDefName/versions/latest" \
  --ssh-key-values $SSHPublicKeyPath \
  --plan-name $ProPlanSku \
  --plan-product $ProPlanOffer \
  --public-ip-sku Standard \
  --plan-publisher $ProPlanPublisher

Alternatively, if you do not yet have SSH keys, you can use the following and replace the --ssh-key-values $SSHPublicKeyPath with --generate-ssh-keys (note that this may overwrite the ssh keypair “id_rsa” and “id_rsa.pub” under .ssh in your home directory):

az vm create \
  --resource-group $sigResourceGroup \
  --subscription $subscriptionID \
  --name myAibGalleryVM \
  --admin-username aibuser \
  --location $location \
  --image "/subscriptions/$subscriptionID/resourceGroups/$sigResourceGroup/providers/Microsoft.Compute/galleries/$sigName/images/$imageDefName/versions/latest" \
  --generate-ssh-keys \
  --plan-name $ProPlanSku \
  --plan-product $ProPlanOffer \
  --public-ip-sku Standard \
  --plan-publisher $ProPlanPublisher

When this completes, you should see something like the following:

{
  "fqdns": "",
  "id": "/subscriptions/[your subscription]/resourceGroups/ibUbuntuProGalleryRG/providers/Microsoft.Compute/virtualMachines/myAibGalleryVM",
  "location": "northeurope",
  "macAddress": "00-0D-3A-B9-32-27",
  "powerState": "VM running",
  "privateIpAddress": "10.0.0.4",
  "publicIpAddress": "20.67.164.125",
  "resourceGroup": "ibUbuntuProGalleryRG",
  "zones": ""
}

Note the publicIpAddress, which in my case is 20.67.164.125. This is what you will use to ssh into the machine in the next step.

Logging into the new VM and testing

ssh aibuser@[the IP address from above]

In my case, this is:

ssh [email protected]

It will likely say:

The authenticity of host '20.67.164.125 (20.67.164.125)' can't be established.
ECDSA key fingerprint is SHA256:hvDCR6zYnEYDhnQSdOhKZzrFQ017nH5FqPL2hty1WE0.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

To which you can respond yes.

Once you are in the VM, you can type sudo ua status --wait and check the output:

$ sudo ua status --wait
SERVICE       ENTITLED  STATUS    DESCRIPTION
cis           yes       disabled  Center for Internet Security Audit Tools
esm-apps      yes       enabled   UA Apps: Extended Security Maintenance (ESM)
esm-infra     yes       enabled   UA Infra: Extended Security Maintenance (ESM)
fips          yes       disabled  NIST-certified core packages
fips-updates  yes       disabled  NIST-certified core packages with priority security updates
livepatch     yes       n/a       Canonical Livepatch service

Enable services with: ua enable <service>

                Account: <redacted>
           Subscription: <redacted>
            Valid until: 9999-12-31 00:00:00+00:00
Technical support level: essential

We can see that this VM is attached to an Ubuntu Pro subscription and that we have great features like esm-apps and esm-infra enabled.

(If you are using a Private Offer that includes support, your Technical support level will read advanced instead of essential.)

And we can confirm that the CIS hardening is in place:

$ cat /etc/issue.net
Authorized uses only. All activity may be monitored and reported.

We can even run a CIS audit:

$ sudo ua enable cis
One moment, checking your subscription first
CIS Audit is already enabled.
See: sudo ua status
$ sudo cis-audit level1_server
Title   Ensure mounting of cramfs filesystems is disabled
Rule    xccdf_com.ubuntu.bionic.cis_rule_CIS-1.1.1.1
Result  pass

Title   Ensure mounting of freevxfs filesystems is disabled
Rule    xccdf_com.ubuntu.bionic.cis_rule_CIS-1.1.1.2
Result  pass

Title   Ensure mounting of jffs2 filesystems is disabled
Rule    xccdf_com.ubuntu.bionic.cis_rule_CIS-1.1.1.3
Result  pass

Title   Ensure mounting of hfs filesystems is disabled
Rule    xccdf_com.ubuntu.bionic.cis_rule_CIS-1.1.1.4
Result  pass
[...]

CIS audit scan completed. The scan results are available in /usr/share/ubuntu-scap-security-guides/cis-18.04-report.html report.

You can bring this onto your local machine by first changing the permissions within your ssh session:

sudo chown aibuser /usr/share/ubuntu-scap-security-guides/cis-18.04-report.html

and then using a separate terminal window to download it with scp:

$ scp [email protected]:/usr/share/ubuntu-scap-security-guides/cis-18.04-report.html .
Authorized uses only. All activity may be monitored and reported.
cis-18.04-report.html                         100%  579KB   3.1MB/s   00:00

For comparison, on an unhardened Ubuntu 18.04 VM, there are 80 CIS failures:

In our image, even without filing in the ruleset-params.conf or taking the manual CIS hardening steps set out in the documentation, the CIS audit should only show 13 failed rules:

We could add further customisations to our deployment template to improve this even further.

You should be able to see the Resource Groups that have been created as part of this tutorial by typing:

az group list --query [].name --output table --subscription $subscriptionID | grep $sigResourceGroup

In my case this returns:

ibUbuntuProGalleryRG
IT_ibUbuntuProGalleryRG_BasicCISUbuntuPro18_e52db49b-9cc0-4d52-936e-ecf570746def

Check you are happy for these to be deleted. If so, type:

az group delete --name [the name from above] --subscription $subscriptionID

for each of these Resource Groups, for example:

$ az group delete --name ibUbuntuProGalleryRG --subscription $subscriptionID
Are you sure you want to perform this operation? (y/n): y
$ az group delete --name IT_ibUbuntuProGalleryRG_BasicCISUbuntuPro18_e52db49b-9cc0-4d52-936e-ecf570746def --subscription $subscriptionID
Are you sure you want to perform this operation? (y/n): y

(You may find that deleting the first automatically deletes the second.)

Congratulations! We have created a Shared Image Gallery with a hardened Ubuntu Pro 18.04 LTS image inside, and launched and tested virtual machines created from this.

I hope that you have found this a helpful introduction to using a Shared Image Gallery with your Ubuntu Pro entitlements. If you have any questions, comments or suggestions, please do click the “Suggest changes” link and comment on the Discourse article for this tutorial.