How to use Spot Virtual Machines?

What are SPOTS?

SPOTS are cloud computing resources (virtual machines) available at discount prices. They run using unused provider capacity and may be stopped when the demand for standard-priced resources increases.

CloudFerro implementation of SPOT instances

Typical SPOTS implementations provide floating prices depending on resources availability. CREODIAS cloud offers a simplified SPOTs pricing model. SPOT instances are available for high-memory, GPU-equipped and ARM CPU-based instances. The prices are flat and not related to changes in availability of resources.

At the time of writing this article, SPOT instances prices were:

2 times cheaper for GPU-equipped instances
3 times cheaper for high-memory instances
1.8 times cheaper for ARM-based instances

If cloud resources are available, SPOT instances may be launched.

The general documentation about SPOT instances usage is “Spot instances on CREODIAS” https://creodias.docs.cloudferro.com/en/latest/cloud/Spot-instances-on-Creodias.html. It covers important technical details. However, it does not provide more hints on when and how to use SPOT instances effectively.

Usage examples

SPOT instances are a great choice for:

Stateless services.
Short life-span entities.
Workloads that can quickly dump current data and status when an interruption signal comes.
Jobs that can be divided into smaller steps executed separately.
Or any combination of the above.

More specific examples are described below.

Test environments with short lifetimes

QA teams often create test environments using IaaS tools, automatically deploy software to be tested execute manual or automated tests, and finally destroy the complete environment. Such test environments have short lifetimes measured in minutes or, at most, hours.

The situation where such tests might be sometimes cancelled due to instance eviction can be acceptable when tests environments costs significantly less.

Additionally, the environment provisioner can be configured so that if SPOTs are not available, on-demand instances will be used for this single test occurrence.

Web services

Most web services can be based on stateless services.

However, you need to plan resources for stable service operation for low and medium load as in case of load spikes. You can configure auto-scaling tools to run on demand instances for low and medium load and attempt running the services necessary for spike handling using SPOT instances if they are available. The cost of spike handling is significantly lower. Since service instances dedicated to handling load spikes typically have a short lifespan, the chance of eviction is relatively low.

Another scenario in this area is to go further with this economical approach and configure scaling in a way that on-demand instances are used only for a low constant load range. All instances required to fulfill medium and high load demand would be run on SPOT instances. However, with this approach monitoring and eventually replacing evicted SPOTS instances with on-demand instances would be crucial to assure your service quality.

Batch processing

You can use SPOT instances as processing workers if your batch process can be divided into smaller tasks or steps, and the processing workflow is configured in such a way that the entire workflow does not pause or stop in case of stopping or failure of a single step.

The general architecture for such a solution should consist of:

Persistent storage
This will be the place for all input data, temporary results saved after each calculation step and results. Technically it can be:
- An on-demand instance with:
  - An NFS server with block storage volume attached.
  - A dedicated service or database.
- A bucket in object storage.
Workers on SPOT instances.
They will process the work in possibly small sub-batches saving output to persistent storage.
Persistent VM acting as a control node or other work scheduling/coordination solution.
All workers should check the status of tasks to be executed here and report which tasks they are taking to avoid repeated tasks or worse, falling into a deadlock.
This part is not a must. You may design the entire process and prepare workers software in such a way that coordination is performed by them - usually by saving work status on storage. However, a dedicated coordination node would contain additional logic and simplify monitoring status and progress.

SPOTS Implementation tips

Resource availability and allocation order

As mentioned at the beginning of this article, SPOT instances may be provisioned if cloud contains enough resources for the selected flavor.
If many users started allocating SPOT resources at the same time, all of them would be treated with the same priority. Instances would be created in an order that requests to the OpenStack API are received.

Speed up instances provisioning by launching them from dedicated image

In the case of SPOT instances, fast VM activation is needed. The best practice to achieve this is to

Install all needed software on a standard VM with the same or smaller flavor than the expected SPOT flavor.
Test it.
Create a snapshot image from the VM.
Use this image as the source of final SPOT instances.

If you do not have much experience with virtual machine images maintenance, you may find it beneficial to learn from available resources and develop your own procedure and tools. I strongly recommend using automation scripts, dedicated tools such as OpenStack Diskimage-Builder, or automation tools like Ansible. Infrastructure as a code tools, as for example Terraform, should be considered.

Below, you will find a simple example of how you can prepare such image using a user-data option when creating the instance.

Please create a new VM using the following command:

openstack server create --image MY_WORKER_IMAGE --flavor spot.hma.large --user-data MY_INSTALL_SCRIPT.sh VM_INSTANCE_NAME

Where MY_INSTALL_SCRIPT.sh contains for example:

#!/bin/sh
apt-get update
apt-get install -y python3 python3-virtualenv
sudo -u eouser bash -c \
'cd /home/eouser && \
virtualenv app_venv && \
source ./app_venv/bin/activate && \
git clone https://github.com/hossainchisty/Photo-Album-App.git && \
cd Photo-Album-App && \
pip install -r requirements.txt'

A VM created in this way will have:

Python installed.
An example application.
Created virtual environment.
All application requirements installed within the virtual environment.

You can then create an image from this machine by executing the following commands:

openstack server stop VM_INSTANCE_NAME 
openstack server image create VM_INSTANCE_NAME --name IMAGE_NAME

The image IMAGE_NAME may be used as a source to create SPOT instances with the same or higher resource requirement flavor.

Reacting on an instance termination attempt

The documentation “Spot instances on CREODIAS” https://creodias.docs.cloudferro.com/en/latest/cloud/Spot-instances-on-Creodias.html mentions that instances can be tagged during their creation or later with callback_url:<url value> tag.
This allows us to react when the OpenStack scheduler tries to delete our SPOT instance.

The first reaction option is to convert the instance to an on-demand instance. You need to prepare a persistent service able to receive message about the attempt to delete the instance. Then include the dedicated endpoint’s full address in the tag. When the service receives the instance UIID in message, it should execute script containing the following OpenStack client commands:

openstack server resize --flavor NON_SPOT_FLAVOR VM_INSTANCE_UIID

When output of command:

openstack server list -f value -c ID -c Status | grep VM_INSTANCE_UIID

shows the status VM_INSTANCE_UIID VERIFY_RESIZE,
then the resize may be confirmed with a command:

openstack server resize confirm VM_INSTANCE_UIID

However, we need to consider the following:

The resize process forces instance to reboot, so if your workload is not prepared, you may lose your data.
It is not possible to resize this instance back to a SPOT flavor.

Manual resizing from a SPOT flavor to on-demand is also possible from the Horizon GUI, but it should be treated as a planned preventive action because humans usually would not have any time to react with the resize after receiving the notification.

If your processing software can save the current state or temporary result, the second reaction option is to place the service on the instance and activate it on a system boot. Then, you need to tag the instance with its own IP address in URL in “callback_url” tag.

When the service on the instance receives a message with information about the instance delete attempt, it may trigger the interruption of the workload and save the current state or temporary result on the persistent storage. In this way, the job can be continued by another still active SPOT instance or on an on-demand instance (already active or provisioned if selected number of workers is requested).

Flexible SPOT instances

When creating SPOT instances, you may need to hardcode some configuration data, such as the addresses of input data servers or storage destinations for the results. This can be done using the solution provided in the previous section. However, if you wish to run many different projects, customizing the instance becomes necessary.

A simple functional solution is to:

Prepare your software to use configuration provided in environment variables
Inject those variables values thru “cloud-init” when the instance is created

For example, you can create an instance in this way:

openstack server create --image MY_WORKER_IMAGE --flavor spot.hma.large --user-data MY_CONFIG_FILE.yaml VM_INSTANCE_NAME

If your program retrieves data and saves results to REST API endpoints, MY_CONFIG_FILE.yaml could contain:

#cloud-config
runcmd:
- echo "INPUT_DATA_API='https://192.168.1.11/input/'" >> /etc/profile
- echo "OUTPUT_DATA_API='https://192.168.1.11/output/'" >> /etc/profile
- echo "DATA_USER='username'" >> /etc/profile

This file can be generated from a template manually or by a tool/script.
If you are using Terraform, the configuration can be applied using Terraform variables in this way:

#cloud-config
runcmd:
- echo "INPUT_DATA_API='${var.input_data_api}'" >> /etc/profile
- echo "OUTPUT_DATA_API='${var.input_data_api}'" >> /etc/profile
- echo "DATA_USER='username'" >> /etc/profile

Releasing SPOT instances resources

Using SPOTs is related to the demand for lowering the costs.
If you wish to release resources when the workload is done to minimize the costs even further, then you can consider the following:

Install and configure a load monitoring tool on the image used to create instances. Check periodically for idle instances, then manually delete idle instances.
Design and develop notifications about finished tasks into your workload software. Then manually delete the idle instance after receiving a notification.
Upgrade the previous solution with notifications sent to a software agent working on the persistent VM which can automatically delete idle instances. This solution would be beneficial when the savings from deleting idle instances exceed the cost of maintaining a persistent VM based on a cheap, low-resource flavor.
Finally, you can build the capability for our software to self-destruct the instance. This can be done in this manner:
- Configure the image with an OpenStack client and credentials.
To do this, follow the steps provided in the documents:
“How to install OpenStackClient for Linux on CREODIAS”
https://creodias.docs.cloudferro.com/en/latest/openstackcli/How-to-install-OpenStackClient-for-Linux-on-Creodias.html
and
“How to generate or use Application Credentials via CLI on CREODIAS”
https://creodias.docs.cloudferro.com/en/latest/cloud/How-to-generate-or-use-Application-Credentials-via-CLI-on-Creodias.html
and information given in the previous chapter “Speed up instances provisioning by launching them from dedicated image”
- Building into your application procedure accessing "http://169.254.169.254/openstack/latest/meta_data.json" to retrieve the instance UUID and then executing self-destruct with command:

openstack server delete INSTANCE_UUID

Kubernetes Autoscaling

We can avoid many of the previous traps by moving to the next level of abstraction: using a Kubernetes cluster created with OpenStack Magnum, featuring autoscaling on SPOT instances. With this solution, SPOT instances will be used to create Kubernetes cluster nodes when necessary. When the demand for resources decreases, nodes will be automatically removed after a configured autoscaling delay.

As autoscaling and eventual node eviction will be performed on the separated and dedicated node group, then core services on the default-workers node group will not be affected. This solution is cost-effective if the cost of maintaining a working cluster is lower than keeping idle SPOT instances. You need to:

Create a production cluster with 3 control nodes and a minimum one work node. More details can be found in one of the CREODIAS mini-guides.
Read documentation about node groups https://creodias.docs.cloudferro.com/en/latest/kubernetes/Creating-Additional-Nodegroups-in-Kubernetes-Cluster-on-Creodias-OpenStack-Magnum.html
Create a new node-group using one of spot.* flavors.
For example:

openstack coe nodegroup create --min-nodes 1 --max-nodes 3 --node-count 1 --flavor spot.hma.medium --role worker CLUSTER_NAME spot-workers

This command crates a node group named “spot-workers” with a single node activated just after the group creation, and maximum 3 nodes.
Note that explicit definition of all options from the command above is necessary to activate autoscaling on the new group.

Define the affinity of all your persistent services to the node group default-worker.
Follow the example of Kubernetes deployment definition:

apiVersion: v1
kind: Pod
metadata:
name: ubuntu-pod
spec:
affinity:
  nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: magnum.openstack.org/nodegroup
      operator: In
      values:
      - default-worker
containers:
- name: ubuntu
  image: ubuntu:latest
  command: ["sleep", "3600"]

Define the affinity of worker pods to the node group based on spot instance nodes. Follow the example of Kubernetes deployment definition. Remember to replace “spot-workers” name with your node-group name if you use another one:

apiVersion: apps/v1
kind: Deployment
metadata:
name: counter-log-deployment
labels:
  app: counter-log
spec:
replicas: 4
selector:
  matchLabels:
  app: counter-log
template:
  metadata:
  labels:
    app: counter-log
  spec:
    affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: magnum.openstack.org/nodegroup
        operator: In
        values:
        - spot-workers
  containers:
  - name: counter-log
    image: busybox:1.28
    args: [/bin/sh, -c, 'i=0; while true; do echo "$i: $(date) $(hostname) $(uuidgen)-$(uuidgen)"; i=$((i+1)); sleep 1; done']
    resources:
    requests:
      cpu: 200m

Final Thoughts

Using SPOT instances certainly brings significant cost savings, but it also provides several other benefits:

You gain a better understanding of how cloud virtual machine instances work and how to efficiently create, configure, and initialize them.
IaaC. If you do not use this already, with staring using SPOTS you have opportunity to learn and start to use Infrastructure as a code tools as Terraform.
Failure (SPOT eviction) is not an accidental behavior but a feature. Fault tolerance must be a core part of the system design.
In the case of long-running processes:
- You need to better understand the workflow to split it into smaller chunks that can be processed by SPOT instances. This may provide an opportunity to enable or improve parallel data processing.
- Data structures for checkpoints and temporary data should be analyzed, which may lead to data size optimization.
Adding components dedicated to coordination gives an opportunity to implement detailed monitoring of the entire system’s processing status and progress.

Author: Mateusz Ślaski, Sales Support Engineer at CloudFerro