When you use Ansible to provision hosts, running a task on a set of hosts in parallel is not a problem. You simply use the `serial` keyword in a playbook. See: http://docs.ansible.com/ansible/latest/playbooks_delegation.html#rolling-update-batch-size
What about when you want to run a task several times in parallel, either on each host, or on a single host? A common use case is when you want to provision infrastructure at a cloud provider, such as Amazon Web Services (AWS). In that case, you normally run the task on localhost and the task just calls out to the cloud provider API.
The reason you might want to do these calls in parallel is because creating cloud resources sometimes takes a long time. AWS EC2 instances and RDS instances are examples of some resources which take a long time to create. If you want to create several, you can use a loop and wait for each to finish, but that takes a long time. As a alternative, you can create them all in parallel.
Here is a example Ansible play of how you can run any Ansible task in parallel and wait for them all to finish.
---
- name: Run tasks in parallel
hosts: localhost
connection: local
gather_facts: no
tasks:
- name: Pretend to create instances
command: "sleep {{ item }}" # Instead of calling a long running operation at a cloud provider, we just sleep.
with_items:
- 6
- 8
- 7
register: _create_instances
async: 600 # Maximum runtime in seconds. Adjust as needed.
poll: 0 # Fire and continue (never poll)
- name: Wait for creation to finish
async_status:
jid: "{{ item.ansible_job_id }}"
register: _jobs
until: _jobs.finished
delay: 5 # Check every 5 seconds. Adjust as you like.
retries: 10 # Retry up to 10 times. Adjust as needed.
with_items: "{{ _create_instances.results }}"
The output will be:
PLAY [Run tasks in parallel] **********************************************************************************************************
TASK [Pretend to create instances] ****************************************************************************************************
changed: [localhost] => (item=6)
changed: [localhost] => (item=8)
changed: [localhost] => (item=7)
TASK [Wait for creation to finish] ****************************************************************************************************
FAILED - RETRYING: Wait for creation to finish (10 retries left).
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'957883465297.7487', 'item': 6, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/home/max/.ansible_async/957883465297.7487'})
FAILED - RETRYING: Wait for creation to finish (10 retries left).
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'450659566541.7513', 'item': 8, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/home/max/.ansible_async/450659566541.7513'})
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'763185500456.7538', 'item': 7, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/home/max/.ansible_async/763185500456.7538'})
PLAY RECAP ****************************************************************************************************************************
localhost : ok=3 changed=3 unreachable=0 failed=0
See the example code at GitHub.
One of the plays runs the tasks in parallel, and one runs then in sequence, for comparison.
Thanks Max. Needed exactly something like this. Small world!
That’s great John! Glad it is useful.
This saves the output to a file. When looping through 100 or so items how do I get that output back into a variable easily?
Maybe I am misunderstanding something Austin.
The output is registered in the _create_instances variable and used in the statement below.
Thanks Max,
I was looking exactly the same, your idea worked for me, awsome đ
previously tooked for me to deploy some 15 VMs around 30 minutes, now it tooks 3 to 4 minutes.
Thanks Max!
My scenario is a bit more compilcated, I have multiple tasks like creating instance A, B, C and then last task to monitor job completion. Do you think the below tasks will work?
tasks:
– name: Pretend to create instances
command: “sleep {{ item }}”
with_items:
– 6
– 8
– 7
register: _create_instances
async: 600
poll: 0
– name: copy files
copy:
src: “{{item}}”
dest: “/mypath/{{item}}”
with_items:
– 1
– 2
– 3
register: _copy_instances
async: 600
poll: 0
– name: download files
get_url:
url: “{{item}}”
dest: “/mypath/{{item}}”
with_items:
– A
– B
– C
register: _download_instances
async: 600
poll: 0
– name: Wait for tasks to finish
async_status:
jid: “{{ item.ansible_job_id }}”
register: _jobs
until: _jobs.finished
delay: 5
retries: 10
with_items:
– “{{ _create_instances.results }}”
– “{{ _copy_instances.results }}”
– “{{ _download_instances.results }}”
Don’t know. It was a while since I worked with that. Let us know how it works out!
Is there any trick to hide these “FAILED – RETRYING: …” lines from the output?
Sorry, no, not that I know of. At least no nice way.
How to run run multiple playbooks at same time ? Any advice please ?
Regards,
Prathap
Don’t know, but I would look at tools outside of Ansible itself.
Maybe you can try the GNU parallel command?
https://www.gnu.org/software/parallel/parallel_tutorial.html
any idea how I can deploy multiple ec2 instances in parallel?
my pb
—
– include_vars: vars/main.yml
– name: create the ec2 instance
ec2:
assign_public_ip: no
group_id: ‘{{ deploy_env.sg_group }}’
instance_type: “{{ deploy_env.instance_type }}”
image: “{{ deploy_env.image }}”
wait: true
wait_timeout: 600
count: 1
region: “{{ deploy_env.region }}”
vpc_subnet_id: “{{ deploy_env.vpc_subnet_id}}”
private_ip: “{{ item.ip }}”
instance_tags:
Name: “{{ item.name }}”
aws_access_key: “{{ aws_access_key_id }}”
aws_secret_key: “{{ aws_secret_access_key }}”
register: ec2
loop: “{{ instances }}”
Hi:
I have an use case exact to the one in the example, but I am needing one enhancement that I cannot figure out how to implement: retry a given number of times each of the parallel task that failed on the first attempt.
I know I can use âretries: nâ for single ansible tasks.
But I wonder if it is possible to combine both functionalities! I.e: run async tasks in parallel, and retry each individual one if failed.
Thanks!!!
Sorry for the late reply Daniel. I don’t know! Havn’t worked with Ansible for a while now. Maybe you figured out a way?