How to run Ansible tasks in parallel

When you use Ansible to provision hosts, running a task on a set of hosts in parallel is not a problem. You simply use the `serial` keyword in a playbook. See: http://docs.ansible.com/ansible/latest/playbooks_delegation.html#rolling-update-batch-size

What about when you want to run a task several times in parallel, either on each host, or on a single host? A common use case is when you want to provision infrastructure at a cloud provider, such as Amazon Web Services (AWS). In that case, you normally run the task on localhost and the task just calls out to the cloud provider API.

The reason you might want to do these calls in parallel is because creating cloud resources sometimes takes a long time. AWS EC2 instances and RDS instances are examples of some resources which take a long time to create. If you want to create several, you can use a loop and wait for each to finish, but that takes a long time. As a alternative, you can create them all in parallel.

Here is a example Ansible play of how you can run any Ansible task in parallel and wait for them all to finish.

---

- name: Run tasks in parallel
  hosts: localhost
  connection: local
  gather_facts: no
  tasks:
    - name: Pretend to create instances
      command: "sleep {{ item }}"  # Instead of calling a long running operation at a cloud provider, we just sleep.
      with_items:
        - 6
        - 8
        - 7
      register: _create_instances
      async: 600  # Maximum runtime in seconds. Adjust as needed.
      poll: 0  # Fire and continue (never poll)

    - name: Wait for creation to finish
      async_status:
        jid: "{{ item.ansible_job_id }}"
      register: _jobs
      until: _jobs.finished
      delay: 5  # Check every 5 seconds. Adjust as you like.
      retries: 10  # Retry up to 10 times. Adjust as needed.
      with_items: "{{ _create_instances.results }}"

The output will be:

PLAY [Run tasks in parallel] **********************************************************************************************************

TASK [Pretend to create instances] ****************************************************************************************************
changed: [localhost] => (item=6)
changed: [localhost] => (item=8)
changed: [localhost] => (item=7)

TASK [Wait for creation to finish] ****************************************************************************************************
FAILED - RETRYING: Wait for creation to finish (10 retries left).
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'957883465297.7487', 'item': 6, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/home/max/.ansible_async/957883465297.7487'})
FAILED - RETRYING: Wait for creation to finish (10 retries left).
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'450659566541.7513', 'item': 8, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/home/max/.ansible_async/450659566541.7513'})
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_no_log': False, u'ansible_job_id': u'763185500456.7538', 'item': 7, u'started': 1, 'changed': True, 'failed': False, u'finished': 0, u'results_file': u'/home/max/.ansible_async/763185500456.7538'})



PLAY RECAP ****************************************************************************************************************************
localhost                  : ok=3    changed=3    unreachable=0    failed=0  

See the example code at GitHub.
One of the plays runs the tasks in parallel, and one runs then in sequence, for comparison.

12 responses on “How to run Ansible tasks in parallel

  1. This saves the output to a file. When looping through 100 or so items how do I get that output back into a variable easily?

    1. Maybe I am misunderstanding something Austin.

      The output is registered in the _create_instances variable and used in the statement below.

  2. Thanks Max,
    I was looking exactly the same, your idea worked for me, awsome 🙂
    previously tooked for me to deploy some 15 VMs around 30 minutes, now it tooks 3 to 4 minutes.

  3. Thanks Max!
    My scenario is a bit more compilcated, I have multiple tasks like creating instance A, B, C and then last task to monitor job completion. Do you think the below tasks will work?
    tasks:
    – name: Pretend to create instances
    command: “sleep {{ item }}”
    with_items:
    – 6
    – 8
    – 7
    register: _create_instances
    async: 600
    poll: 0

    – name: copy files
    copy:
    src: “{{item}}”
    dest: “/mypath/{{item}}”
    with_items:
    – 1
    – 2
    – 3
    register: _copy_instances
    async: 600
    poll: 0

    – name: download files
    get_url:
    url: “{{item}}”
    dest: “/mypath/{{item}}”
    with_items:
    – A
    – B
    – C
    register: _download_instances
    async: 600
    poll: 0

    – name: Wait for tasks to finish
    async_status:
    jid: “{{ item.ansible_job_id }}”
    register: _jobs
    until: _jobs.finished
    delay: 5
    retries: 10
    with_items:
    – “{{ _create_instances.results }}”
    – “{{ _copy_instances.results }}”
    – “{{ _download_instances.results }}”

  4. Is there any trick to hide these “FAILED – RETRYING: …” lines from the output?

  5. How to run run multiple playbooks at same time ? Any advice please ?

    Regards,
    Prathap

  6. any idea how I can deploy multiple ec2 instances in parallel?

    my pb


    – include_vars: vars/main.yml
    – name: create the ec2 instance
    ec2:
    assign_public_ip: no
    group_id: ‘{{ deploy_env.sg_group }}’
    instance_type: “{{ deploy_env.instance_type }}”
    image: “{{ deploy_env.image }}”
    wait: true
    wait_timeout: 600
    count: 1
    region: “{{ deploy_env.region }}”
    vpc_subnet_id: “{{ deploy_env.vpc_subnet_id}}”
    private_ip: “{{ item.ip }}”
    instance_tags:
    Name: “{{ item.name }}”
    aws_access_key: “{{ aws_access_key_id }}”
    aws_secret_key: “{{ aws_secret_access_key }}”
    register: ec2
    loop: “{{ instances }}”

Leave a Reply to Max Wenzin Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.