xmake-async-jobs

Async Jobs, Parallelism & Scheduling

Xmake's Lua layer has a cooperative coroutine scheduler underneath. Any on_build/on_build_files/on_install hook you write runs inside it, and can spawn parallel subjobs via async.runjobs (with three different input shapes) or by yielding explicitly via the scheduler.

Use this skill when writing a rule or task that needs to do work in parallel — file generation, batch downloads, distributed jobs, etc.

1. async.runjobs — the main entry point

import("async.runjobs")

runjobs("my-jobs", function (index, total, opt)
    print("job %d/%d", index, total)
    os.sleep(500)
end, {
    total = 100,
    comax = 6
})

Parameters:

• name — used in scheduler output / timing.
• jobs — can be a function (called total times), a jobpool, or a jobgraph.
• Options:
  • total — required when jobs is a function.
  • comax — max concurrent coroutines (default 4).
  • timeout — ms for on_timer callback (default 500).
  • on_timer = function(indices) — periodic callback with running job indices.
  • on_exit = function(abort_errors) — called after all jobs finish or error.
  • waiting_indicator = true — simple spinner; or {chars = {'/', '-', '\\', '|'}}.
  • curdir — set cwd for each job.
  • isolate = true — isolate coroutine environments.
  • distcc = client — farm out to a distcc client.
  • remote_only = true — force remote execution.

Example with timer:

runjobs("download", function (i, total, opt)
    os.vrunv("curl", {"-O", urls[i]})
end, {
    total = #urls,
    comax = 8,
    timeout = 1000,
    on_timer = function (indices)
        utils.vprint("still running: %s", table.concat(indices, ","))
    end
})

2. async.jobgraph — DAG-ordered jobs

When jobs have dependencies (A must finish before B), build a job graph:

import("async.jobgraph")
import("async.runjobs")

local jobs = jobgraph.new()

local function jobfunc(i, total, opt)
    print("job %s (%d/%d)", opt.progress:percent(), i, total)
end

jobs:add("root", jobfunc)
for i = 1, 3 do
    jobs:add("mid/" .. i, jobfunc)
    jobs:add_orders("mid/" .. i, "root")       -- mid/i depends on root
    for j = 1, 5 do
        local leaf = "leaf/" .. i .. "/" .. j
        jobs:add(leaf, jobfunc)
        jobs:add_orders(leaf, "mid/" .. i)
    end
end

runjobs("build", jobs, {comax = 4})