Process Task Runner
Available on: Open Source EditionEnterprise Edition>= 0.18.0
Run tasks as local processes.
How to use the Process task runner
Here is an example of a Shell script configured with the Process task runner which runs a Shell command as a child process in the Kestra host:
id: process_script_runner
namespace: company.team
tasks:
- id: shell
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.core.runner.Process
commands:
- echo "Hello World!"
The Process task runner doesn’t have any additional configuration beyond the type
property.
Benefits
The Process task runner is useful if you want to access local files e.g. to take advantage of locally configured software libraries and virtual environments.
Combining task runners with Worker Groups
You can combine the Process task runner with Worker Groups to run tasks on dedicated servers that might have specific software libraries or configurations. This powerful combination allows you to leverage the compute resources of your Worker Groups while running tasks as local processes without the overhead of containerization.
The example below shows how to combine the Process task runner with Worker Groups to fully leverage the GPU resources of a dedicated server:
id: python_on_gpu
namespace: company.team
tasks:
- id: gpu_intensive_ai_workload
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
commands:
- python main.py
workerGroup:
key: gpu
taskRunner:
type: io.kestra.plugin.core.runner.Process
Note that Worker Group is an Enterprise Edition functionality. If you want to try it out, please reach out.
Was this page helpful?