Overview of parallel processing in Transducers.jl
Transducers.jl supports thread-based (foldxt) and process-based (foldxd) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing foldl is useful. Using multiple cores or machines for your computation is as easy as replacing foldl with foldxt or foldxd; you don't need to re-write your transducers or reducing functions.
See also:
- Parallel processing tutorial in Transducers.jl manual.
- API documentation of
foldxtandfoldxd.
Thread-based parallelism
To use multiple threads, the julia process must be started with appropriate environment variable JULIA_NUM_THREADS. See Julia manual for more information about how to enable and verify multi-threading in Julia:
Transducers.jl supports thread-based parallelism for Julia ≥ 1.0. You can use it by replacing foldl with foldxt. With Julia ≥ 1.3, Transducers.jl supports early termination to avoid unnecessary computation while guaranteeing the result to be deterministic; i.e., it does not depend on how computation tasks are scheduled.
Process-based parallelism
Transducers.jl supports process-based parallelism using Distributed.jl. You can use it by replacing foldl with foldxd. It can be used for horizontally scaling the computation. It is also useful for using external libraries that are not "thread-safe."