Transducer interface

Core interface for transducers

Transducers.Transducer — Type.

Transducer

The abstract type for transducers.

source

Transducers.AbstractFilter — Type.

AbstractFilter <: Transducer

The abstract type for filter-like transducers. outtype is appropriately defined for child types.

source

Transducers.R_ — Type.

Transducers.R_{X}

When defining a transducer type X, it is often required to dispatch on type rf::R_{X} (Reducing Function) which bundles the current transducer xform(rf)::X and the inner reducing function inner(rf)::R_.

source

Transducers.inner — Function.

Transducers.inner(rf::R_)

Return the inner reducing function of rf.

source

Transducers.xform — Function.

Transducers.xform(rf::R_{X}) -> xf :: X

Return the transducer xf associated with rf. Returned transducer xf is "atomic"; i.e., it is not a Composition transducer type.

source

Transducers.start — Function.

Transducers.start(rf::R_{X}, state)

This is an optional interface for a transducer. Default implementation just calls start of the inner reducing function; i.e.,

start(rf::Reduction, result) = start(inner(rf), result)

If the transducer X is stateful, it can "bundle" its private state with wrap:

start(rf::R_{X}, result) = wrap(rf, PRIVATE_STATE, start(inner(rf), result))

where PRIVATE_STATE is an initial value for the private state that can be used inside next via wrapping.

See Take, PartitionBy, etc. for real-world examples.

Side notes: There is no related API in Clojure's Transducers. Transducers.jl uses it to implement stateful transducers using "pure" functions. The idea is based on a slightly different approach taken in C++ Transducer library atria.

source

Transducers.next — Function.

Transducers.next(rf::R_{X}, state, input)

This is the only required interface. It takes the following form (if start is not defined):

next(rf::R_{X}, result, input) =
    # code calling next(inner(rf), result, possibly_modified_input)

See Map, Filter, Cat, etc. for real-world examples.

source

Transducers.complete — Function.

Transducers.complete(rf::R_{X}, state)

This is an optional interface for a transducer. If transducer X has some internal state, this is the last chance to "flush" the result.

See PartitionBy, etc. for real-world examples.

If both complete(rf::R_{X}, state) and start(rf::R_{X}, state) are defined, complete must unwarp state before returning state to the outer reducing function. If complete is not defined for R_{X}, this happens automatically.

source

Transducers.outtype — Function.

outtype(xf::Transducer, intype)

Output item type for the transducer xf when the input type is intype.

source

Helpers for stateful transducers

Transducers.wrap — Function.

wrap(rf::R_{X}, state, iresult)

Pack private state for reducing function rf (or rather the transducer X) with the result iresult returned from the inner reducing function inner(rf). This packed result is typically passed to the outer reducing function.

This is intended to be used only in start. Inside next, use wrapping.

Consider a reducing step constructed as

rf = Reduction(xf₁ |> xf₂ |> xf₃, f, intype)

where each xfₙ is a stateful transducer and hence needs a private state stateₙ. Then, calling start(rf, result)) is equivalent to

wrap(rf,
     state₁,                     # private state for xf₁
     wrap(inner(rf),
          state₂,                # private state for xf₂
          wrap(inner(rf).inner,
               state₃,           # private state for xf₃
               result)))

or equivalently

result₃ = result
result₂ = wrap(inner(inner(rf)), state₃, result₃)
result₁ = wrap(inner(rf),        state₂, result₂)
result₀ = wrap(rf,               state₁, result₁)

The inner most step function receives the original result as the first argument while transducible processes such as mapfoldl only sees the outer-most "tree" result₀ during the reduction. The whole tree is unwraped during the complete phase.

See wrapping, unwrap, and start.

source

Transducers.unwrap — Function.

unwrap(rf, result)

Unwrap wraped result to a private state and inner result. Following identity holds:

unwrap(rf, wrap(rf, state, iresult)) == (state, iresult)

This is intended to be used only in complete. Inside next, use wrapping.

source

Transducers.wrapping — Function.

wrapping(f, rf, result)

Function f must take two argument state and iresult, and return a tuple (state, iresult). This is intended to be used only in next, possibly with a do block.

next(rf::R_{MyTransducer}, result, input) =
    wrapping(rf, result) do my_state, iresult
        # code calling `next(inner(rf), iresult, possibly_modified_input)`
        return my_state, iresult  # possibly modified
    end

See wrap, unwrap, and next.

source

Interface for reducibles

Transducers.__foldl__ — Function.

__foldl__(rf, init, reducible::T)

Left fold a reducible with reducing function rf and initial value init. This is primary an API for overloading when the reducible "container" or "context" (e.g., I/O stream) of type T can provide a better reduction mechanism than the default iterator-based one.

For a simple iterable type MyType, a valid implementation is:

function __foldl__(rf, val, itr::MyType)
    for x in itr
        val = next(rf, val, x)
        @return_if_reduced complete(rf, val)
    end
    return complete(rf, val)
end

although in this case default __foldl__ can handle MyType and thus there is no need for defining it. In general, defining __foldl__ is useful only when there is a better way to go over items in reducible than Base.iterate.