pipeline

Builtin index · Statistics & ML

result = pipeline(stage1, ..., finalStage)

Input: stage1, ..., finalStage · Output: result

What it is

Composes one or more stage names (strings naming functions already defined in the program) into a single unfitted pipeline spec — every stage but the last must be a function (X) -> X; the last must be (X, y) -> Model.

Overloads

One call shape — this function is not overloaded.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

centre(X) := X - mean(X)
final(X, y) := ridge_model(X, y, 0.1)

pipe = pipeline("centre", "final")
fitted = pipe.fit(X, y)
print("pipeline R^2 {fitted.score(X, y):.4f}")

pipeline R^2 0.9992

from specialized

standardize_cols(M) := (M - mean(M)) ./ std(M)
final(A, b) := ridge_model(A, b, 0.1)

pipe = pipeline("standardize_cols", "final")
fitted = pipe.fit(split.X_train, split.y_train)
print("held-out R2: {round(fitted.score(split.X_test, split.y_test), 4)}")

held-out R2: 0.991

from statistics-ml

How it works

pipeline(stage1, stage2, ..., finalStage) — composes named functions (by name, the same "string, not a value" convention pmap/spawn already use) into one unfitted spec. Every stage but the last is a plain (X) -> Xtransformed transform (a one-liner := function closing over any hyperparameters it needs, since Qu has no partial application); the last is an (X, y) -> Model estimator, e.g. ols_model/ ridge_model/a user function wrapping one with fixed hyperparameters. Not implemented: the spec's pipeline stage name = ... end pipeline block syntax — that's a real grammar change; this gets the same composability and fit/predict/score protocol without touching the parser.

From the note above this builtin in the interpreter, where the reasoning was written down as it was built.