vxml_pipeline - v0.2.0

VXML Pipeline

Package Version Hex Docs

Run and debug VXML transformation pipelines.

The package provides two related execution layers:

The package also includes reusable desugarers, pipeline monitors, command-line argument handling, local-desugarer tooling, and a test framework for desugarers.

Add it to a Gleam project with:

gleam add vxml_pipeline

Desugarers

The atomic unit encapsulating a named VXML-to-VXML transformation is a Desugarer. It accepts one VXML tree and either returns a transformed VXML tree with warnings or returns a DesugaringError:

pub type DesugarerTransform =
  fn(VXML) ->
    Result(#(VXML, List(DesugaringWarning)), DesugaringError)

pub type Desugarer {
  Desugarer(
    name: String,
    ...
    transform: DesugarerTransform,
  )
}

The generated vxml_pipeline/desugarers module exports the reusable desugarer constructors:

import vxml_pipeline/desugarers as ds

let pipeline = [
  ds.rename(#("Chapter", "section")),
  ds.append_class(#("section", "chapter")),
]

Each constructor validates or prepares its parameters when it creates the Desugarer. The resulting transform is then ready to be applied repeatedly.

Authoring a desugarer

Client-defined desugarers are ordinary Gleam modules. Their constructors accept a public-facing Param, prepare it once as a private InnerParam, and capture the prepared value in a reusable transform. Preparation is the place for validation and one-time work such as compiling a regular expression.

The vxml_pipeline/authoring module packages this preparation and transform construction into a Desugarer:

import gleam/list
import gleam/regexp.{type Regexp}
import vxml.{Line, T}
import vxml/blame as blame
import vxml_pipeline/authoring
import vxml_pipeline/core.{
  type Desugarer, type DesugarerTransform, type DesugaringError,
  DesugaringError,
}
import vxml_pipeline/nodemaps_2_transform as n2t
import vxml_pipeline/testing

pub const name = "redact_matching_lines"

pub fn constructor(param: Param) -> Desugarer {
  authoring.desugarer(
    name: name,
    param: param,
    prepare: param_to_inner_param,
    transform: inner_param_to_transform,
  )
}

type Param = String

type InnerParam {
  InnerParam(pattern: Regexp)
}

fn param_to_inner_param(param: Param) -> Result(InnerParam, DesugaringError) {
  case regexp.from_string(param) {
    Ok(pattern) -> Ok(InnerParam(pattern))
    Error(_) ->
      Error(DesugaringError(
        blame.no_blame,
        "Invalid regular expression: " <> param,
      ))
  }
}

fn inner_param_to_transform(inner: InnerParam) -> DesugarerTransform {
  let nodemap: n2t.OneToOneNoErrorNodemap = fn(vxml) {
    case vxml {
      T(_, lines) ->
        T(
          ..vxml,
          lines: list.map(lines, fn(line) {
            Line(
              ..line,
              content: case regexp.check(inner.pattern, line.content) {
                True -> "[redacted]"
                False -> line.content
              },
            )
          }),
        )
      _ -> vxml
    }
  }

  n2t.one_to_one_no_error_nodemap_2_desugarer_transform(nodemap)
}

By convention, Param is a structural type composed from tuples, lists, options, strings, integers, and booleans. A pipeline can therefore call the constructor without importing a desugarer-specific record. InnerParam is private and may use named records, compiled values, or any other representation that makes the implementation clear and efficient.

authoring also provides constructors for infallible, no-parameter, and outside-aware desugarers. authoring.blame(name, line_no) creates provenance for VXML introduced by a desugarer.

Most reusable desugarers define a typed operation on one node and use vxml_pipeline/nodemaps_2_transform to walk the complete tree. A desugarer may instead implement DesugarerTransform directly when its operation requires different traversal behavior.

The repository’s desugarer conventions are recorded in the desugarer style guide. Desugarer testing and application-local desugarer registries are described later in this README.

Pipelines and run_pipeline

A Pipeline is an ordered list of desugarers:

pub type Pipeline =
  List(Desugarer)

run_pipeline applies each desugarer to an existing VXML tree. It does not assemble input, parse a source format, split output, emit another format, or write files. This makes it suitable for standalone use when an application already owns those operations.

The function requires:

import vxml_pipeline as vp
import vxml_pipeline/core
import vxml_pipeline/desugarers as ds

let pipeline: core.Pipeline = [
  ds.rename(#("Chapter", "section")),
]

let result =
  vp.run_pipeline(
    vxml,
    pipeline,
    [],    // monitors
    vp.PipelineUXOptions(
      monitor_interactive_mode: False,
      report_long_running_desugarers: True,
      feedback_margin: 0,
    ),
  )

On success, run_pipeline returns the transformed VXML, accumulated warnings, and one duration per desugaring step, in pipeline order. It returns a PipelineExecutionError when a desugarer fails, a monitor stops execution, or the user exits an interactive run. Timing is part of pipeline execution; monitors do not measure desugarer durations.

No renderer or command-line setup is required for standalone execution. A Gleam program can construct or parse a VXML value, build a pipeline, call run_pipeline, and handle the returned Result directly.

Renderers

A Renderer places a pipeline inside a complete file-oriented process. It coordinates application-provided stages for obtaining VXML and turning the final VXML into output files.

Data flow

The complete renderer has the following stages:

input path
  -> Assembler
  -> List(InputLine)
  -> Parser
  -> VXML
  -> Filterer
  -> VXML
  -> Pipeline
  -> VXML
  -> Splitter
  -> List(OutputFragment(classifier, VXML))
  -> Emitter
  -> List(OutputFragment(classifier, List(OutputLine)))
  -> Writer
  -> output files
  -> optional Prettifier

Applications provide the stages that are specific to their input format and output format. The package supplies defaults for common single-file XML, Writerly, HTML, JSX, file-writing, and Prettier workflows.

Building a renderer

The following is a complete single-file XML-to-JSX renderer of the same shape as test/renderer_integration_test.gleam:

import vxml_pipeline as vp
import vxml_pipeline/core
import vxml_pipeline/desugarers as ds

fn pipeline() -> core.Pipeline {
  [
    ds.rename(#("Chapter", "section")),
    ds.append_class(#("section", "chapter")),
  ]
}

pub fn run() {
  let options = vp.vanilla_options()

  let parameters =
    vp.RendererParameters(
      input_dir: "input.xml",
      output_dir: "build/output",
      prettifier_behavior: vp.PrettifierOff,
    )

  let renderer =
    vp.Renderer(
      assembler: vp.default_file_assembler,
      parser: vp.default_xml_parser,
      filterer: vp.default_filterer(_, options, []),
      pipeline: pipeline(),
      splitter: vp.stub_splitter(".tsx"),
      emitter: vp.stub_jsx_emitter,
      writer: vp.default_writer,
      prettifier: vp.empty_prettifier,
    )

  vp.run_renderer(renderer, parameters, options)
}

run_renderer returns the relative paths written during the run. Its error type retains the error type of each application-provided stage. It prints its own progress block and terminates that block with one blank line.

The default writer creates missing parent directories and overwrites output files. Prettifier modes are:

Writerly input

Writerly support is kept in a separate adapter module because the renderer core operates on InputLine, OutputLine, and VXML rather than on its syntax directly:

import vxml_pipeline/writerly_defaults as wd

wd.default_writerly_assembler
wd.default_writerly_parser
wd.default_writerly_emitter

The assembler accepts either a file or a directory tree. It reports the assembled directory tree as verbose feedback and respects the renderer’s path-selection options.

Feedback

Assemblers, parsers, filterers, splitters, emitters, and writers return their payload together with Feedback:

pub type Feedback {
  NoFeedback
  SomeFeedback(List(FeedbackBlock))
}

pub type FeedbackBlock {
  FeedbackBlock(lines: List(String), margin: FeedbackMargin)
}

pub type FeedbackMargin {
  AtRunnerMargin
  Verbatim
}

Stage implementations construct their own feedback. The renderer prints it when verbose output is enabled. AtRunnerMargin places the block at the renderer runner’s margin; Verbatim leaves its lines unindented.

Monitors

A monitor observes the initial VXML and the result of every desugaring step. It may retain state, produce feedback, or stop the pipeline with an error:

vp.new_monitor(
  "example",
  initial_state,
  fn(vxml, state, context) {
    // context contains the step number and adjacent desugarers
    Ok(#(next_state, vp.NoFeedback))
  },
)

PipelineStepContext contains:

A monitor error becomes a PipelineMonitorError and identifies the monitor, step number, and message. Monitor feedback is sent to the pipeline runner as discrete feedback blocks. Interactive mode pauses once for each feedback block.

The built-in --track, --dump, and --validate-vxml facilities are implemented as monitors. Applications can install VXML validation directly:

vp.vxml_validation_monitor(
  False, // do not warn about boundary whitespace in attribute values
)

Invalid VXML stops the pipeline at the first invalid state. Passing True also reports leading or trailing attribute-value whitespace as monitor feedback; that whitespace remains valid VXML.

empty_text_node_monitor() is a narrower monitor that rejects only text nodes with no lines. It does not validate tags, attributes, or individual line contents. The corresponding command-line form is --validate-vxml -lines-only. It cannot be combined with -warn-attribute-whitespace.

Renderer parameters and options

RendererParameters contains values required for a run:

vp.RendererParameters(
  input_dir: "src/content",
  output_dir: "build/output",
  prettifier_behavior: vp.PrettifierOff,
)

RendererOptions controls optional behavior. Start from vanilla_options() and update only the relevant fields:

let options =
  vp.RendererOptions(
    ..vp.vanilla_options(),
    verbose: True,
    artifacts: True,
    warnings: True,
  )

Options include filtering selectors, monitor configuration, tracking and dump factories, timing-table configuration, warning and artifact reporting, long-running-step reports, output-table widths, and stage dumps.

Command-line integration

The package can parse its standard renderer options together with a declared set of application-specific options. Application-specific values are stored in ParsedCLIArguments.user_args; the package groups them but does not interpret them.

A typical entry point has this shape:

import argv
import vxml_pipeline as vp
import gleam/io
import local_desugarers
import on

pub fn main() {
  io.println("")

  let args = argv.load().arguments

  use args <- on.error_ok(
    vp.read_from_dot_last_command(args),
    handle_cli_error,
  )

  use arguments <- on.error_ok(
    vp.process_command_line_arguments(args, ["--local-option"]),
    handle_cli_error,
  )

  use help_requested <- on.error_ok(
    vp.handle_help_requests(arguments, local_cli_usage),
    handle_cli_error,
  )

  use maintenance_requested <- on.error_ok(
    vp.handle_maintenance_requests(
      arguments,
      local_desugarers.assertive_tests,
    ),
    handle_cli_error,
  )

  use _ <- on.stay(case help_requested || maintenance_requested {
    True -> on.Return(Nil)
    False -> on.Stay(Nil)
  })

  use _ <- on.error_ok(
    vp.write_to_dot_last_command(args),
    handle_cli_error,
  )

  // Construct and run the application renderer here.
}

The application can apply parsed standard values to its defaults with:

vp.amend_renderer_parameters_by_arguments(parameters, arguments)
vp.amend_renderer_options_by_arguments(options, arguments)

CLIError distinguishes argument parsing, .last-command reading, decoding and writing, maintenance operations, and application-defined failures. Applications can wrap a local failure with ClientSideError and use cli_error_message for a common presentation path.

.last-command

read_from_dot_last_command replaces the argument list only when it is exactly ["--last-command"]. If --last-command appears with another argument, command-line parsing rejects it. write_to_dot_last_command stores an unambiguous encoding of the effective argument list in .last-command.

The read and write functions use the current working directory. Applications normally write only a command that proceeds to ordinary rendering; help, maintenance, and local one-off operations can return before the write.

Standard diagnostic options

The built-in help describes all accepted forms. The principal options are:

Use --track-help for the selector-window, step-range, and output-formatting syntax accepted by --track.

Testing desugarers

vxml_pipeline/testing provides VXML-to-VXML test data and collection helpers:

pub fn assertive_tests() {
  testing.collection(
    name: name,
    data: [
      testing.data(
        param: "password|secret",
        source: "
          <> document
            <>
              'ordinary line'
              'the password is swordfish'
        ",
        expected: "
          <> document
            <>
              'ordinary line'
              '[redacted]'
        ",
      ),
    ],
    constructor: constructor,
  )
}

The test machinery parses the source VXML, applies the desugarer, serializes the result, and compares it with the expected VXML.

Run the complete package test suite, including the renderer integration test and generated desugarer tests, with:

gleam test

Run only the renderer integration test with gleam run -m renderer_integration_test.

From a checkout of this repository, run the package’s generated desugarer test registry with:

gleam run -m desugarer_tests
gleam run -m desugarer_tests -- rename

Local desugarers in an application

An application can keep private desugarers in this fixed layout:

src/
  desugarers/
    example.gleam
  local_desugarers.gleam
  local_desugarer_tests.gleam

Generate src/local_desugarers.gleam from src/desugarers/:

gleam run -m vxml_pipeline/generate_local_desugarers_dot_gleam

Renumber authoring.blame line references:

gleam run -m vxml_pipeline/renumber_local_desugarer_blames

The generated registry exports each module’s constructor and an assertive_tests list. A small application-owned local_desugarer_tests.gleam can pass that list to vxml_pipeline/testing.test_desugarers.

When command-line maintenance handling is installed, the application also accepts:

These operations use paths relative to the application’s current working directory.

Module guide

License

MIT

✨ Search Document