Skip to content

Core Engine

This document describes the internal architecture of dft-core, the engine that powers Dataface.

If you're new to the codebase, start with the Architecture Overview and Platform Overview, then read this page for engine internals.

Core modules

The engine lives in dataface/core/ and is organized around a small set of subsystems:

Module Path Role
Compile dataface/core/compile/ Parse YAML, validate it, and normalize it into typed compiled objects
Execute dataface/core/execute/ Execute compiled queries against sources and adapters
Render dataface/core/render/ Turn compiled dashboards plus data into SVG, HTML, PNG, PDF, or terminal output
Inspect dataface/core/inspect/ Inspect schemas, model shapes, and profiling metadata
Serve dataface/core/serve/ HTTP server layer for serving dashboards

Pipeline

Dataface transforms YAML dashboard definitions into rendered output through a compile/render flow with lazy execution:

flowchart LR
    YAML -->|parse, validate,\nnormalize| COMPILE
    COMPILE -->|CompiledFace| RENDER
    RENDER -->|lazy, per chart| EXECUTE
    EXECUTE -->|data| RENDER
    RENDER -->|SVG| CONVERT
    CONVERT --> Output
Stage Module What it does
Compile dataface/core/compile/ Produce a guaranteed CompiledFace
Execute dataface/core/execute/ Execute queries on demand during render
Render dataface/core/render/ Walk layout and render charts and composed faces
Convert dataface/core/render/converters/ Transform native SVG output into other formats

Execute is a service called by render. It is not a separate linear stage in the user-facing flow.

Important invariants

  • Trust the normalizer. Downstream code should treat compiled objects as already-resolved typed structures.
  • Keep compile as the foundation. Other subsystems may depend on compile types, but compile should not depend on render or execute internals.
  • Keep app concerns out of core. Django, editor UX, and product workflow belong in apps/, not here.
  • Prefer clear failures over silent fixes. Core should validate aggressively and surface bad input early.

Data Shape Boundary

Dataface treats query execution and chart rendering as separate responsibilities:

  • The query layer owns dataset meaning, grain, and transformation.
  • The render layer owns visual encoding and presentation.
  • If a chart needs different grouping, aggregation, bucketing, or semantic ordering, change the query rather than reshaping data in the viz layer.

This boundary keeps chart behavior predictable:

  • Queries define what the data means.
  • Charts define how that already-shaped data is encoded and displayed.
  • Render code should avoid hidden data rewrites that make the visual output depend on chart-local transformation logic.

For internal architecture discussions, a useful chart vocabulary is:

  • data: the bound dataset or query result
  • encoding: per-chart authored mappings and intent such as x, y, color, and title
  • structure: reusable chart-skeleton defaults such as axis side or legend placement
  • theme: reusable visual styling such as palette, typography, and surfaces

The metaphor is:

  • theme is like CSS
  • structure is like HTML or physical scaffolding

This is internal terminology for reasoning about responsibilities. Authored YAML does not need to be nested under these names.

We also use a context spectrum for chart reasoning:

  • data type -> field semantics -> observed values -> comparative context -> human context

This spectrum is about how much information is available to support chart selection and enrichment, not about moving data-shaping responsibility into the render layer. The canonical definition lives in dataface/core/render/chart/DESIGN.md.

Dependency direction

flowchart BT
    compile["compile/"]
    execute["execute/"] --> compile
    render["render/"] --> compile
    render --> execute
    inspect["inspect/"] --> compile
    serve["serve/"] --> render

When to read which part

  • Read compile/ when changing schema, normalization, or validation behavior.
  • Read execute/ when changing adapter behavior, query execution, or caching.
  • Read render/ when changing chart semantics, layout behavior, or output formats.
  • Read inspect/ when changing schema inspection, profiling, or metadata generation.
  • Read serve/ when changing the HTTP serving layer around the engine.