First Workflow#

Part 1 of 1 in Overview

▶ 01. First Workflow

This tutorial shows a complete FAST-HEP workflow from start to finish.

You are not expected to understand every detail yet.

The goal is to see the entire analysis pipeline once before we explore each piece in dedicated tutorials.

By the end of this page, you will have run a workflow that:

toy data
→ derived quantity
→ histogram
→ plot

and produced a publication-style analysis output with only a small amount of YAML.

What this tutorial demonstrates#

This workflow includes:

  • a dataset definition

  • a data source

  • a derived variable

  • a histogram

  • a rendered plot

The following tutorials explain each concept individually.

The tutorials listed above cover the building blocks needed to understand this workflow.

As you progress through the workshop you will encounter many additional topics, including remote data access, skimming, systematics, distributed execution, GPU acceleration, and complete public analysis records.

Tutorial files#

tutorials/00-overview/first-workflow/
├── author.yaml
├── README.md
└── expected/

The workflow itself is defined in:

author.yaml

Workflow overview#

This workflow:

  1. generates toy events

  2. computes a derived variable

  3. fills a histogram

  4. renders a plot

        flowchart TD
  read_events["read.events<br/>source<br/>workshop.toy_source"]
  stage_BasicVars["stage.BasicVars<br/>transform<br/>hep.define"]
  stage_MuonPt["stage.MuonPt<br/>transform<br/>hep.hist"]
  render_MuonPt_0["render.MuonPt.0<br/>sink<br/>hep.render.hist1d"]

  read_events -->|stream -> stream| stage_BasicVars
  stage_BasicVars -->|stream -> stream| stage_MuonPt
  stage_MuonPt -->|hist -> target| render_MuonPt_0

FAST-HEP compiles workflows into dependency-aware execution graphs.

You do not need to understand every node in this graph yet.

For now, the important idea is that data flows through a series of operations:

source
→ transform
→ summarise
→ render

The remainder of the workshop explores each of these steps separately.

The workflow file#

FAST-HEP workflows are declarative.

Rather than writing explicit event loops, you describe:

  • what data should be processed

  • which quantities should be computed

  • which outputs should be produced

FAST-HEP then builds an execution plan automatically.

This workshop focuses on learning by doing. Throughout the tutorials you will build workflows incrementally and discover the language through practical examples.

For a more complete description of the workflow language and FAST-HEP architecture, see:

The complete workflow is shown at the end of this tutorial.

Run the tutorial#

From the repository root:

pixi run fasthep run tutorials/00-overview/first-workflow/author.yaml \
  --outdir build/tutorials/00-overview/first-workflow

Inspect the outputs#

The workflow produces outputs under:

build/tutorials/00-overview/first-workflow/

The most important directories are:

build/tutorials/00-overview/first-workflow/
├── artifacts/
├── compile/
├── graph/
├── reports/
└── run_summary.yaml

  • artifacts/ contains user-facing outputs such as plots and histograms

  • compile/ contains compiler products and execution plans

  • graph/ contains workflow visualisations

  • reports/ contains diagnostics and metadata

  • run_summary.yaml summarises the workflow execution

Expected outputs#

A curated set of expected outputs is provided in:

tutorials/00-overview/first-workflow/expected/

These files highlight the most important results from the build directory.

Plot#

The workflow produces a histogram of the derived muon transverse momentum.

Muon transverse momentum histogram

Expected MuonPt.png plot produced by this tutorial.#

Histogram metadata#

The expected directory also contains a small histogram manifest (expected/snippets/histograms.manifest.json):

{
  "histograms": [
    {
      "id": "MuonPt",
      "path": "artifacts/histograms/MuonPt.pkl",
      "producer": "stage.MuonPt"
    }
  ]
}

This demonstrates how histogram products are recorded and tracked.

Your generated outputs should look similar, although exact values may vary slightly between environments and package versions.

Full workflow#

Show author.yaml
version: 1.0

use:
  profiles:
    - registry
    - fasthep_carpenter:registry
    - fasthep_render:registry
    - fasthep_workshop:registry

data:
  datasets:
    - name: toy
      eventtype: mc
      nevents: 100000
      files: [toy://dy]
  defaults:
    eventtype: mc
    tree_primary: events

sources:
  events:
    kind: workshop.toy_source
    stream_type: event_stream
    nevents: 100000
    seed: 1729
    dataset: toy
    eventtype: mc
    weight: 1.0

styles:
  muon_pt:
    op: hep.render.hist1d
    figure:
      size: [8, 6]
      dpi: 150
    axes:
      x:
        name: Muon_Pt
        label: "Muon $p_T$ [GeV]"
        limits: [0, 120]
      y:
        name: events
        label: "Events"
        scale: linear

analysis:
  stages:
    - id: BasicVars
      op: hep.define
      params:
        variables:
          - name: Muon_Pt
            expr: "sqrt(Muon_Px ** 2 + Muon_Py ** 2)"

    - id: MuonPt
      op: hep.hist
      params:
        axes:
          - name: Muon_Pt
            source: Muon_Pt
            type: regular
            bins:
              low: 0
              high: 120
              nbins: 60
      render:
        style: muon_pt
        when: final
        out: MuonPt.png

What happens next?#

This tutorial showed the complete workflow all at once.

The rest of the workshop introduces each concept separately:

  1. reading data

  2. transforming data

  3. creating histograms

  4. rendering outputs

one step at a time.

Continue with tutorials/01-read-data/01-root-files

which introduces the first real analysis input: ROOT files.