How to generate test files in a CI pipeline

A binary fixture in a repository stays in its history for good, cannot be reviewed in a diff, and stops being possible once the file is large. Generate the files inside the pipeline from a recipe instead. The recipe is text, the bytes come out the same every time, and a last step proves that nothing moved.

The short answer

Install tfg, run tfg generate fixtures.yaml --out ./fixtures before the tests, and tfg verify ./fixtures/manifest.json after them. Both steps fail the build by themselves, with an exit code that says why.

Why not commit them

Why a fixture should not live in the repository

It stays in the history. Deleting a binary later does not make a clone smaller, because every version of it is still there.
A diff cannot show what changed. A reviewer sees that a PDF is different and nothing else. A recipe changes by a line.
Large files do not fit. GitHub refuses a push that holds a file over 100 MB, so a test of a 500 MB upload limit has nothing to commit.

The recipe is the thing to commit. The same recipe and seed write the same bytes on every machine, so the file generated in the pipeline is the file you had on your laptop.

The recipe

A recipe that lives beside the tests

This one writes twenty five invoices that should be accepted and two images over a limit that should be turned away, and the manifest records both expectations:

version: 1
seed: 7741

targets:
  - id: invoices
    format: pdf
    count: 25
    size: 300kb
    expected: accept

  - id: over_the_limit
    format: png
    count: 2
    size: 12mb
    expected:
      outcome: reject
      reason: size_limit

tfg validate fixtures.yaml checks it without writing anything, and names every problem at once.

GitHub Actions

A workflow that installs the tool and builds the fixtures

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      TFG_VERSION: "0.4.0"
    steps:
      - uses: actions/checkout@v4

      - name: install tfg
        run: |
          base=https://github.com/donislawdev/TestingFilesGenerator/releases/download/v$TFG_VERSION
          curl -fsSLO "$base/tfg_${TFG_VERSION}_linux_amd64.tar.gz"
          curl -fsSLO "$base/verify-SHA256SUMS.txt"
          sha256sum --check --ignore-missing verify-SHA256SUMS.txt
          tar -xzf "tfg_${TFG_VERSION}_linux_amd64.tar.gz" ./tfg
          sudo mv tfg /usr/local/bin/

      - name: build the fixtures
        run: tfg generate fixtures.yaml --out ./fixtures

      - name: run the tests
        run: pytest tests/

      - name: nothing moved
        run: tfg verify ./fixtures/manifest.json

The checksum line compares the archive with verify-SHA256SUMS.txt from the same release. The version is pinned, so a new release never changes a build you did not touch.

GitLab CI

The same thing as a GitLab job

fixtures:
  image: ubuntu:24.04
  variables:
    TFG_VERSION: "0.4.0"
  script:
    - apt-get update -qq && apt-get install -y -qq curl ca-certificates
    - base=https://github.com/donislawdev/TestingFilesGenerator/releases/download/v$TFG_VERSION
    - curl -fsSLO "$base/tfg_${TFG_VERSION}_linux_amd64.tar.gz"
    - curl -fsSLO "$base/verify-SHA256SUMS.txt"
    - sha256sum --check --ignore-missing verify-SHA256SUMS.txt
    - tar -xzf "tfg_${TFG_VERSION}_linux_amd64.tar.gz" ./tfg
    - ./tfg generate fixtures.yaml --out ./fixtures
    - pytest tests/
    - ./tfg verify ./fixtures/manifest.json

When it goes red

What makes a step fail, and why

Every ending has its own exit code, so the step fails on its own and the log says which one. The ones a pipeline meets:

3 - the recipe is not valid. Nothing was written, and every problem is named
4 - the format cannot do what was asked, for example a size below its smallest
6 - there is not enough disk space
7 - tfg verify found a file that does not match its manifest
8 - the run finished, but not everything was produced

A failed run prints nothing on standard output, so a log parser never reads an error as data. The whole table is on the documentation page.

PowerShell

A PowerShell script needs one more line

PowerShell does not carry the exit code of a program out of a .ps1 file. Run one with -File and the script answers 0 even when the tool inside refused the work, so a build that should be red goes green. The last line is the whole fix:

tfg generate fixtures.yaml --out ./fixtures
exit $LASTEXITCODE

That is how PowerShell behaves, not something about this tool. cmd, bash and zsh need nothing extra.

Several jobs

Sharing the fixtures between jobs

There is usually no need to upload them. Because the same recipe writes the same bytes, each job can run its own tfg generate, which is quicker than an upload and a download. When a job has to receive files from another, run tfg verify on the manifest after the transfer, and it tells you whether what arrived is what was written.

Corrupt test files adds files that are broken on purpose to the same recipe.
The use cases show what else a run in a pipeline can check.
The documentation has every command, recipe key and exit code.