Documentation

Everything the tool does, arranged as the questions people actually arrive with. The README in the repository is the full reference and always matches the build you downloaded.

What commands are there?

Each one does a single thing:

tfg generate    produce files, from a recipe or from flags
tfg validate    check a recipe and write nothing
tfg verify      check a directory against a manifest
tfg cleanup     remove the files a manifest lists
tfg recipe fmt  print a recipe in its settled shape
tfg preset      build a set of files from a named test question
tfg formats     list the formats this build supports
tfg version     print the tool version
tfg license     print the licence and what it means for generated files

How do I generate a single file of an exact size?

Name the format, the size and where it goes. Sizes count in 1024s, so 2mb is 2097152 bytes. A plain byte count works too, so --size 10485761 asks for exactly that many.

tfg generate --format png --size 2mb --out ./out

The useful flags on generate:

Flag	What it does
`--format <id>`	format of the files, for example `txt`
`--size <size>`	exact size of every file, such as `10mb` or a plain byte count
`--size-range <a-b>`	a size drawn per file from a range, such as `1kb-8kb`. The draw comes from the seed
`--boundary <size>`	three files around a limit: one byte under, the limit, one byte over
`--count <n>`	how many files to produce. Default `1`
`--name <template>`	name template, for example `invoice_{index:04}.txt`
`--out <dir>`	directory to write into
`--seed <n>`	run seed. The same seed gives the same bytes
`--set <k>=<v>`	a format setting, repeatable
`--expected <outcome>`	`accept`, `reject`, `sanitize` or `unspecified`
`--dry-run`	count and show, write nothing at all
`--json`	write the manifest to standard output

What does a recipe look like?

A recipe is a YAML file describing a whole run. Commit it beside your tests and the fixtures stop being binaries in your repository - anyone can rebuild them, byte for byte, from a file of a few hundred characters.

# fixtures.yaml
version: 1
seed: 7741

defaults:
  label: true

targets:
  - id: invoices
    format: pdf
    count: 25
    size: 300kb
    name: invoice_{index:04}.pdf
    properties:
      pages: 3
      page_size: a4
    expected: accept

  - id: over_the_limit
    format: png
    count: 2
    size: 12mb
    expected:
      outcome: reject
      reason: size_limit

  - id: bundle
    format: zip
    contains:
      - format: txt
        count: 200
        size: 4kb

output:
  dir: ./fixtures
  manifest: manifest.json

tfg validate fixtures.yaml
tfg generate fixtures.yaml

Every target needs exactly one of size, size-range, boundary or contains. Two of them is an error and so is none. An invalid recipe writes no files at all and reports every problem at once rather than the first, each one naming the setting it is about.

How do I declare what my system should do with a file?

Short form when the outcome is enough, long form when the reason matters:

expected: accept

expected:
  outcome: reject
  reason: size_limit

The outcomes are accept, reject, sanitize and unspecified. The reasons are a closed list so a report can group by them: content_malformed, count_limit, dimensions_limit, duplicate, encoding_invalid, extension_rule, filename_invalid, filename_too_long, filename_traversal, malware_signature, mime_mismatch, nesting_depth, none, size_limit and size_zero.

A reason names the rule in play, not the verdict. That is why the same reason can sit under either outcome - a file one byte under a limit is accept, and the rule it is about is still size_limit.

What is in the manifest?

It is written next to the files at the end of every run, including a run that was interrupted. One entry per file:

{
  "manifest_version": "1.0",
  "tool": { "name": "testing-files-generator", "version": "0.1.0" },
  "run": {
    "id": "run_b359aa8d94",
    "seed": 0,
    "command": "tfg generate --format png --size 2mb --out ./out",
    "platform": { "os": "windows", "arch": "amd64" },
    "complete": true
  },
  "summary": {
    "file_count": 1,
    "total_bytes": 2097152,
    "by_format": { "png": 1 },
    "by_expected": { "unspecified": 1 }
  },
  "files": [
    {
      "path": "files_0001.png",
      "bytes": 2097152,
      "format": "png",
      "fidelity": "full",
      "determinism": "byte",
      "seed": "8dc2d18c",
      "hashes": { "sha256": "1a1f7c..." },
      "properties": { "width": 640, "height": 480 },
      "expected": {
        "outcome": "unspecified",
        "detail": "No expectation was declared for this file.",
        "confidence": "policy_dependent"
      }
    }
  ]
}

A recipe_hash is added when the run came from a recipe, and preset with overrides when it came from a preset, so a manifest can always be traced back to what produced it.

What is a preset?

A ready made set of files that answers a common testing question, so you do not have to design the set yourself. Presets are ordinary recipes underneath, and eject prints the recipe so you can edit it from there.

size-boundaries

Is a size limit enforced exactly where it is declared?

tfg preset list
tfg preset show size-boundaries
tfg generate --preset size-boundaries --limit 10mb --out ./edges
tfg preset eject size-boundaries > my.yaml

show tells you what the set would cost before you build it, and says outright when a number is a placeholder of ours rather than a limit of yours.

What do the exit codes mean?

Every ending has its own code, machine readable output goes to standard output, and a failed run prints nothing there. The table is a frozen contract - changing what a code means requires a major version bump.

Code	Meaning
`0`	Everything worked.
`1`	An unexpected error inside the tool.
`2`	Wrong command or flag.
`3`	The recipe is not valid.
`4`	The format cannot do what was asked.
`5`	A read or a write failed.
`6`	Not enough disk space.
`7`	verify found a mismatch.
`8`	The run finished but not everything was produced.
`130`	Interrupted with Ctrl+C.
`143`	Stopped by a signal, which is what a CI timeout looks like.

- name: build the fixtures
  run: tfg generate fixtures.yaml --out ./fixtures

- name: run the tests
  run: pytest tests/

- name: nothing moved
  run: tfg verify ./fixtures/manifest.json

A run stopped with Ctrl+C still leaves a manifest and never leaves a half written file behind, so a cancelled job can still be cleaned up by the next one.

Is there a desktop window?

Yes, the same engine with a window on it, for the testing that is not scripted. It is not a cut down version: a test compares the two interfaces capability by capability, and anything only one of them can do has to be declared and justified rather than quietly drifting apart.

The screens are one batch, presets, several batches at once, and about. It shows what a run would cost before writing anything, reports progress while it runs, and can be cancelled part way without leaving a half written file behind. It does not open a recipe file yet - recipes are a command line thing for now, and the window builds its batches in the form.