How to make a corrupt file for testing
A validator that has only ever been shown healthy files has not really been tested. Here is how to get a file that is broken on purpose, comes out at exactly the size you ask for, and carries a manifest saying what your system should do with it.
The short answer
tfg generate --format png --size 2mb --damage zero-head --out ./out writes a PNG of
exactly 2097152 bytes whose first bytes are zeros, and the manifest beside it records that your
system should reject it.
The usual way
Why a file corrupted by hand makes a poor test
The usual ways are a hex editor, a script that flips a few random bytes, or cutting a file short
with head or truncate. They work once, and then they cost you:
- It is different every time. A random byte lands somewhere new on each run, so a failure on Tuesday may not come back on Wednesday.
- It changes the size. A file cut short is smaller than the limit it was meant to sit under, so the size check answers before the content check and the test passes for the wrong reason.
- It often goes unnoticed. Plain text still reads with a byte changed in the middle, and a forgiving image reader just draws it, so the file you meant to be broken is accepted.
- It says nothing about what should happen. The file is only bytes, and whoever reads the test next has to guess whether acceptance or rejection was intended.
What you get
A damaged file is still the size you asked for
The file is generated normally and broken afterwards, on its way to the disk. It keeps the size you asked for, and the same command writes the same bytes again.
tfg generate --format png --size 2mb --damage zero-head --out ./out
tfg generate --format pdf --size 1mb --count 5 --damage zero-head:bytes=16 --out ./broken
tfg generate --format zip --size 5mb --damage truncate:missing=30 --out ./cut
tfg generate --format jpg --size 500kb --damage append-junk --out ./tail
Settings go after a colon. The flag can be repeated, and the damages are applied in the order you write them. It works with every one of the 32 formats.
What it can do
Which damages are there?
This is the list the program prints, read from it when this page is built. tfg damage
prints the same, and tfg damage <id> says what one of them takes.
| Damage | What it does to the bytes | Smallest file | Settings |
|---|---|---|---|
append-junk |
Writes random bytes after the end of the file and keeps exactly the size you asked for, the way a file with something stuck to its end arrives. | 64 | bytes |
truncate |
Cuts the end off the file and keeps exactly the size you asked for, the way an interrupted upload or copy leaves it. | 4 | missing |
zero-head |
Overwrites the first bytes of the file with zeros, leaving its length alone. Most readers look there first, so this is the damage almost anything notices. | 8 | bytes |
zero-head writes zeros over the start of the file. Most readers look there first, at
the signature and the header that say what the file is, so almost any reader notices. Plain text
and logs have no signature and are turned away as well, because a run of zero bytes is not text.
Below four bytes some formats come out with damage that no reader complains about, which is why
the setting starts at four.
truncate cuts the end off the file, the way an interrupted upload or copy leaves
it, and the file is still the size you asked for: it is generated larger, and only its start
reaches the disk. missing says how much of the whole file is gone, half unless you
say otherwise.
append-junk writes random bytes after the end of the file, 64 unless you say
otherwise, the way a file arrives with something stuck to its end. The same command appends the
same bytes.
What the manifest says
A manifest that says what should happen
Where the damage breaks the rules of the format, the entry says your system should reject it, with the damage recorded beside it:
"expected": {
"outcome": "reject",
"reason": "content_malformed",
"confidence": "certain"
},
"damage": [
{
"type": "zero-head",
"settings": {
"bytes": "8"
}
}
]
Where readers differ, the entry says so instead of guessing. A text file cut short is still text, and most viewers skip bytes after the end of a PNG, so the entry leaves the answer to your system, offers both, and says why:
"expected": {
"outcome": "unspecified",
"reason": "content_malformed",
"detail": "A PNG ends at its IEND chunk and most viewers skip what follows it.",
"confidence": "policy_dependent",
"policy_options": ["accept", "reject"]
}
tfg damage <id> lists the formats where a damage leaves the answer to your
system, and why.
Four requests are refused before anything is written, because each would leave a file on disk that the manifest describes wrongly:
- a file smaller than the damage needs, which would come out unchanged
-
expected: acceptbeside a damage that breaks the rules of the format, because nothing that follows them accepts the file. Where the answer depends on your system,acceptis allowed. Writesanitizeif your system is meant to repair the file, orunspecifiedif that is the question you are asking -
truncateafterappend-junk, which would cut away the bytes that were added. Puttruncatefirst -
truncateorappend-junkon an archive that takes its size from what it holds, because there is no size to work from
In a recipe
Healthy and broken files in one run
Put both in one recipe, and the manifest carries the expectation of every file, so the test does not need a list of which is which:
version: 1
targets:
- id: healthy
format: pdf
size: 1mb
expected: accept
- id: broken
format: pdf
size: 1mb
damage:
- zero-head
In a test
Turning it into a test
The test reads the manifest and checks that what happened is what was declared. It needs no list of file names:
import json, os
directory = "healthy-and-broken"
manifest = json.load(open(os.path.join(directory, "manifest.json")))
for entry in manifest["files"]:
outcome = entry["expected"]["outcome"]
if outcome == "unspecified":
continue
response = upload(os.path.join(directory, entry["path"]))
if outcome == "reject":
assert not response.ok
else:
assert response.ok
A file whose outcome is unspecified is left out here, because what your system
should do with it is your own policy. Its entry says why, and policy_options lists
the answers there are.
A good refusal is a clean one. A message that says what was wrong is the answer you want. A server error, a hang or a half stored file is the defect this test exists to find.
Next
Where to go from here
- The upload-validation preset asks the other two questions of a form, the size and the type.
- Test files in CI runs a recipe like this one in a pipeline.
-
The documentation has every flag of
tfg generate.