Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,26 @@ jobs:
REFERENCE_YUV="$prefix.yuv" target/release/incremental-check ".heic-test-assets/incremental-corpus/$name.heic"
done

- name: Bounded decoder oracle comparisons
run: |
ruby scripts/incremental/generate-bounded.rb .heic-test-assets/incremental-corpus
required=()
for path in .heic-test-assets/bounded-corpus/pass-*.heic \
.heic-test-assets/ente-test-fixtures/media/heic/v1/files/IMG_0983.HEIC \
.heic-test-assets/ente-test-fixtures/media/heic/v1/files/1718_rotate_90_cw.HEIC; do
required+=(--require-pass "$path")
done
ruby scripts/incremental/verify-bounded.rb "${required[@]}" \
.heic-test-runs/incremental-oracle/primary-oracle \
target/release/incremental-allocation target/release/incremental-compare \
.heic-test-assets/libheif/examples \
.heic-test-assets/libheif/tests/data \
.heic-test-assets/libheif/fuzzing/data/corpus \
.heic-test-assets/ente-test-fixtures/media/heic/v1/files \
.heic-test-assets/stress-corpus \
.heic-test-assets/incremental-corpus \
.heic-test-assets/bounded-corpus

- name: Upload verify report
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
Expand All @@ -159,5 +179,8 @@ jobs:
path: |
.heic-test-runs/verify/run/report.txt
.heic-test-runs/incremental/*.json*
.heic-test-runs/bounded/*.json*
.heic-test-runs/bounded/summary.txt
.heic-test-assets/bounded-corpus/
.heic-test-assets/incremental-corpus/
if-no-files-found: ignore
9 changes: 8 additions & 1 deletion API.md
Original file line number Diff line number Diff line change
Expand Up @@ -432,7 +432,14 @@ The primary item may be a grid of uniform HEVC tiles or a single HEVC image
(`hvc1`/`hev1`). Every coded item must be opaque 8-bit 4:2:0 HEVC with one IDR
slice; WPP is supported. A single image is decoded as a grid with one tile.
`clap`/`irot`/`imir` are applied, and Exif orientation applies when no
effective container orientation is present. RGB matrix/TRC ICC profiles and
effective container orientation is present. Container transforms for which
libheif first converts 4:2:0 chroma to 4:4:4 return `Unsupported`. Each is
judged at the size the image has when it applies: a `clap` with an odd left or
top edge, an anticlockwise 90° rotation (`irot` 1) of an odd width, a 180°
rotation with either side odd, an anticlockwise 270° rotation (`irot` 3) of an
odd height, or a mirror with either side odd. A display area that starts on an
odd source row or column is also refused, which is stricter than libheif when
a rotation or mirror comes before the crop. RGB matrix/TRC ICC profiles and
SDR nclx colour with BT.709, BT.2020 or Display P3 primaries are converted to
sRGB. HDR transfer functions, grayscale or
lookup-table ICC profiles, unprofiled wide-gamut HEVC colour, alpha, and
Expand Down
54 changes: 54 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,60 @@ libheif corpus and the ente fixtures.
Generated reports and PNG artifacts are under `.heic-test-runs/`. Use
`--keep-artifacts` with `verify` when debugging a pixel mismatch.

## Bounded decoder

CI also compares `decode_bounded` with the strict libheif primary-image oracle
described in the next section. Build the oracle and generate the incremental
corpus as shown there, then run:

```bash
cargo build --release --locked --features incremental-experiment --bins
ruby scripts/incremental/generate-bounded.rb .heic-test-assets/incremental-corpus
ruby scripts/incremental/verify-bounded.rb \
--require-pass .heic-test-assets/bounded-corpus/pass-odd-width-crop.heic \
.heic-test-runs/incremental-oracle/primary-oracle \
target/release/incremental-allocation target/release/incremental-compare \
.heic-test-assets/libheif/examples \
.heic-test-assets/libheif/tests/data \
.heic-test-assets/libheif/fuzzing/data/corpus \
.heic-test-assets/ente-test-fixtures/media/heic/v1/files \
.heic-test-assets/stress-corpus \
.heic-test-assets/incremental-corpus \
.heic-test-assets/bounded-corpus
```

`generate-bounded.rb` wraps the incremental corpus's 256×256 encode into the
ignored `.heic-test-assets/bounded-corpus`, or into an optional output folder.
These files are kept out of `incremental-corpus` because `verify.rb` requires
every file there to decode with the incremental experiment. The script refuses
an output folder that overlaps the incremental corpus, or that is not empty
and lacks the `.generate-bounded` marker it writes, and it only replaces the
`pass-*` and `refuse-*` files in its own folder. `refuse-*` files use rotations, mirrors or
grid canvases for which libheif converts to 4:4:4, so `decode_bounded` must
refuse them; if one decodes, the pixel comparison fails. `pass-*` files are
the matching controls and the Display P3 and BT.2020 nclx cases. CI passes
every `pass-*` file, `IMG_0983.HEIC` and `1718_rotate_90_cw.HEIC` with
`--require-pass`, which fails the run unless that file passes at both sizes.

Every HEIC, HEIF and AVIF file under the given paths is decoded at max side
6000 and 97 with a 512 MiB budget (`BOUNDED_BUDGET_MIB` overrides it). The
runner denies any allocation that would take the live heap past the budget.
Each successful decode is compared with the oracle's image after the
comparator applies the Exif orientation reported by
`exif_orientation_hint_from_path`, converts an embedded ICC profile or the
item's BT.2020 or Display P3 nclx primaries to sRGB, and area-averages to the
output size. The `rgb8-rounding` gate described below applies. The Exif value
comes from the decoder's own parser and the colour conversion from moxcms;
the comparator's orientation mapping and area average are independent.

Refusals with `Unsupported`, `LimitExceeded` or `MemoryBudgetExceeded` are
counted but do not fail the run. The run fails on a crash, a timeout, an
oracle timeout or crash, `AllocationFailed` for any file, any other error on a
file the oracle decodes, a successful decode of a file the strict oracle
rejects, a dimension or pixel mismatch, or a required file that does not pass.
Results and a summary are written under `.heic-test-runs/bounded`;
`BOUNDED_TEST_ROOT` overrides the folder.

## Incremental bounded decoder

The opt-in `incremental-experiment` feature is tested separately because
Expand Down
44 changes: 44 additions & 0 deletions scripts/incremental/generate-bounded.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
require 'fileutils'
require 'open3'
require 'rbconfig'

abort 'Usage: generate-bounded.rb INCREMENTAL_CORPUS_DIRECTORY [OUTPUT_DIRECTORY]' unless (1..2).cover?(ARGV.size)
corpus = File.expand_path(ARGV[0])
source = File.join(corpus, 'direct.hevc')
abort "Missing #{source}; generate the incremental corpus first" unless File.file?(source)
root = File.expand_path('../..', __dir__)
output = File.expand_path(ARGV[1] || File.join(root, '.heic-test-assets/bounded-corpus'))
within = ->(path, directory) { path == directory || path.start_with?("#{directory}/") }
abort 'The output directory must not overlap the incremental corpus' if within.call(output, corpus) || within.call(corpus, output)
marker = File.join(output, '.generate-bounded')
if Dir.exist?(output) && !Dir.empty?(output) && !File.file?(marker)
abort "#{output} is not empty and was not created by generate-bounded.rb"
end
FileUtils.mkdir_p(output)
File.write(marker, '')
FileUtils.rm_f(Dir.glob(File.join(output, '{pass,refuse}-*.heic')))

cases = {
'pass-odd-width-crop' => { 'CROP' => '131x192' },
'pass-odd-width-crop-rotate-270' => { 'CROP' => '131x192', 'ROTATION' => '3' },
'pass-odd-height-crop-rotate-90' => { 'CROP' => '192x129', 'ROTATION' => '1' },
'pass-odd-canvas-grid-rotate-270' => { 'GRID' => '1', 'CANVAS' => '255x256', 'ROTATION' => '3' },
'pass-nclx-display-p3' => { 'TILE_NCLX' => '12/13/6/0' },
'pass-nclx-bt2020' => { 'TILE_NCLX' => '9/1/9/0' },
'refuse-odd-width-crop-rotate-90' => { 'CROP' => '131x192', 'ROTATION' => '1' },
'refuse-odd-width-crop-rotate-180' => { 'CROP' => '131x192', 'ROTATION' => '2' },
'refuse-odd-height-crop-rotate-180' => { 'CROP' => '192x129', 'ROTATION' => '2' },
'refuse-odd-height-crop-rotate-270' => { 'CROP' => '192x129', 'ROTATION' => '3' },
'refuse-odd-width-crop-mirror-vertical-axis' => { 'CROP' => '131x192', 'MIRROR' => '0' },
'refuse-odd-width-crop-mirror-horizontal-axis' => { 'CROP' => '131x192', 'MIRROR' => '1' },
'refuse-odd-canvas-grid-rotate-180' => { 'GRID' => '1', 'CANVAS' => '255x256', 'ROTATION' => '2' }
}
keys = %w[GRID CANVAS CROP ROTATION MIRROR TILE_NCLX PRIMARY_NCLX REFERENCE_COUNT]
cases.each do |name, options|
path = File.join(output, "#{name}.heic")
environment = keys.to_h { |key| [key, nil] }.merge(options)
_, stderr, status = Open3.capture3(environment, RbConfig.ruby, File.join(__dir__, 'wrap.rb'), source, path,
'256', '256', '1', '1')
abort "wrap.rb failed for #{name}:\n#{stderr}" unless status.success?
puts "Generated #{path}"
end
12 changes: 11 additions & 1 deletion scripts/incremental/primary-oracle.cc
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,19 @@ int main(int argc, char **argv) {
int warnings = heif_image_get_decoding_warnings(image.get(), 0, nullptr, 0);
int width = heif_image_get_width(image.get(), heif_channel_interleaved);
int height = heif_image_get_height(image.get(), heif_channel_interleaved);
std::string nclx_json = "null";
heif_color_profile_nclx *nclx = nullptr;
if (heif_image_handle_get_nclx_color_profile(handle.get(), &nclx).code == heif_error_Ok && nclx) {
nclx_json = "{\"primaries\":" + std::to_string(static_cast<int>(nclx->color_primaries)) +
",\"transfer\":" + std::to_string(static_cast<int>(nclx->transfer_characteristics)) +
",\"matrix\":" + std::to_string(static_cast<int>(nclx->matrix_coefficients)) +
",\"full_range\":" + (nclx->full_range_flag ? "true" : "false") + "}";
heif_nclx_color_profile_free(nclx);
}
std::cout << "{\"primary_id\":" << primary << ",\"width\":" << width
<< ",\"height\":" << height << ",\"warnings\":" << warnings
<< ",\"strict\":" << (options->strict_decoding ? "true" : "false") << "}\n";
<< ",\"strict\":" << (options->strict_decoding ? "true" : "false")
<< ",\"nclx\":" << nclx_json << "}\n";
if (argc >= 3 && std::string(argv[2]) != "-") {
std::vector<uint8_t> icc(heif_image_get_raw_color_profile_size(image.get()));
if (!icc.empty()) {
Expand Down
152 changes: 152 additions & 0 deletions scripts/incremental/verify-bounded.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
require 'digest'
require 'fileutils'
require 'json'
require 'open3'
require 'timeout'

required = []
arguments = []
until ARGV.empty?
argument = ARGV.shift
if argument == '--require-pass'
abort '--require-pass needs a file' if ARGV.empty?
required << File.expand_path(ARGV.shift)
else
arguments << argument
end
end
abort 'Usage: verify-bounded.rb [--require-pass FILE]... ORACLE RUNNER COMPARATOR INPUT_OR_DIRECTORY...' if arguments.size < 4
oracle, runner, comparator, *sources = arguments.map { |p| File.expand_path(p) }
root = File.expand_path('../..', __dir__)
output = File.expand_path(ENV.fetch('BOUNDED_TEST_ROOT', File.join(root, '.heic-test-runs/bounded')))
budget_mib = Integer(ENV.fetch('BOUNDED_BUDGET_MIB', '512'))
sides = [6000, 97]
refusals = %w[unsupported limit_exceeded memory_budget_exceeded]
fatal = %w[timeout crash allocation_failed unexpected_error bounded_only dimension_mismatch pixel_mismatch
oracle_timeout oracle_crash]
FileUtils.mkdir_p(output)

inputs = sources.flat_map do |source|
if File.directory?(source)
Dir.glob(File.join(source, '**', '*')).select { |p| File.file?(p) && p.match?(/\.(heic|heif|avif)\z/i) }
else
[source]
end
end.uniq.sort
abort 'No inputs found' if inputs.empty?
missing = required - inputs
abort "Required inputs not found: #{missing.join(', ')}" unless missing.empty?

def run(*args)
Open3.popen3(*args, pgroup: true) do |input, out, err, thread|
input.close
stdout = Thread.new { out.read }
stderr = Thread.new { err.read }
begin
status = Timeout.timeout(120) { thread.value }
[status, stdout.value, stderr.value]
rescue Timeout::Error
Process.kill('KILL', -thread.pid)
thread.value
[nil, stdout.value, stderr.value]
end
end
end

def oriented(width, height, orientation)
orientation >= 5 ? [height, width] : [width, height]
end

manifest = {
binaries: [oracle, runner, comparator].to_h { |p| [p, Digest::SHA256.file(p).hexdigest] },
inputs: inputs.size,
required: required,
budget_mib: budget_mib,
sides: sides,
profile: 'rgb8-rounding'
}
File.write(File.join(output, 'manifest.json'), JSON.pretty_generate(manifest))
counts = Hash.new(0)
failures = []
File.open(File.join(output, 'results.jsonl'), 'w') do |report|
inputs.each do |path|
dir = File.join(output, Digest::SHA256.hexdigest(path))
FileUtils.mkdir_p(dir)
png = File.join(dir, 'reference.png')
status, stdout, stderr = run(oracle, path, png, 'strict')
oracle_outcome =
if status.nil?
'oracle_timeout'
elsif status.exitstatus.nil? || ![0, 1, 2].include?(status.exitstatus)
'oracle_crash'
end
oracle_error = stderr.strip unless status&.success?
reference = status&.success? ? JSON.parse(stdout) : nil
if oracle_outcome
counts[oracle_outcome] += 1
failures << "#{path}: #{oracle_outcome}: #{oracle_error}"
end
sides.each do |side|
raw = File.join(dir, 'actual.rgb')
FileUtils.rm_f(raw)
status, stdout, stderr = run(runner, 'decode-bounded', path, side.to_s, budget_mib.to_s, raw)
decoded = begin
JSON.parse(stdout.lines.last.to_s)
rescue JSON::ParserError
nil
end
entry = { path: path, side: side, reference: reference, oracle_error: oracle_error, decoded: decoded }
outcome =
if status.nil?
'timeout'
elsif decoded.nil? || ![0, 1].include?(status.exitstatus)
entry[:stderr] = stderr.lines.last(20).join
'crash'
elsif status.exitstatus == 1
if decoded['kind'] == 'allocation_failed'
'allocation_failed'
elsif reference.nil?
'both_refused'
elsif refusals.include?(decoded['kind'])
'refused'
else
'unexpected_error'
end
elsif reference.nil?
'bounded_only'
else
orientation = decoded.fetch('exif_orientation')
expected = oriented(reference.fetch('width'), reference.fetch('height'), orientation)
if [decoded.fetch('original_width'), decoded.fetch('original_height')] != expected
entry[:expected_original] = expected
'dimension_mismatch'
else
primaries = reference.dig('nclx', 'primaries') || 0
status, stdout, stderr = run(comparator, 'png', png, raw, decoded.fetch('width').to_s,
decoded.fetch('height').to_s, side.to_s, 'rgb8-rounding',
orientation.to_s, primaries.to_s)
entry[:metrics] = JSON.parse(stdout) rescue nil
entry[:comparator] = stderr.strip
status&.success? ? 'pass' : 'pixel_mismatch'
end
end
entry[:outcome] = outcome
report.puts(JSON.generate(entry))
report.flush
FileUtils.rm_f(raw)
counts[outcome] += 1
if fatal.include?(outcome)
detail = decoded&.fetch('error', nil) || entry[:comparator] || entry[:stderr] || oracle_error
failures << "#{path} (side #{side}): #{outcome}: #{detail}"
elsif required.include?(path) && outcome != 'pass'
failures << "#{path} (side #{side}): required to pass but #{outcome}: #{decoded&.fetch('error', nil)}"
end
end
FileUtils.rm_f(png)
end
end
summary = counts.sort.map { |outcome, count| "#{outcome}=#{count}" }.join(' ')
File.write(File.join(output, 'summary.txt'), "#{summary}\n#{failures.join("\n")}\n")
puts summary
failures.each { |failure| puts "FAIL #{failure}" }
exit(failures.empty? && counts['pass'].positive? ? 0 : 1)
Loading
Loading