Use OIIO RAW extensions for file validation #231

mikaelsundell · 2026-01-03T01:16:59Z

Skip hard-coding file extensions when collecting input files and instead query OIIO for the RAW extensions it actually supports. The list is initialized once and reused both for directory scanning and --list-formats.

This also prepares for GUI use, where the same extension list will be needed for file dialogs and drag-and-drop filtering.

No API changes, no tests added at this point.

Checklist:

I have read the contribution guidelines.
I have updated the documentation, if applicable. (Check if there is no
need to update the documentation, for example if this is a bug fix that
doesn't change the API.)
I have ensured that the change is tested somewhere in the testsuite
(adding new test cases if necessary).
My code follows the prevailing code style of this project. If I haven't
already run clang-format before submitting, I definitely will look at the CI
test that runs clang-format and fix anything that it highlights as being
nonconforming.

codecov-commenter · 2026-01-03T01:19:57Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.70%. Comparing base (28b746a) to head (b6b6871).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #231      +/-   ##
==========================================
+ Coverage   85.48%   85.70%   +0.21%     
==========================================
  Files          11       11              
  Lines        2164     2197      +33     
  Branches      326      331       +5     
==========================================
+ Hits         1850     1883      +33     
  Misses        314      314

Files with missing lines	Coverage Δ
include/rawtoaces/image_converter.h	`100.00% <ø> (ø)`
src/rawtoaces_util/image_converter.cpp	`73.75% <100.00%> (+0.85%)`	⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 28b746a...b6b6871. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

soswow · 2026-01-03T05:16:54Z

No API changes

You meant no breaking API changes, but API is changed - there is a new member added + whole new command line option. That is be definition API change, just minor - non-breaking one. Not sure how it all be released. That's Q to Anton.

no tests added at this point.

I think we should have all new functionality added to be fully covered with unit tests.

soswow

Please cover all new functionality added with unit tests

soswow · 2026-01-03T05:30:11Z

Rest of it looks very good, btw!

mikaelsundell · 2026-01-03T12:00:57Z

You’re right - this does introduce an API change in the strict sense, as it adds a new public member and a new command-line option. What I meant was that the change is non-breaking: existing behavior and existing calls remain unaffected.

The new command-line option is informational only and simply exposes the list of RAW formats already supported internally via OIIO; it does not alter processing behavior or defaults.

The tricky part is whether we want to keep the new extension check in check_and_add_file, as this could potentially affect users who deliberately force formats that are not listed by OIIO.

I don’t mind adding a test to verify the reported formats if we agree this is the right direction. If the change in check_and_add_file feels too aggressive, we can either revert back to existing code or limit this logic to the GUI application instead.

antond-weta · 2026-01-03T21:17:36Z

Looks good.
Could you please add unit tests and python bindings for the new functionality.

mikaelsundell · 2026-01-03T21:58:49Z

Perfect, will freeze the code as-is and add the last changes.

Skip hard-coding file extensions when collecting input files and instead query OIIO for the RAW extensions it actually supports. The list is initialized once and reused both for directory scanning and --list-formats. This also prepares for GUI use, where the same extension list will be needed for file dialogs and drag-and-drop filtering. No API changes. Signed-off-by: Mikael Sundell <[email protected]>

Added get_supported_formats to python bindings Signed-off-by: Mikael Sundell <[email protected]>

Signed-off-by: Mikael Sundell <[email protected]>

mikaelsundell · 2026-01-03T23:58:09Z

First time adding tests, please input if anything is not right or missing. Tested the python binding and I can successfully request supported formats:

>>> converter = rawtoaces.ImageConverter()
>>> formats = converter.get_supported_formats()
>>> print(formats)
['.3fr', '.arw', '.bay', '.bmq', '.cap', '.cine', '.cr2', '.cr3', '.crw', '.cs1', '.dc2', '.dcr', '.dng', '.drf', '.dsc', '.erf', '.fff', '.ia', '.iiq', '.k25', '.kc2', '.kdc', '.mdc', '.mef', '.mos', '.mrw', '.nef', '.nrw', '.orf', '.pef', '.pxn', '.qtk', '.raf', '.raw', '.rdc', '.rw2', '.rwl', '.rwz', '.sr2', '.srf', '.srw', '.sti', '.x3f']

soswow · 2026-01-04T04:39:31Z

src/rawtoaces_util/image_converter.cpp

+            {
+                auto colon = group.find( ':' );
+                if ( colon == std::string::npos )
+                    continue;


Any way we get have a unit test that will get over here and over at line 61?

The continue statements are filters used while parsing OIIO’s extension_list, which is external input. They are internal implementation details.

The intended behavior of this function is to return the supported RAW extensions. By asserting the presence of well-known RAW formats such as .cr2 and .dng, the tests already verify that the parsing and filtering behave correctly. If any of the continue logic were incorrect, those extensions would be missing or incorrect and the test would fail.

That said, you still think an additional test is needed?

Thanks for adding the tests! Ideally we should be having close to 100% coverage over all newly added code. In some cases that would require splitting some functionality into a separate function, just so it can be executed will different inputs from unit tests.

We need to get over 90% total coverage to get the OpenSSF Best Practices badge. Sasha has been working on improving the test coverage of the existing code, but ideally everybody adding new code should be doing their part as well.

Just to temper expectations:

I think the three badge levels are graduated: no specific coverage requirement for "passing", 80% for silver, 90% for gold.

Every code base is different, but anecdotally, when we first started looking at improving OIIO's coverage, it was around 65% (as a sprawling, 10+ year old code base). It was easy (i.e. just a lot of hard work) to get to around 75%, since there were lots of HUGE uncovered sections that represented features we had that just weren't tested in the testsuite. But then going from 75% to 80+ was getting really hard -- it seemed like real work to eke out every additional 0.1%. We are currently at around 82% or so, and there is no low-hanging fruit left. Most of the uncovered lines now are in short isolated segments (sometimes just a few lines), and it's prohibitive to construct new tests that only add coverage to tiny portions of the code base each. While we're always striving to improve testing (and definitely aiming to get close to 100% coverage on NEW functionality), I consider 90% an essentially unattainable target for OIIO. Given the amount of work it takes to add coverage of each new block of uncovered lines, I think we could direct all development resources to test construction for literally years to get to > 90%. It would be possible, but not justifiable versus other things we could be working on.

Thanks for the insight, Larry. That is great info. OIIO is indeed huge and a pretty old code base. On the other hand, rawtoaces is pretty small, and a huge chunk of the code base has been rewritten from scratch about 6 months ago. So I think we are in a good position to get good coverage. I'm not sure if 90% is attainable, but we are currently sitting at 85%, and I can't say that we had spent huge amount of time to achieve that (Sasha may want to disagree).

I'm expecting the coverage to drop somewhat when we start working on the GUI component though.

yea, nah, it wasn't this hard. All branches of the code is pretty reachable. We are not at 90 or 95 only because I haven't gotten there yet. I am confident we can do it.

Ah, very good. If you're already in high 80s and think adding more coverage isn't hard, by all means go for it. I personally believe that untested code should be presumed to be buggy. I've been humbled by my own bugs far too many times to trust myself when I don't have a test.

I'm just saying: It can be extremely hard to get close to 100%, at some point you get diminishing returns where the resources needed to write the tests are so great that they are not giving you a positive ROI versus taking a chance and fixing a bug if it is ever encountered. Don't beat yourself up about making that judgment call when you feel like you have to make it. At the end of the day, it's about using your finite development resources where they do the most good. Very often, the best use of resources is getting better test coverage. But sometimes there are bigger fish to fry. This is a matter of the project leader setting the priorities that feel right for the particular project.

soswow · 2026-01-04T04:41:52Z

There is tests/python/test_image_converter.py full of tests for python binding. like test_converter_get_supported_cameras that is already there

Signed-off-by: Mikael Sundell <[email protected]>

soswow · 2026-01-05T05:09:40Z

@mikaelsundell I hope you can forgive me, but there was a refactoring-of-sort PR just merged that introduced some utils for tests (for setup and assertion). You might get a conflict and you might want to use those new goodies. sry about that, I was working on it for some time.

Signed-off-by: Mikael Sundell <[email protected]>

soswow · 2026-01-06T04:52:21Z

tests/test_image_converter.cpp

+
+    const auto &exts = rta::util::supported_raw_extensions();
+    OIIO_CHECK_EQUAL( exts.empty(), false );
+}


not sure if this test is needed.

No longer needed, removed!

soswow · 2026-01-06T04:55:54Z

tests/test_image_converter.cpp

+
+    OIIO_CHECK_EQUAL( found_cr2, true );
+    OIIO_CHECK_EQUAL( found_dng, true );
+}


can we please have a negative case:

bool found_png = false; ... if ( line == ".png" ) found_png = true; ... OIIO_CHECK_EQUAL( found_png, false );

soswow · 2026-01-06T05:04:59Z

src/rawtoaces_util/image_converter.cpp

+    static const std::set<std::string> extensions = [] {
+        std::string extensionlist;
+        if ( !OIIO::getattribute( "extension_list", extensionlist ) )
+            return std::set<std::string>{};


I believe these two lines (check) can be removed safely?
I am right that the result will be the same?
Or maybe we can assign default (empty vector) value and just call getattribute. If value is not found - it will just stay empty. no?

extension_list should exist for all valid OIIO builds, now changed to:

bool ok = OIIO::getattribute( "extension_list", extensionlist ); OIIO_ASSERT( ok && "OIIO did not provide extension_list" );

…he return value Add a negative PNG case to test_parse_parameters_list_formats. Signed-off-by: Mikael Sundell <[email protected]>

Match clang-format version used in GitHub CI Signed-off-by: Mikael Sundell <[email protected]>

soswow suggested changes Jan 3, 2026

View reviewed changes

mikaelsundell force-pushed the ms-260102-formats branch from 9ee0003 to 1b25d54 Compare January 3, 2026 23:46

mikaelsundell added 3 commits January 3, 2026 18:50

Added test_parse_parameters_list_formats() to image converter tests

e40c80d

Added get_supported_formats to python bindings Signed-off-by: Mikael Sundell <[email protected]>

Clang format fix for py_util.cpp after adding get_supported_formats

d3008cf

Signed-off-by: Mikael Sundell <[email protected]>

mikaelsundell force-pushed the ms-260102-formats branch from 763219c to d3008cf Compare January 3, 2026 23:51

soswow reviewed Jan 4, 2026

View reviewed changes

Added test_converter_get_supported_formats to python tests

b3ca909

Signed-off-by: Mikael Sundell <[email protected]>

mikaelsundell added 3 commits January 5, 2026 22:34

Added tests to improve branch coverage

fbeb2cc

Signed-off-by: Mikael Sundell <[email protected]>

Merge branch 'main' into ms-260102-formats

bcca5a1

Signed-off-by: Mikael Sundell <[email protected]>

Clang format fix for test_image_converter.cpp after merge

999c30a

Signed-off-by: Mikael Sundell <[email protected]>

soswow reviewed Jan 6, 2026

View reviewed changes

soswow approved these changes Jan 6, 2026

View reviewed changes

soswow reviewed Jan 6, 2026

View reviewed changes

mikaelsundell added 2 commits January 6, 2026 10:56

Treat missing extension_list as empty input instead of branching on t…

cd5c2ad

…he return value Add a negative PNG case to test_parse_parameters_list_formats. Signed-off-by: Mikael Sundell <[email protected]>

Removed test_supported_raw_extensions, no longer needed

b6b6871

Match clang-format version used in GitHub CI Signed-off-by: Mikael Sundell <[email protected]>

soswow approved these changes Jan 6, 2026

View reviewed changes

antond-weta merged commit 5d62f9a into AcademySoftwareFoundation:main Jan 6, 2026
20 checks passed

Use OIIO RAW extensions for file validation #231

Use OIIO RAW extensions for file validation #231

Uh oh!

Conversation

mikaelsundell commented Jan 3, 2026

Checklist:

Uh oh!

codecov-commenter commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

soswow commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

soswow left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soswow commented Jan 3, 2026

Uh oh!

mikaelsundell commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antond-weta commented Jan 3, 2026

Uh oh!

mikaelsundell commented Jan 3, 2026

Uh oh!

mikaelsundell commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soswow commented Jan 4, 2026

Uh oh!

soswow commented Jan 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Jan 3, 2026 •

edited

Loading

soswow commented Jan 3, 2026 •

edited

Loading

soswow left a comment •

edited

Loading

mikaelsundell commented Jan 3, 2026 •

edited

Loading

mikaelsundell commented Jan 3, 2026 •

edited

Loading