Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add option to show diff in assert_output and assert_equal #65

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

justinledford
Copy link

fixes #60

@justinledford justinledford changed the title add option to show diff in assert_output add option to show diff in assert_output and assert_equal Jan 9, 2024

while (( $# > 0 )); do
case "$1" in
-d|--diff) show_diff=1; shift ;;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be a breaking change for some users that want to check if something is -d/--diff

Unfortunately, we don't have a -- interface established here. I think at minimum we should check that there are still (at least) two parameters left after the shift.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@@ -137,6 +141,7 @@ assert_output() {
case "$1" in
-p|--partial) is_mode_partial=1; shift ;;
-e|--regexp) is_mode_regexp=1; shift ;;
-d|--diff) show_diff=1; shift ;;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, --diff might be a valid output to check for. At least, we have -- here to differentiate options from output but we should at least check that there still is a trailing output.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@gioele
Copy link

gioele commented May 27, 2024

Why an option? The diffing could happen automatically if diff is available.

| batslib_decorate 'output differs' \
| fail
if (( show_diff )); then
diff <(echo "$expected") <(echo "$output") \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unified file format is more common than the so called normal format. It would be better to use diff -u instead of just diff here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, done

| batslib_decorate 'output differs' \
| fail
else
batslib_print_kv_single_or_multi 8 \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps batslib_print_kv_single_or_multi` could grow support for diffing expected and actual?

For instance it could use wdiff instead of diff if it detects single-line text.

Copy link
Author

@justinledford justinledford left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the comments. I have addressed them and also added test cases for the new functionality.

@@ -137,6 +141,7 @@ assert_output() {
case "$1" in
-p|--partial) is_mode_partial=1; shift ;;
-e|--regexp) is_mode_regexp=1; shift ;;
-d|--diff) show_diff=1; shift ;;
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

| batslib_decorate 'output differs' \
| fail
if (( show_diff )); then
diff <(echo "$expected") <(echo "$output") \
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, done


while (( $# > 0 )); do
case "$1" in
-d|--diff) show_diff=1; shift ;;
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@jasonkarns
Copy link
Member

Is there an opportunity here instead of trying to bake this into the matcher, to instead have a form of assert_output | diff? (diff here being function we own, not the command)

Having a generic diff utility would be more composable to other existing and future matchers. And allow multiple types of formatters to coexist. (Rather than an explosion of matcher options.)

This would require having a structured format/spec the assertions can construct that "renders" can consume as an api.

@mmetc
Copy link

mmetc commented Jul 12, 2024

I'm just giving my 2c as a happy bats user

For TUI output, I find it way faster to reconcile the differences with a tool like github.com/dandavison/delta

So I know what you mean about colors and the meaning of red, but it would be very useful to customize the diff command in some way.

Then when it comes to comparing contents, I would welcome anything that makes it easier to write stuff like this. I am not proud but it makes tests more compact and readable. That's what you mean by matcher I suppose

# Compare ignoring the key order, and allow "expected" without quoted identifiers.
# Preserve the output variable in case the following commands require it.
assert_json() {
    local oldout="${output}"
    # validate actual, sort
    run -0 jq -Sen "${output}"
    local actual="${output}"

    # handle stdin, quote identifiers, sort
    local expected="$1"
    if [[ "${expected}" == "-" ]]; then
        expected="$(cat)"
    fi
    run -0 jq -Sn "${expected}"
    expected="${output}"

    #shellcheck disable=SC2016
    run jq -ne --argjson a "${actual}" --argjson b "${expected}" '$a == $b'
    #shellcheck disable=SC2154
    if [[ "${status}" -ne 0 ]]; then
        echo "expect: $(jq -c <<<"${expected}")"
        echo "actual: $(jq -c <<<"${actual}")"
        diff <(echo "${actual}") <(echo "${expected}")
        fail "json does not match"
    fi
    output="${oldout}"
}
export -f assert_json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Print colorful diff between actual output and an expected output on test failure via smth like colordiff
5 participants