Contemplating the Spec(Test) Methodology for integrated testing

[hahwul]

I received an idea from @ksg97031 regarding Spec(test).

Currently, most of our testing functionality consists of unit tests at the Func level, but there's a need for integrated testing in the form of identifying actual example files and detected endpoints.

I share the same sentiment, and I'm still pondering how to structure this. If you have any thoughts or suggestions, please feel free to share! Personally, I think there might be a way to create a separate directory or repository for testing purposes.

[ksg97031]

Taking the OSS Review Toolkit (ORT) project as an example, I believe that this approach is a good one:

Utilize separate directories for testing within the ORT project. Each of these directories is designed to contain the expected results, which are crucial for successful testing.

For example, consider the bundler > funTest > synthetic within the ORT project. This can provide a clear illustration of this testing strategy. (Explore others here)

Additionally, the analyzer > funTest > synthetic within the ORT project demonstrates how this approach can be applied in analyzer case.

[hahwul]

@ksg97031
Hello again :D
I've structured the test source as file-based and made code improvements to enable testing through Crystal Spec (#38). Currently, I've based the expected results on counts, like the number of detected endpoints. However, I'm contemplating whether relying solely on numerical checks for the detection results might lead to incorrect identifications. So, I'm pondering whether to use more detailed expected results, such as string matching, for instance. Both approaches have their pros and cons.

Do you have any thoughts or suggestions on this?

crystal spec -v
# ...
# Functional test to fixtures/kemal/
#  test detect
#  test analyze
# ...

[ksg97031]

@hahwul
Wow, I also believe that using the expected result count is a good way to begin, as it makes it easier to integrate with other frameworks. However, as you're aware, relying solely on numerical comparisons for expected outputs might not always provide an intuitive way to identify differences.

Even though we might opt for counting for now, it seems that accurate string comparisons will be necessary in the future.

And if the program's behavior is asynchronous, how about considering using sorting to compare the expected outputs? Additionally, it might be worth thinking about printing error cases for better understanding.

[hahwul]

Hi all
I've decided on the direction of Functional testing, taking @ksg97031's input into strong consideration. Just merged a #38 PR moments ago, and now we're splitting the spec directory into unit tests and functional tests. For functional tests, you can add simplified source code of the app you're testing under the ./spec/functional_test/fixtures/ directory and write code for functional testing under the ./spec/functional_test/tester/s directory.

Here's an example of the code:

# https://github.com/hahwul/noir/blob/main/spec/functional_test/testers/crystal_kemal_spec.cr
require "../func_spec.cr"

extected_endpoints = [
  Endpoint.new("/", "GET"),
  Endpoint.new("/socket", "GET"),
  Endpoint.new("/query", "POST", [Param.new("query", "", "body")]),
]

FunctionalTester.new("fixtures/kemal/", {
  :techs     => 1,
  :endpoints => 3,
}, extected_endpoints).test_all

you can check based on the number of detections for techs, endpoints, and so on, or you can directly input Endpoint information for precise comparisons. Both arguments are optional, allowing you to configure the test flow by either initializing FunctionalTester with values or passing them as nil.

crystal spec -v

If you have any questions or concerns, please don't hesitate to reach out anytime! Goodbye👋🏼