Skip to content

Feat all child steps #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Feat all child steps #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
89eb1ff
feat: Report child steps in extra property of a step
apatelia Oct 20, 2024
cf75819
docs(README): Fix table formatting in README
apatelia Oct 20, 2024
2c0c803
style(Docs): Fix formatting of code within docs
apatelia Oct 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 61 additions & 22 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ reporter: [
outputFile: 'custom-name.json', // Optional: Output file name. Defaults to 'ctrf-report.json'.
outputDir: 'custom-directory', // Optional: Output directory path. Defaults to '.' (project root).
minimal: true, // Optional: Generate a minimal report. Defaults to 'false'. Overrides screenshot and testType when set to true
testStepsOnly: false, // Optional: Include only 'test.steps' category steps in the report. Read 'Advanced Usage' section below for details.
screenshot: false, // Optional: Include screenshots in the report. Defaults to 'false'.
annotations: false, // Optional: Include annotations in the report. Defaults to 'false'.
testType: 'e2e', // Optional: Specify the test type (e.g., 'api', 'e2e'). Defaults to 'e2e'.
Expand Down Expand Up @@ -136,30 +137,68 @@ Replace directory with the path to the directory containing the CTRF reports you

The test object in the report includes the following [CTRF properties](https://ctrf.io/docs/schema/test):

| Name | Type | Required | Details |
| ------------ | ---------------- | -------- | ----------------------------------------------------------------------------------- |
| `name` | String | Required | The name of the test. |
| `status` | String | Required | The outcome of the test. One of: `passed`, `failed`, `skipped`, `pending`, `other`. |
| `duration` | Number | Required | The time taken for the test execution, in milliseconds. |
| `start` | Number | Optional | The start time of the test as a Unix epoch timestamp. |
| `stop` | Number | Optional | The end time of the test as a Unix epoch timestamp. |
| `suite` | String | Optional | The suite or group to which the test belongs. |
| `message` | String | Optional | The failure message if the test failed. |
| `trace` | String | Optional | The stack trace captured if the test failed. |
| `rawStatus` | String | Optional | The original playwright status of the test before mapping to CTRF status. |
| `tags` | Array of Strings | Optional | The tags retrieved from the test name |
| `type` | String | Optional | The type of test (e.g., `api`, `e2e`). |
| `filepath` | String | Optional | The file path where the test is located in the project. |
| `retries` | Number | Optional | The number of retries attempted for the test. |
| `flaky` | Boolean | Optional | Indicates whether the test result is flaky. |
| `browser` | String | Optional | The browser used for the test. |
| `screenshot` | String | Optional | A base64 encoded screenshot taken during the test. |
| `steps` | Array of Objects | Optional | Individual steps in the test, especially for BDD-style testing. |
| Name | Type | Required | Details |
| ------------ | ---------------- | -------- | ---------------------------------------------------------------------------------- |
| `name` | String | Required | The name of the test. |
| `status` | String | Required | The outcome of the test. One of:`passed`, `failed`, `skipped`, `pending`, `other`. |
| `duration` | Number | Required | The time taken for the test execution, in milliseconds. |
| `start` | Number | Optional | The start time of the test as a Unix epoch timestamp. |
| `stop` | Number | Optional | The end time of the test as a Unix epoch timestamp. |
| `suite` | String | Optional | The suite or group to which the test belongs. |
| `message` | String | Optional | The failure message if the test failed. |
| `trace` | String | Optional | The stack trace captured if the test failed. |
| `rawStatus` | String | Optional | The original playwright status of the test before mapping to CTRF status. |
| `tags` | Array of Strings | Optional | The tags retrieved from the test name |
| `type` | String | Optional | The type of test (e.g.,`api`, `e2e`). |
| `filepath` | String | Optional | The file path where the test is located in the project. |
| `retries` | Number | Optional | The number of retries attempted for the test. |
| `flaky` | Boolean | Optional | Indicates whether the test result is flaky. |
| `browser` | String | Optional | The browser used for the test. |
| `screenshot` | String | Optional | A base64 encoded screenshot taken during the test. |
| `steps` | Array of Objects | Optional | Individual steps in the test, especially for BDD-style testing. |
| `extra` | Object | Optional | Custom data relevant to the test. |

## BDD styled tests

⚠️ BDD styled tests that use [Cucumber](https://cucumber.io/docs/guides/overview/) as test runner are not supported. Only [playwright-bdd](https://vitalets.github.io/playwright-bdd/#/) is supported because it uses [Playwright Test](https://playwright.dev/docs/test-configuration) as a test runner to execute the tests.

## Advanced usage

Some features require additional setup or usage considerations.

### Test steps

Test steps are included in the `steps` property of a `Test` object. However, this `steps` property includes only top level steps. This means any children steps i.e. implementation details, are not included in it.

Children steps are included in the `extra` property as `childSteps`.

Every step has following properties of its own.

| Name | Type | Required | Details |
| -------- | ------ | -------- | ---------------------------------------------------------------------------------- |
| `name` | String | Required | The name/title of the step. |
| `status` | String | Required | The outcome of the step. One of:`passed`, `failed`, `skipped`, `pending`, `other`. |
| `extra` | Object | Optional | Custom data relevant to the step. |

Apart from including child steps, `extra` property is also used to report a steps' category, execution duration and the location in a source file. `extra` object contains following additional properties generated specifically for this JSON report.

| Name | Type | Details |
| ------------ | --------------- | --------------------------------------------------------------------------------------------- |
| `category` | String | The category of the step. One of:`hook`, `expect`, `pw:api`, `test.step` |
| `duration` | String | The execution duration of the step, in milliseconds. |
| `location` | Object | Location the step in test suite, including file name, line number and column number. |
| `childSteps` | Array of object | Child steps of the current step, if any. Child steps have the same properties that of a step. |

#### Include test steps only

By default, the generated report includes steps and their child steps of all [Playwright\'s built-in test step categories](https://playwright.dev/docs/api/class-teststep#test-step-category), i.e. `hook`, `expect`, `pw:api` and `test.step`.

For any reasons, if you want to include only the steps belonging to 'test.step' category, then set `testStepsOnly: true` in the configuration.

You can refer to some [examples here](/docs/test-steps-only.md) to understand the impact of `testStepsOnly` option.

> If you are **not** using test.step() in your tests, or using BDD styled tests with 'Background' keyword, then most likely you will need to include all steps in the report. Consider using the default behavior in such scenarios.

### Annotations

By setting `annotations: true` you can include annotations in the test extra property.
Expand Down Expand Up @@ -209,10 +248,10 @@ export const test = _test.extend<{ _autoAttachMetadata: void }>({
})

// ---------------------------------------------------------
await use(/** our test doesn't need this fixture direcly */);
await use(/** our test doesn't need this fixture directly */);
// ---------------------------------------------------------

// AFTER: There's nothing to cleanup in this fixutre
// AFTER: There's nothing to cleanup in this fixture
}, { auto: true }],
})

Expand Down Expand Up @@ -240,7 +279,7 @@ CTRF is a universal JSON test report schema that addresses the lack of a standar

**Language and Framework Agnostic:** It provides a universal reporting schema that works seamlessly with any programming language and testing framework.

**Facilitates Better Analysis:** With a standardized format, programatically analyzing test outcomes across multiple platforms becomes more straightforward.
**Facilitates Better Analysis:** With a standardized format, programmatically analyzing test outcomes across multiple platforms becomes more straightforward.

## Support Us

Expand Down
Loading
Loading