Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark tests as expected to fail #2641

Open
daniel-vergien-bdr opened this issue Jan 7, 2025 · 8 comments
Open

Mark tests as expected to fail #2641

daniel-vergien-bdr opened this issue Jan 7, 2025 · 8 comments
Assignees
Milestone

Comments

@daniel-vergien-bdr
Copy link

daniel-vergien-bdr commented Jan 7, 2025

It would be nice to have a way to mark a test a expected to fail. This would come in handy if you write a test for a bug before the bug is fixed.
Junit-pioneer has such a feature: https://junit-pioneer.org/docs/expected-to-fail-tests/

@daniel-vergien-bdr daniel-vergien-bdr changed the title Mark tests as exprected to fail Mark tests as expected to fail Jan 7, 2025
@ptrthomas
Copy link
Member

interesting - will need to think about this a bit and how it would co-exist with soft assertions: #1446

maybe we should introduce a @mustfail or @fail built-in tag

@daniel-vergien-bdr
Copy link
Author

I think a build-in tag would be nice. My idea would be to evaluate this as the last step after a test is finished. If the tag is present the result is inverted. This way it would not interfere with any other stuff.

@daniel-vergien-bdr
Copy link
Author

On a first look in the source code I did not find a place where e.g. the other build-in tags are evaluated. If you give me a hint I might prepare a MR.

@ptrthomas
Copy link
Member

dve added a commit to Fork-World/karate that referenced this issue Jan 14, 2025
dve added a commit to Fork-World/karate that referenced this issue Jan 14, 2025
ptrthomas added a commit that referenced this issue Jan 16, 2025
#2641 introduce @fail tag to mark tests which are expected to fail
@ptrthomas ptrthomas added this to the 1.5.2 milestone Jan 16, 2025
@ptrthomas
Copy link
Member

@daniel-vergien-bdr @dve merged with thanks, much appreciated !

@dve
Copy link

dve commented Jan 16, 2025

Are 1.5.2-SNAPSHOT versions published somewhere?

@ptrthomas
Copy link
Member

@dve no, I have to push a button to do it. we typically wait for a few more changes to accumulate and release RC versions if teams need something sooner

@dve
Copy link

dve commented Jan 16, 2025

I would love a RC as soon as you can do it ;-)

And just for the record: the change might be a breaking change if someone used the @fail for something strange already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants