Skip to content

InstrumentationTests don't fail when failure is expected #15837

@PhilLab

Description

@PhilLab

⚠️ Before posting ⚠️

  • This is a bug, not a question or an enhancement.
  • I've searched for similar issues and didn't find a duplicate.
  • I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
  • I agree to follow Nextcloud's Code of Conduct.

Steps to reproduce

  1. Open Instrumentation test, e.g. DrawerActivityIT.kt
  2. Add an assertion which must fail into the activity scenario:
    @Test
    @UiThread
    fun switchAccountViaAccountList() {
        launchActivity<FileDisplayActivity>().use { scenario ->
            scenario.onActivity { sut ->
                onIdleSync {
                    EspressoIdlingResource.increment()
                    sut.setUser(user1)
    
                    Assert.assertEquals(true, false)  //<----------------- impossible assertion
    
  3. Start the test, either via UI or via gradlew command

Expected behaviour

The test should fail

Actual behaviour

  1. The test doesn't fail
  2. and also doesn't really execute the UI steps.

But when adding the impossible assertion outside, it fails as expected:

@Test
@UiThread
fun switchAccountViaAccountList() {
    Assert.assertEquals(true, false)  //<----------------- impossible assertion
    launchActivity<FileDisplayActivity>().use { scenario ->
        scenario.onActivity { sut ->
            onIdleSync {
                EspressoIdlingResource.increment()
                sut.setUser(user1)

Android version

API 35

Device brand and model

Emulator (Google free)

Stock or custom OS?

Stock

Nextcloud android app version

current master (e1a80b6)

Nextcloud server version

Using a reverse proxy?

I don't know

Android logs

No response

Server error logs

Additional information

Screen recording showing successful tests despite impossible assertion. First via UI, then via gradlew command

Nextcloud-Android-Test-success.mp4

Screen recording showing how the same steps yield the expected test failure if the assertion is added outside of the scenario

Nextcloud-Android-Test-fail.mp4

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions