v5: Discussion

[nohwnd]

🙋‍♀️

I am raising a few questions in v5 readme and this is the place to start discussion so you don't have to go through all the respective issues and ask there.

[rdbartram]

I like the idea of Add-Dependency to simplify importing scripts, but what about modules? I had some test classes I created for pester testing vmware PowerCLi but I had to mark them as .ps1 and use add-dependency in the beforeall in order that they worked. Previously, i had them as using module statements at the top of the .test.ps1 but that doesn't work anymore.

Could we do something with Add-Dependency so it could import modules/classes?

Where is the code for Add-Dependency? If I use get-command, I see its in the pester.runtime. Where is that?

Finally, I'm abit confused if we should be using Add-Dependency or not because here #1283, you say using beforeall and not add-dependency, but in the readme you say using Add-Dependency. Thanks for the clarification

[nohwnd]

@rdbartram when I wrote Add-Dependency I made it do the same thing as ., but then I realized that I am not using it as such. Instead I am using the Add-Dependency in the same manner as BeforeAll, so the goal of the issue is to use the same syntax so there is one less cmdlet to think about.

I don't know if I am able to do anything about using module, so that is still something I need to research.

All in all the readme is going a bit outdated, as I go and form how it will all work. :)

[rdbartram]

ok. so i'll stick to dot sourcing in the beforeall as in v4. Is a cool idea though, to simplify the dependencies. Could even make it something like frontmatter at the start of the tests.ps1

[Stephanevg]

After writing this, I am not 100% sur if this should be in this thread.
Although the -Passthru object format is not mentionned in the readme document, I think it would make sense to discuss it a bit, as the change is pretty heavy compared to the v4 one. If you prefere having me open a issue of it own to track this just let me know.

Hi,
I looked at the output of the -Passthru object a bit today, and there is really a very big difference with the v4 one.

Describe "simple test" {
    it 'Should be false' {
        $true | Should be $false
    }

    it 'Should be true' {
        $true | should be $true
    }
}

Currently, executing the following test in v4 will result in this output:

pester v4.X output

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2968309
TestResult        : {@{ErrorRecord=Expected $false, but got $true.; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=False; Show=All; 
                    FailureMessage=Expected $false, but got $true.; Time=00:00:00.0120719; Name=Should be false; Result=Failed; Context=; StackTrace=at <ScriptBlock>, 
                    C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1: line 3
                    3:         $true | Should be $false}, @{ErrorRecord=; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=True; Show=All; 
                    FailureMessage=; Time=00:00:00.0058684; Name=Should be true; Result=Passed; Context=; StackTrace=}}

pester v5 output

Content              : C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1
Type                 : File
PSTypename           : ExecutedBlockContainer
OneTimeTestSetup     : 
Focus                : False
DiscoveryDuration    : 00:00:00.2508997
ExecutedAt           : 9/28/2019 9:19:08 AM
EachTestTeardown     : 
Id                   : 
ShouldRun            : True
PluginData           : {}
ScriptBlock          : 
AggregatedPassed     : False
Tests                : {}
FrameworkData        : {PreviouslyGeneratedTests, PreviouslyGeneratedBlocks}
Passed               : True
IsRoot               : True
OwnDuration          : 00:00:00
BlockContainer       : @{Content=C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1; Type=File}
FrameworkDuration    : 00:00:00.5209437
Root                 : @{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.2508997; ExecutedAt=9/28/2019 9:19:08 AM; EachTestTeardown=; Id=; ShouldRun=True; 
                       PluginData=System.Collections.Hashtable; ScriptBlock=; AggregatedPassed=False; Tests=System.Collections.Generic.List`1[System.Object]; Name=Root; 
                       FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=True; OwnDuration=00:00:00; BlockContainer=; FrameworkDuration=00:00:00; Root=; OneTimeBlockTeardown=; 
                       Blocks=System.Collections.Generic.List`1[System.Object]; Exclude=False; OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=; EachTestSetup=; 
                       EachBlockTeardown=; Last=True; StandardOutput=; Tag=; Parent=; Duration=00:00:00; ErrorRecord=System.Collections.Generic.List`1[System.Object]}
OneTimeBlockTeardown : 
Blocks               : {@{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.0230244; ExecutedAt=9/28/2019 9:19:09 AM; EachTestTeardown=; Id=0; ShouldRun=True; 
                       PluginData=System.Collections.Hashtable; ScriptBlock=
                           it 'Should be false' {
                               $true | Should be $false
                           }
                       
                           it 'Should be true' {
                               $true | should be $true
                           }
                       ; AggregatedPassed=True; Tests=System.Collections.Generic.List`1[System.Object]; Name=simple test; FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=False; 
                       OwnDuration=00:00:00.0137001; BlockContainer=; FrameworkDuration=00:00:00.3347260; Root=; OneTimeBlockTeardown=; Blocks=System.Collections.Generic.List`1[System.Object]; Exclude=False; 
                       OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=System.String[]; EachTestSetup=; EachBlockTeardown=; Last=True; StandardOutput=; Tag=System.String[]; 
                       Parent=; Duration=00:00:00.0137001; ErrorRecord=System.Collections.Generic.List`1[System.Management.Automation.ErrorRecord]}}
Exclude              : False
OneTimeTestTeardown  : 
Executed             : True
OneTimeBlockSetup    : 
EachBlockSetup       : 
EachTestSetup        : 
EachBlockTeardown    : 
Duration             : 00:00:00.0137001
ErrorRecord          : {}

I have written quite some automation around the object that -Passthru outputs in v3 / v4 and I was wondering if the object currently returned in v5 would be definitive one?

as much as I love all that extra information we get on the details of every run, I miss some of the information I have been used to through the different versions of pester.
Although that going up a major version we allow our selfs to introduce some breaking changes, I am a bit afraid of how much of my automation and CI checks I will need to change.

Also, it seems like that the information contained in the v5 object contains the details of the internal PesterRun.

Discussion

The part that would be really missing I think is the header of the v4 object.

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2968309
TestResult: 

I know that my self (and quite a lot of people I work with) use something like this to see if all tests passed.

$PesterRun = Invoke-Pester .\SimpleTest.ps1 -PassThru

If($PesterRun.FailedCount -ge 1){
  # Query failed pestertests from $PesterRun.TestsResults
}

This might be a naïve thought, but would it make sense to simply add a property to the existing v4 object, and put the new object of v5 in that one?
Something like RunDetails perhaps ?

Implementing the -Passthru as described above, could result in a -Passthruobject that would look like this

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2642866
TestResult        : {@{ErrorRecord=Expected $false, but got $true.; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=False; Show=All; 
                    FailureMessage=Expected $false, but got $true.; Time=00:00:00.0105793; Name=Should be false; Result=Failed; Context=; StackTrace=at <ScriptBlock>, 
                    C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1: line 3
                    3:         $true | Should be $false}, @{ErrorRecord=; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=True; Show=All; 
                    FailureMessage=; Time=00:00:00.0055245; Name=Should be true; Result=Passed; Context=; StackTrace=}}
RunDetails        : 
                    
                    Content              : C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1
                    Type                 : File
                    PSTypename           : ExecutedBlockContainer
                    OneTimeTestSetup     : 
                    Focus                : False
                    DiscoveryDuration    : 00:00:00.2698095
                    ExecutedAt           : 9/28/2019 9:25:02 AM
                    EachTestTeardown     : 
                    Id                   : 
                    ShouldRun            : True
                    PluginData           : {}
                    ScriptBlock          : 
                    AggregatedPassed     : False
                    Tests                : {}
                    FrameworkData        : {PreviouslyGeneratedTests, PreviouslyGeneratedBlocks}
                    Passed               : True
                    IsRoot               : True
                    OwnDuration          : 00:00:00
                    BlockContainer       : @{Content=C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1; Type=File}
                    FrameworkDuration    : 00:00:00.4905946
                    Root                 : @{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.2698095; ExecutedAt=9/28/2019 9:25:02 AM; EachTestTeardown=; Id=; ShouldRun=True; 
                                           PluginData=System.Collections.Hashtable; ScriptBlock=; AggregatedPassed=False; Tests=System.Collections.Generic.List1[System.Object]; Name=Root; 
                                           FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=True; OwnDuration=00:00:00; BlockContainer=; FrameworkDuration=00:00:00; Root=; OneTimeBlockTeardown=; 
                                           Blocks=System.Collections.Generic.List1[System.Object]; Exclude=False; OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=; 
                    EachTestSetup=; 
                                           EachBlockTeardown=; Last=True; StandardOutput=; Tag=; Parent=; Duration=00:00:00; ErrorRecord=System.Collections.Generic.List1[System.Object]}
                    OneTimeBlockTeardown : 
                    Blocks               : {@{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.0299721; ExecutedAt=9/28/2019 9:25:03 AM; EachTestTeardown=; Id=0; ShouldRun=True; 
                                           PluginData=System.Collections.Hashtable; ScriptBlock=
                                               it 'Should be false' {
                                                   True | Should be False
                                               }
                                           
                                               it 'Should be true' {
                                                   True | should be True
                                               }
                                           ; AggregatedPassed=True; Tests=System.Collections.Generic.List1[System.Object]; Name=simple test; FrameworkData=System.Collections.Hashtable; Passed=True; 
                    IsRoot=False; 
                                           OwnDuration=00:00:00.0153029; BlockContainer=; FrameworkDuration=00:00:00.3050340; Root=; OneTimeBlockTeardown=; 
                    Blocks=System.Collections.Generic.List1[System.Object]; Exclude=False; 
                                           OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=System.String[]; EachTestSetup=; EachBlockTeardown=; Last=True; StandardOutput=; 
                    Tag=System.String[]; 
                                           Parent=; Duration=00:00:00.0153029; ErrorRecord=System.Collections.Generic.List1[System.Management.Automation.ErrorRecord]}}
                    Exclude              : False
                    OneTimeTestTeardown  : 
                    Executed             : True
                    OneTimeBlockSetup    : 
                    EachBlockSetup       : 
                    EachTestSetup        : 
                    EachBlockTeardown    : 
                    Duration             : 00:00:00.0153029
                    ErrorRecord          : {}

Like that we won't break code that is strongly based the structure of the v4 object, and still offer new functionality to the Pester end users.

[nohwnd]

There is a function that summarizes the new object to the old one. It does not do everything exactly yet but it should in the future. It’s probably not published right now but it is used internally to get the summary at the end of the run. That is imho the way to go. Just put the adapter function in the pipeline after Invoke-Pester and before your existing build infrastructure and you are good to go.

[BoSorensen]

Could you also mention the breaking change to Should -Throw, it was introduced in #1003 and means that the substring match has been replaced with the -like operator: link to the v4 docs https://pester.dev/docs/usage/assertions for reference.

I guess it means that it is not possible to use Should -Throw that works both for v4 and v5, if you want to do substring matching.

Perhaps #675 could be reconsidered ?

[nohwnd]

@BoSorensen Okay gotcha. I think that #675 or something similar would be useful for migration where you can run your test suite in a mixed mode. It would probably be difficult to merge the test results, but I have a Legacy result converter, so maybe not. 🙂

[nohwnd]

@Stephanevg Coverter to legacy object will be merged shortly. #1472

[sdecker]

Are all the things listed in "Breaking changes" expected to be that way in the final release? It feels like many are "known issues". But some are actual breaking changes. If so could you please separate this section into two?

For example, not an exhaustive list
True breaking changes: these seems like they are expected not to work going forward

• Legacy syntax Should Be (without -) is removed, see Migrating from Pester v3 to v4
• Assert-VerifiableMocks was removed, Assert-MockCalled was renamed (but aliases exist), see Should -Invoke
• Assert-VerifiableMocks was removed, Assert-MockCalled was renamed (but aliases exist), see Should -Invoke

Really should be known issues: And these seem like known issues expected to be resolved or improved before final release

• Set-ItResult is published but does not work
• Code coverage report is not available.
• Mocks and Discovery are slow (whole execution is about 40% slower than on v4)

As it stands it's hard to make a business case of the value of moving to Pester 5 without knowing which are permanent.

[nohwnd]

@sdecker good point! I expect some of the known issues to be solved before 5.0 and some before 5.1, will, take that into consideration as well :)

[nohwnd]

@sdecker done 🙂

[dbafromthecold]

@nohwnd this may be me missing something, but has the -Script parameter for Invoke-Pester been removed?

Just tried updating the module to RC 5 and am getting...

A parameter cannot be found that matches parameter name 'Script'

When running something like...

Invoke-Pester -Script @{Path = "C:\pestertest\pestertest1.ps1";
Parameters = @{Param1 = $Value1;}}
-TestName 'MyPesterTest'
-PassThru -Show None

I checked the breaking features of the docs and didn't see it mentioned there. The function itself has -Script in the examples but it's not declared.

[nohwnd]

@dbafromthecold you are right the -Script does not exist anymore. It did too much stuff. It is called -Path now and accepts paths. The parametrized script running is not implemented and is documented here. #1485

I added -Script to breaking changes list.

[dbafromthecold]

@nohwnd thanks! will get updating my scripts and testing.

[bergmeister]

@nohwnd I find the guidance to put everything that used to not be controlled into a BeforeAll block confusing because of the following scenario:
At the start a module with test helper functions gets imported (not within Pester controlled blocks initially). At first I just moved all the existing test initialization into a BeforeAll block but then inside the Describe block, the imported function is not available any more, therefore I had to move the bit with the module import outside of the BeforeAll block for it to work again. What do you think?

[nohwnd]

@bergmeister yeah that one is confusing. I saw you do that in PSScriptAnalyzer (worked recently a bit more on moving it to v5, not sure if I pushed into my branch). The original readme for alpha2 had guidance around this. And I did not come up with a good pattern for this yet. I experimented with BeforeDiscovery { }, and had Anywhere { } and Add-Dependency {} blocks before. All with different timings when they will run. But the way I ran the tests also changed two times.

Anyway, describing it in an article is probably the best thing to do now, because there are more situations where you might want to put your code outside of pester blocks.

[Stephanevg]

Ok, sorry in advance for the HUGE spam:

Hi,

I wanted to share some feedback regarding the experience of migrating to Pester v5.
I took a simple test of PSTHML https://github.com/Stephanevg/PSHTML/blob/master/Tests/link.tests.ps1

And tried to migrate it.

I had basically two things I needed to do:

1. Adding all the 'Arrange' code into the a beforeAll
2. I had to add the parameter names to the IT block

Old version

Describe "Testing link" {


        $Class = "MyClass"
        $Id = "MyID"
        $Style = "Background:green"
        $href = "woop"
        $rel = "author"
        $CustomAtt = @{"MyAttribute1"='MyValue1';"MyAttribute2"="MyValue2"}
        $string = link -href $href -rel $rel  -Attributes $CustomAtt -Style $Style -Class $class -id $id

        if($string -is [array]){
            $string = $String -join ""
        }

        it "Should contain opening and closing tags" {
            $string -match '^<link.*>' | should be $true
            #$string -match '.*</link>$' | should be $true

        }

# More it blocks...
} # End Describe block

Output

PS /Users/stephanevg/Code/PSHTML/Tests> $e = invoke-pester -Path ./link.tests.ps1

Starting test discovery in 1 files.
Discovering tests in /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1.
Found 5 tests. 64ms
Test discovery finished. 79ms


Running tests from '/Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1'
Context Testing PSHTML
 Describing Testing link
   [-] Should contain opening and closing tags 10ms (2ms|7ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:35
   [-] Testing common parameters: Class 6ms (2ms|4ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:42
   [-] Testing common parameters: ID 6ms (2ms|4ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:45
   [-] Testing common parameters: Style 6ms (3ms|4ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:48
   [-] Testing Attributes parameters 8ms (4ms|5ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:56
Tests completed in 176ms
Tests Passed: 0, Failed: 5, Skipped: 0 NotRun: 0

New version Result

Describe "Testing link" {
BeforeAll -Scriptblock {

            $Class = "MyClass"
            $Id = "MyID"
            $Style = "Background:green"
            $href = "woop"
            $rel = "author"
            $CustomAtt = @{"MyAttribute1"='MyValue1';"MyAttribute2"="MyValue2"}
            $string = link -href $href -rel $rel  -Attributes $CustomAtt -Style $Style -Class $class -id $id
    
            if($string -is [array]){
                $string = $String -join ""
            }
        }


        
        it -Name "Should contain opening and closing tags" -test {
            $string -match '^<link.*>' | should -be $true
            #$string -match '.*</link>$' | should be $true

        }
# more IT blocks

} # End describe block

It fixed my test.

I wrote a lot (but like, really A LOT!) of tests without specifying the parameter names.

Pester Configuration Object

I LOVE IT!

Up until I have tested, that is really something that could make us think of migrating to v5.

One small suggestion, is maybe to expose the constructors to the end users via Functions (using a 'hybrid' approach). so that the creation of the Configuration is abstracted from the user.

example:

Function New-PesterConfiguration {
[CmdletBinding()]
Param(
  [System.Io.FileInfo]$Path,
  [String[]]$Tag,
  [String[]]$ExcludeTag
  [Switch]$CodeCoverage
)

If($CodeCoverage){
      $ccstate = $true
   }else{
       $ccstate = $False
    }
$configuration = [PesterConfiguration]@{
    Run = @{
        Path = $Path
    }
    Filter = @{
        Tag = $Tag
        ExcludeTag = $ExcludeTag
    }
    Should = @{
        ErrorAction = $PsBoundParameter.ErrorAction
    }
   
    CodeCoverage = @{
        Enable = $ccstate
    }
}

Return $Configuration

}

Like this the users will be able to benefit from all the guidance a Powershell function can offer when creation a new pester configuration (Comment based help, validate sets etc...).

Logging

I couldn't find how to enable the logging in the readme file.
I think it would be great to have the possibility to save the logs to file.

ConvertTo-Pester4Object

Thanks for adding that @nohwnd 👍
I tested it, and it works like a charm.

ConvertTo-NunitReport

it works really well. I like the new -AsString parameter.
ExportTo-NunitReport works as I would expect as well.

Note: I couldn't find a ConvertTo-JunitReport. It is probably on it's way, but we do rely on the JunitXML document as we use Gitlab, and it only supports that format.
IF there would a new cmdlet ConvertTo-JunitReportto come, I guess a Export-JunitReport should also be implemented. Wouldn't it make sense to Have a generic 'Export-pesterReport' function (or something around these lines) which could persist the following to file:

• NunitXML
• JunitXML
• PesterObject deserialized (CLIXML? / JSon ? )
  So, with a -Format parameter.

Internally, that function could either have it's own logic, or simply call the other ConverTo-J/NUnitReport functions.

I am just raising the thought here, simply to have the discussion now, and avoid having to rename the cmdlets once v5 is officially out.

Overall impression of RC1

I really really really like the Configuration Object. That will definitely be something very usefull. Especially for framework authors I think.

I have to admit that it confuses me a little bit that we have the that small pester message 'Amount of failed / passed tests etc...' AND the pester object that is returned at each run.

Saw this discussion -> #1480
I think adding -Passthru back would make it more clear. It is a pattern that people already know, and less scripts will need to be updated.

I have just tested for regular Unit tests. Moving things to BeforeAllblocks seems to be the right migration path. (Also, It would be great if the migration can be smoothed a little bit with some Migration help scripts.)

I do have another test case I would like to takle, and that is for Framekworks that are written around Pester itself. At work we have a framwork that works 'kind of' like DBAChecks.

I do have the impression that these frameworks will benefit a lot of the refactoring of Pester.

When reading through the v5 readme with my "boss's vision" I don't see that much that can appeal us to migration from v4 to v5.
Of course, there is logging, the discovery, the new structure of the object, the decoupling of commands etc... But in the end, If I just want to use Pester's basic functionality to test my code (written with v4 syntax) v5 doesn't seem to offer something that will want us to really really migration to it, and go through the hassle of migrating ALL our tests (belive, we have a LOT of tests) to v5.

For now, the Pester v5 migration sounds like a super lot of work for me, without any direct benefits (outside of use the latest version of pester).

I can image that we would end up having a double approach:

1. leave our existing Tests written in v4 as is, and call Invoke-Pester from the v4 module .
2. when writing new tests, we would embrace Pester v5 recommendations (beforeAll etc...) , and use Invoke-Pester from the v5 module.

We would need to know in advance for which pester version a Tests.ps1 file is written. Not sure how to get that one to work.

[nohwnd]

@Stephanevg

I had to add the parameter names to the IT block
Why was that necessary? That should not be necessary. Were there errors?

One small suggestion, is maybe to expose the constructors to the end users via Functions (using a 'hybrid' approach). so that the creation of the Configuration is abstracted from the user.

That will come in the future on way or another. Right now each section can be created via a hashtable now, or by creating the whole object using $config = [PesterConfiguration]::Default and then assigning to the different properties. I understand that this is not ideal, but my immediate goal was to expose a simple api that can be used interactively, and then have a complete "api" into which everything else will be translated. So internally I can represent all the configuration via a single object, not matter in which way it is created.

I am planning to add a more intermediate api, which will have more parameters on Invoke-Pester, or possible functions to create the sections of the config. The thing is that I need to think about it a lot to make it usable. And I don't have the bandwidth to do that before 5.0.

That is why I am exposing the "raw" config object because it allows to config everything that is there to configure.

Like this the users will be able to benefit from all the guidance a Powershell function can offer when creation a new pester configuration (Comment based help, validate sets etc...).

Did you see the descriptions on each of the properties? Or that is not discoverable?

ConvertTo-JunitReport.

Is that in v4?

Export-pesterReport

That is what I wanted to avoid. Generic functions that do many things. You are also IMHO unlikely to have both NUnit and JUnit reports in one CI pipeline.

I couldn't find how to enable the logging in the readme file. I think it would be great to have the possibility to save the logs to file.

That would be in the debug section of config. https://github.com/pester/Pester/tree/v5.0#advanced-interface WriteDebugMessages = $true and WriteDebugMessagesFrom = "Filter"

It currently uses Write-Host to write to the screen, so it does not save into a file. And there is a rather huge perf penalty for enabling it. Imho adding -Log param with the basics like "Skip", "Mock", "Discovery", "Filter" would make it way more usable. Again, not enough time to polish all this out :)

I have to admit that it confuses me a little bit that we have the that small pester message 'Amount of failed / passed tests etc...' AND the pester object that is returned at each run.

That is already reverted and in rc2 there is -PassThru again. 🙂

migration can be smoothed a little bit with some Migration help scripts.

There is a script linked in the readme that puts all file setups to BeforeAll, I will try to update it to do the same for the code after Describe but before It

When reading through the v5 readme with my "boss's vision" I don't see that much that can appeal us to migration from v4 to v5.

Give it a try on a new project, and take advantage of stuff like skipping whole blocks based on the platform variables that PowerShell 6 has, or tagging on everything, even child Describes and tests, and mocks that almost never require you to specify the scope.

And then when you go back to v4 you will probably feel annoyed by all of this missing. As I was yesterday 😁

And there is more stuff coming, I have an idea how to make the mocks debuggable, will probably make parallel test running more native and so on. v5 is where new stuff will be happening.

We would need to know in advance for which pester version a Tests.ps1 file is written. Not sure how to get that one to work.

In the config you could define that all your pester5 tests have .Tests5.ps1 extension, there is an option for that in the Run section. That way Pester5 will pick up it's tests, and v4 will pickup the legacy tests.

[nohwnd]

@Stephanevg tagging you one more time, I posted the previous answer too soon. It was not a spam at all, thanks for sharing your experience 👍

[bergmeister]

Around -TestCases, which is now also evaluated at discovery time:
I have a test suite here with multiple, long test cases for different Context blocks. The first solution was to declare variables for the test case arrays but for long and multiple test case objects meaning that test and test case are far apart. I then though of inlining the test case array, but that would meaning that the test name and test scriptblock are far apart.
The final solution that I came up with is using the named parameters of It:

It -TestCases @(
    @{
        foo = 'bar'
    }
    @{
        foo = 'baz'
    }
) -Name 'ItName' -Test {
    $foo | Should -Match 'ba'
}

or

It 'ItName' -Test {
    $foo | Should -Match 'ba'
} -TestCases @(
    @{
        foo = 'bar'
    }
    @{
        foo = 'baz'
    }
)

I think I slightly prefer the last version. Just posting as seeing this pattern might help others

[nohwnd]

Around -TestCases, which is now also evaluated at discovery time:

Does this have any impact on this? I don't see anything that would actually run in your case, it's just a bunch of strings.

The final solution that I came up with is using the named parameters of It:

That looks okay to me, especially if the test is just a couple of lines long, then you can easily see that testcases are specified below.

---

If the problem actually was that the evaluation takes long time, then I think -LazySkip and -LazyTestCases which would take scriptblocks could be introduced.

[bergmeister]

Does this have any impact on this?

It was more around finding the most readable version, especially when touching a lot of code now, not performance

On a related note: Is there an easier/lazier way of using TestCases with just an array of strings instead of an array of hashtables if I have a simple test that only takes an array of test values All I could think of is doing something like this

$testValues= 'foo', 'bar', 'baz'
$testCases = $Instances.ForEach{ @{ _ = $_ } }

It would be nice if Pester shipped with something like an array to hashtable converter with the hashtable key being _ so that one can use the intuitive $_ in the test itself (not sure what to do about the angle syntax in the test name itself)

[nohwnd]

It would be nice if Pester shipped with something like an array to hashtableConverter with the hashtable key being _ so that one can use the intuitive $_ in the test itself (not sure what to do about the angle syntax in the test name itself)

And that is why it does not yet ship with such function, because people want to use it in different ways 🙂

[nohwnd]

Anyways I will start cutting Gherkin from Pester 5, and start renaming to __ where necessary.

[bergmeister]

And that is why it does not yet ship with such function, because people want to use it in different ways 🙂

I only partially agree: I agree that people need the freedom to do it in their way but at the moment, if one supplies e.g. an array of strings to the -TestCases parameter, then Pester fails with a parameter binding error that it cannot convert it to an IDictionary. Therefore having at least one default value of using an object array would be useful. Anyone who want to do some custom logic can still use their own array to hashtable converter. This way I don't see how adding support for arrays in -TestCases would break any existing usage or constrain usage.

How about having a -Whatif on Invoke-Pester to run only the test discovery? This would be useful as a simple build validation that there are no syntax errors, etc. without having to run the entire test suite?

[nohwnd]

having at least one default value of using an object array would be useful

You are right. The problem is that consuming arrays of any type is hard to make practical without inventing a lot of arbitrary rules that are hard to come up with, and then hard to define in code. So sticking with the least amount of ways to provide the test cases which still allow you to do everything, even though you sometimes need to adapt array to hashtable is what I do now. We can continue discussing it here #832 there are some nice ideas in that thread. 🙂

How about having a -Whatif on Invoke-Pester to run only the test discovery?

Internally there is Find-Test. And publishing it as a separate cmdlet or as additional parameter on Invoke-Pester is a decision that prevents me from publishing it. I also want to first get the functionality that current Pester has, without adding a lot of new stuff, because the smaller the api is at the start the better.

[nohwnd]

Anyways I will start cutting Gherkin from Pester 5, and start renaming to __ where necessary.

Wrong thread 😁

[plastikfan]

Hi @nownd, looks like I need to look at this aspect of my code again. I thought the excpetion was occuring as a result of the assertion failure, but it's probably something else.

[majkinetor]

I switched to Pester 5 on latest project, here are my observations so far:

1. Code lenses in VSCode is a killer feature. Entire team is more effective now as they always had to previously do mumbo-jumbo to make current test easier to debug (moving it to a top and putting exit bellow it, skipping contexts before etc.)
2. Tags/Skip on everything was must have - now I can run a selection of tests from the shell without any problem, I can also mark any pester entity with predefined tags and devs can quickly execute them.
3. VSCode integrated console crashes every 10-20 minutes or so, after executing tests. Not a big deal.
4. I miss integrated HTML report such as Format-Pester that can be easily and without any dependencies produced on CI builds. This is particularly problematic now that JUnit format is removed, the one that most CI servers know how to view (although I still prefer just offline report and not depend on 3rd party).
5. I really think tests should have time shown (Get current test name #1611) in the output (so I don't have to look into XML just for it)

All in all, its totally great and much better then 5- versions. I use it INSTEAD dotnet test for CORE REST services. This had some challenges like code coverage and metrics that I had to overcome (not pester related but I will see to publish my InfluxDb metric sender script and Grafana dashboard that tracks behaviors across multiple runs; code coverage turned out to be doable with dotCover CLI).

The only thing that I really miss now are parallel builds.

[nohwnd]

@majkinetor
1 & 2 That is great to hear 🥳
3 Same here, but it seems to be more stable in the Preview version of the plugin, but still pretty flaky.
4 there is proposal for html output I think, but I don't want to integrate yet another tool. Fixing JUnit output is long overdue, but I will hopefully get to it.
5 Maybe I would consider it for the Detailed output. will have to look how it looks like.

[majkinetor]

4 there is proposal for html output I think, but I don't want to integrate yet another tool. Fixing JUnit output is long overdue, but I will hopefully get to it.

Sounds like basic stuff IMO. Its standalone too (doesn't introduce more complexity). I am sure that there are 0 people who would object on having it, plus it has potential to popularize the tool (as nice dashboards always do). I hope to find some time to make one.

[LethiferousMoose]

@nohwnd Based on your comment above:
"An AfterAll block cannot reside above the first Describe block. - yes, but that should be temporary limitation".

Is this something that is coming the future?
What is the best way to migrate tests that currently use a try/finally block for setup/teardown if AfterAll is not yet supported?

[nohwnd]

@LethiferousMoose implemented in #1707 shipped in 5.1.0-beta2

[rcdailey]

I'm struggling with how to share variables between test cases in Pester 5. I do something like this:

$InstallDir = 'C:\Program Files\MyProduct'

Describe "Product Installation" {
    Context "Bin Directory" {
        It "Program exists" {
            Join-Path $InstallDir 'Bin\MyProgram.exe' | Should -Exist
        }
    }
}

Inside the It block, it can't access the value for $InstallDir. If I put the definition of this variable inside Describe.BeforeAll or Describe.BeforeEach, it still doesn't work. How am I supposed to share variables between multiple test cases like this?

[LethiferousMoose]

BeforeAll normally works for me, I do that in a lot of tests. Here's one of mine that works:

Describe 'Write-Logger' {
    BeforeAll {
        [string] $projectRoot = $PSScriptRoot + '/../../../'
        [string] $logPath = $projectRoot + 'log'
        [string] $logFolder = "${PID}_" + $Host.InstanceId.ToString()
        [string] $logFile = '000_Pester_Invoke-Pester.log'
        [string] $logFilePath = Join-Path -Path $logPath -ChildPath ($logFolder + '/' + $logFile)
    }

    Context 'Logging enabled' {
        It 'Write with default settings' {
            Write-Logger -Message 'TestDefault1', 'TestDefault2'
            Start-Sleep -Seconds 1

            Test-Path $logFilePath | Should -Be $true
            [string[]] $fileContent = Get-Content $logFilePath
            $fileContent.Count | Should -Be 2
            $fileContent[0] | Should -Match '\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} TestDefault1'
            $fileContent[1] | Should -Match '\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} TestDefault2'
        }
    }
}

[nohwnd]

@rcdailey put $InstallDir = 'C:\Program Files\MyProduct' into BeforeAll as @LethiferousMoose suggests:

BeforeAll { 
    $InstallDir = 'C:\Program Files\MyProduct'
}

[chscott]

I maintain a module, and I like to be able to both test individual functions as I'm modifying them and then test those same functions once they are bundled into a module. Prior to v5, I could have both test modes in the same file, but I've been unable to make that work due to the new scoping in v5. Now I have two test files like these:

# Load the module
Import-Module 'MyModule.psm1'

InModuleScope 'MyModule' {
  Describe 'MyFunction' {
    # Tests
  }
}

Describe 'MyFunction' {
  BeforeAll {
    # Load the script
    . 'MyFunction.ps1'
  }

  # Tests
}

This arrangement works for me, but I have to maintain the tests in two different files, which is a pain. Is it possible to eliminate this type of duplication?

[rjmholt]

Finally piling into this issue with a question myself...

I've got a set of tests where I want to generate the testcases based on files that are checked in to the repo, i.e. I can't declare them statically:

    # Test cases come from the examples folder
    $exampleDir = (Resolve-Path "$PSScriptRoot/../../examples").Path
    $examples = Get-ChildItem -LiteralPath $exampleDir

    $testCases = $examples | ForEach-Object {
        $basePath = $_.FullName
        $scriptPath = Join-Path -Path $basePath -ChildPath 'test.ps1'
        $templatePath = Join-Path -Path $basePath -ChildPath 'template.json'

        if ((Test-Path -LiteralPath $scriptPath) -and (Test-Path -LiteralPath $templatePath))
        {
            @{ Name = $_.Name; ScriptPath = $scriptPath; TemplatePath = $templatePath }
        }
    }

(This is because I want my examples to be continuously validated against my module).

But trying to declare $testCases anywhere (BeforeAll at the top level in the file, BeforeAll in Describe, directly in Describe) will evaluate the code, but not pass it through to my test. I assume this is a consequence of the discovery phase concept.

Is there a way for me to dynamically generate test cases, or otherwise dynamically produce tests, in Pester 5?

[nohwnd]

@chscott please show me how you did it before. That sounds like something you should definitely be able to do in v5 :)

@rjmholt you are looking for the mythical Generating tests section of my usage docs PR, that section I am yet to write.

Here are some common gotchas and the discovery process explained

But there are few examples of generating tests already in our tests. This one being most complete. But there are more in that file.

Pester/tst/Pester.RSpec.ts.ps1

Lines 1189 to 1240 in d689c3c

	t "Data provided to container are accessible in ForEach on each level" {
	$scenarios = @(
	@{
	Scenario = @{
	Name = "A"
	Contexts = @(
	@{
	Name = "AA"
	Examples = @(
	@{ User = @{ Name = "Jakub"; Age = 31 } }
	@{ User = @{ Name = "Tomas"; Age = 27 } }
	)
	}
	@{
	Name = "AB"
	Examples = @(
	@{ User = @{ Name = "Peter"; Age = 30 } }
	@{ User = @{ Name = "Jaap"; Age = 22 } }
	)
	}
	)
	}
	}
	@{
	Scenario = @{
	Name = "B"
	Contexts = @{
	Name = "BB"
	Examples = @(
	@{ User = @{ Name = "Jane"; Age = 25 } }
	)
	}
	}
	}
	)
	$sb = {
	param ($Scenario)
	Describe "Scenario - <name>" -ForEach $Scenario {
	Context "Context - <name>" -ForEach $Contexts {
	It "Example - <user.name> with age <user.age> is less than 35" -ForEach $Examples {
	$User.Age | Should -BeLessOrEqual 35
	}
	}
	}
	}
	$container = New-PesterContainer -ScriptBlock $sb -Data $scenarios
	$r = Invoke-Pester -Container $container -PassThru -Output Detailed

In that example I am combining all three things together: passing external data, and generating describes & contexts, and generating Its. And also the new ability to expand dotted code, and passing any array to -Foreach / -TestCases.

More examples are also in these release notes: https://github.com/pester/Pester/releases/tag/5.1.0-beta1 Just know that New-TestContainer was renamed to New-PesterContainer in 5.1.0-rc1

[chscott]

@chscott please show me how you did it before. That sounds like something you should definitely be able to do in v5 :)

I no longer have an example of it that I could readily find, so let me show what doesn't work in v5. I'm not married to this approach; if there's a better way to accomplish what I'm trying to do, I'm all ears.

$test = @{
  Name        = 'returns True if the arrays are equal'
  ScriptBlock = {
    $test = @(
      'element1',
      'element2'
    )
    $expected = @(
      'element1',
      'element2'
    )
    Assert-ArrayEquality $test $expected | Should -Be $true
  }
}

$describe = @{
  Name        = 'Assert-ArrayEquality'
  ScriptBlock = {
    Write-Host "Inside Describe: test.Name is '$($test.Name)'"

    It -Name $test.Name -Test $test.ScriptBlock
  }
}

if ($global:MODULE) {
  Write-Host "Before InModuleScope: test.Name is '$($test.Name)'"
  InModuleScope $global:MODULE {
    Describe -Name $describe.Name -Fixture $describe.ScriptBlock
  }
} else {
  Write-Host "Before Describe: test.Name is '$($test.Name)'"
  Describe -Name $describe.Name -Fixture $describe.ScriptBlock
}

The central challenge is trying to reuse the same Describe script block in and out of a module context. When $global:MODULE is not set, things work fine:

Starting discovery in 1 files.
Discovering in C:\src\module\common\test\unit\Assert-ArrayEquality.Tests.ps1.
Before Describe: test.Name is 'returns True if the arrays are equal'
Inside Describe: test.Name is 'returns True if the arrays are equal'
Found 1 tests. 8ms
Discovery finished in 12ms.

Running tests from 'C:\src\module\common\test\unit\Assert-ArrayEquality.Tests.ps1'
Describing Assert-ArrayEquality
  [+] returns True if the arrays are equal 1ms (1ms|1ms)
Tests completed in 78ms
Tests Passed: 1, Failed: 0, Skipped: 0 NotRun: 0

But when I set $global:MODULE, I lose access to the variables.

Starting discovery in 1 files.
Discovering in C:\src\module\common\test\unit\Assert-ArrayEquality.Tests.ps1.
Before InModuleScope: test.Name is 'returns True if the arrays are equal'
System.Management.Automation.ParameterBindingValidationException: Cannot bind argument to parameter 'Name' because it is an empty string.

I have the same issue if I try to move the InModuleScope portion to just the It level:

$test = @{
  Name        = 'returns True if the arrays are equal'
  ScriptBlock = {
    $test = @(
      'element1',
      'element2'
    )
    $expected = @(
      'element1',
      'element2'
    )
    Assert-ArrayEquality $test $expected | Should -Be $true
  }
}

$describe = @{
  Name        = 'Assert-ArrayEquality'
  ScriptBlock = {
    Write-Host "Inside Describe: test.Name is '$($test.Name)'"

    if ($global:MODULE) {
      InModuleScope $global:MODULE {
        It -Name $test.Name -Test $test.ScriptBlock
      }
    } else {
      It -Name $test.Name -Test $test.ScriptBlock
    }
  }
}

Describe -Name $describe.Name -Fixture $describe.ScriptBlock

[chscott]

I got some clues from #1603 that I was able to use to get this working.

$test = @{
  Name        = 'returns True if the arrays are equal'
  ScriptBlock = {
    $test = @(
      'element1',
      'element2'
    )
    $expected = @(
      'element1',
      'element2'
    )
    Assert-ArrayEquality $test $expected | Should -Be $true
  }
}

$describe = @{
  Name        = 'Assert-ArrayEquality'
  ScriptBlock = {
    Write-Host "Inside Describe: test.Name is '$($test.Name)'"
    It -Name $test.Name -Test $test.ScriptBlock
  }
}

if ($global:MODULE) {
  Write-Host "Before InModuleScope: test.Name is '$($test.Name)'"
  InModuleScope $global:MODULE -Parameters @{ Name = $describe.Name; Fixture = $describe.ScriptBlock } {
    param($Name, $Fixture)
    Describe -Name $Name -Fixture $Fixture
  }
} else {
  Write-Host "Before Describe: test.Name is '$($test.Name)'"
  Describe Describe -Name $describe.Name -Fixture $describe.ScriptBlock
}

[nohwnd]

@chscott Seems a bit complicated. Seems to me that you just need to wrap the whole file in InModuleScope when the $globalModule is defined, or don't wrap it when executing normally. So I would just put all the test code in the scriptblock and then either invoke it in module scope or not. This way you can use the normal Pester syntax and avoid juggling with variables.

$script:aaa = "script"
Get-Module m | Remove-Module
New-Module -Name m -ScriptBlock {
    $script:aaa = "module"
} -PassThru | Import-Module

$scriptBlock = { 
    Describe "Assert-ArrayEquality" {
        It "I run outside of module because the script variable is not shadowed by the module variable" {
            $script:aaa | Should -Be "script"
        }

        It "I run in module because the script variable is shadowed by the module variable" {
            $script:aaa | Should -Be "module"
        }
    }
}

# outside module
$global:MODULE = $null
if ($global:Module) { 
    InModuleScope -ModuleName $global:Module -ScriptBlock $scriptBlock
}
else { 
    & $scriptBlock
}

# in module
$global:MODULE = "m"
if ($global:Module) { 
    InModuleScope -ModuleName $global:Module -ScriptBlock $scriptBlock
}
else { 
    & $scriptBlock
}

# output: 

Found 4 tests. 14ms
Discovery finished in 24ms.

Running tests from 'C:\Users\jajares\Desktop\in-out.tests.ps1'
Describing Assert-ArrayEquality
  [+] I run outside of module because the script variable is not shadowed by the module variable 7ms (3ms|4ms)
  [-] I run in module because the script variable is shadowed by the module variable 9ms (8ms|1ms)
   Expected strings to be the same, but they were different.
   String lengths are both 6.
   Strings differ at index 0.
   Expected: 'module'
   But was:  'script'
   at $script:aaa | Should -Be "module", C:\Users\jajares\Desktop\in-out.tests.ps1:14
   at <ScriptBlock>, C:\Users\jajares\Desktop\in-out.tests.ps1:14

Describing Assert-ArrayEquality
  [-] I run outside of module because the script variable is not shadowed by the module variable 8ms (5ms|3ms)
   Expected strings to be the same, but they were different.
   String lengths are both 6.
   Strings differ at index 0.
   Expected: 'script'
   But was:  'module'
   at $script:aaa | Should -Be "script", C:\Users\jajares\Desktop\in-out.tests.ps1:10
   at <ScriptBlock>, C:\Users\jajares\Desktop\in-out.tests.ps1:10
  [+] I run in module because the script variable is shadowed by the module variable 4ms (2ms|1ms)
Tests completed in 409ms
Tests Passed: 2, Failed: 2, Skipped: 0 NotRun: 0

[plastikfan]

I'm having problems with InModuleScope, trying to test an internal module function, so I'm making use of InModuleScope as follows, for function under test format-ColouredLine:

  BeforeAll {
    Get-Module Elizium.Loopz | Remove-Module
    Import-Module .\Output\Elizium.Loopz\Elizium.Loopz.psm1 `
      -ErrorAction 'stop' -DisableNameChecking;

    InModuleScope Elizium.Loopz {
      [string]$script:LineKey = 'LOOPZ.HEADER-BLOCK.LINE';
      [string]$script:CrumbKey = 'LOOPZ.HEADER-BLOCK.CRUMB-SIGNAL';
      [string]$script:MessageKey = 'LOOPZ.HEADER-BLOCK.MESSAGE';

      function show-result {
        param(
          [string]$Ruler,
          [object[]]$Snippets
        )
        Write-Host "$Ruler";
        Write-InColour -TextSnippets $Snippets;
      }
    }
  }

  BeforeEach {
    InModuleScope Elizium.Loopz {
      [string]$script:_ruler = '........................................................................................................................................';
    }
  }
...

  Context 'given: Plain Line' {
    It 'should: create coloured line without crumb or message' -Tag 'Current' {
      InModuleScope Elizium.Loopz {
        [System.Collections.Hashtable]$passThru = @{
          'LOOPZ.HEADER-BLOCK.LINE' = $LoopzUI.EqualsLine;
        }

        $line = format-ColouredLine -PassThru $passThru -LineKey $LineKey -CrumbKey $CrumbKey;
        $line[0][0] | Should -BeExactly $LoopzUI.EqualsLine;
        show-result -Ruler $_ruler -Snippets $line;
      }
    }
  } # given: Plain line

and running this test results in

CommandNotFoundException: The term 'show-result' is not recognized as a name of a cmdlet, function, script file, or executable program.

Note how my other variables are correctly bound in LineKey, CrumbKey and MessageKey.

Also, just to note, the only thig failing in this test is accessing the function show-result, which has been defined inside the module scope and being invoked also from inside module scope. The rules pertaining to variables and functions do not appear to be the same with regards to InModuleScope. So how can I define test function show-result, so that it is accessible to test code?

I suppose to simplify further:

  Context 'ask: question' {
    It 'should: just work' -Tag 'Current' {
      InModuleScope Elizium.Loopz {
        show-result -Ruler '.........................................' -Snippets @(@('First Snippet', 'red'), @('Second Snippet', 'blue'));
      }
    }
  }

... which fails for the exact same reason.

[plastikfan]

I've finally resolved this problem of mine. If you prefix the function with script:, ie

      function script:show-result {
        param(
          [string]$Ruler,
          [object[]]$Snippets
        )
        Write-Host "$Ruler";
        Write-InColour -TextSnippets $Snippets;
      }

then this fixes the issue. This was a pure guess on my part and could do with an explanation and being documented.

Actually, using the scope specifier 'script:' works when you're running InModuleScope. I discovered that when testing a public Module function (and hence not using InModuleScope), any test function declared inside BeforeAll/After or inside It, which needs to be invoked, should be defined in the global scope; script scope does not work in this scenario.

[hugh-martin]

First, thanks very much for the inclusion of -ForEach in v5.1.1!! I'm starting to migrate my operational validation tests to v5.1x and it's going well. One thing I'm finding with v5 is that I'm using hashtables more and more to move data around to make things easier. One issue I'm running into is validating the hashtable data coming into a function via a parameter. Right now, I'm looping through the keys to make sure they are what's expected and then looping through the values and validating the data for each. In short, it's a pain.

Is it safe to assume that I cannot pass a PowerShell class to Invoke-Pester's -TestCases param? If so, what's the best way to validate hashtable data that's coming into a function? Thanks.

[hugh-martin]

Here's a situation I'm running into that I'm not sure how best to convert to v5. In one of our operational validation tests, we're looping through VMware datastores. In that process, we're collecting various properties of each datastore. Based on those properties, we currently have If / Else statements that determine which It statements to run. This doesn't work as is in v5. I realize I could move the If / Else statements into the BeforeAll scriptblock and generate new hashtables as needed and use TestCases to go back through the datastores, but that doesn't seem very efficient. Suggestions? Thanks.

Here's a bit of the code from each section:

BeforeDiscovery {
    
    #...
    #Create an array of all datastores
    $script:arrDatastores = Get-Datastore | Sort-Object Name
    #...
    
}

Describe "Storage checks" -ForEach $arrDatastores {
    
    BeforeAll {
        $objDatastore = $_
        
        [string]$dsName = $objDatastore.Name
        [int]$dsFreeGB = $objDatastore.FreeSpaceGB
        
        #Check availability and accessibility
        if ($objDatastore.State -eq "Available" -and $objDatastore.Accessible -eq $true) {
            $bolDatastoreIsHealthy = $true
        }
        else {
            $bolDatastoreIsHealthy = $false
        }
        
        #Check free space (skip datastores that don't have VMs on them)
        
        #Get all the VMs on this datastore
        $objVMsOnThisDatastore = $objDatastore | Get-VM
        
        
        #Determine how many of the VMs on this datastore are connected only via the virtual CD drive
        [boolean]$VMDisksOnThisDatastore = $false
        foreach ($objVMOnThisDatastore in $objVMsOnThisDatastore) {
            $objHardDisks = $objVMOnThisDatastore | Get-HardDisk
            foreach ($objHardDisk in $objHardDisks) {
                $DiskDatastoreName = ($objHardDisk | Get-Datastore).Name
                if ($DiskDatastoreName -eq $dsName) {
                    $VMDisksOnThisDatastore = $true
                }
            }
            #Break out of the loop if any disks are found; no need to continue searching
            if ($VMDisksOnThisDatastore) {
                Break
            }
        }
        
    } #End BeforeAll
    
    
    Context "Datastore availability and free space checks (vCenter level)" {

        it "Datastore <dsName> should be both available and accessible." {
            $bolDatastoreIsHealthy | Should -Be $true
        }
        
        #Here's where the trouble begins...

        if ($VMDisksOnThisDatastore) {
            #This datastore has VM disks on it
            It "Datastore <dsName> (containing one or more VMs with <dsFreeGB> GB free space) should have more than <MinVMDatastoreFreeSpaceGB> GB free" {
                $dsFreeGB | Should -BeGreaterThan $script:MinVMDatastoreFreeSpaceGB
            }
        }
        else {
            #This datastore does NOT have VM disks on it
            It "Datastore <dsName> (containing no VMs with <dsFreeGB> GB free space) should have more than <MinUtilDatastoreFreeSpaceGB> GB free" {
                $dsFreeGB | Should -BeGreaterThan $MinUtilDatastoreFreeSpaceGB
            }
        }
    }

    Context "Next group of tests for this datastore..." {
        #...
    }

}

[fflaten]

@hugh-martin The problem is that your if-test is running during Discovery, while you variable is set during Run (when BeforeAll is executed). To minimize changes, you could move the if-test inside the tests and skip the test when it's not needed, e.g. if ($VMDiskOnThisDatastore) { Set-ItResult -Skipped -Because "Not relevant" } and vice versa.

However the "proper" way would be to move data-collection/logic related to test-generation into the discovery phase. Ex.

BeforeDiscovery {
    $numbersToTest= 1..5 | % { [pscustomobject]@{Number = $_ } }
}

Describe 'Test <_.Number>' -ForEach $numbersToTest {
    BeforeAll {
        #$MyNumber from BeforeAll doesn't exist here
        $number = $_.Number
    }

    BeforeDiscovery {
        #$number from BeforeAll doesn't exist here
        $MyNumber = $_.Number
        $isEvenNumber = $MyNumber % 2 -eq 0
    }

    Context 'Using If' {
        if($isEvenNumber) {
            It 'Test even number <number>' {
                $number % 2 | Should -Be 0
            }
        } else {
            It 'Test odd number <number>' {
                $number % 2 | Should -Be 1
            }
        }
    }

    Context 'Using TestCases (ugly)' {
        It 'Test even number <number>' -TestCases (@($isEvenNumber) -eq $true) {
            $number % 2 | Should -Be 0
        }
        It 'Test odd number <number>' -TestCases (@($isEvenNumber) -eq $false) {
            $number % 2 | Should -Be 1
        }
    }

    Context 'Using Skip switch (skipped tests are shown in output/report)' {
        It 'Test even number <number>' -Skip:($isEvenNumber -eq $false) {
            $number % 2 | Should -Be 0
        }
        It 'Test odd number <number>' -Skip:($isEvenNumber -eq $true) {
            $number % 2 | Should -Be 1
        }
    }
}

[hugh-martin]

I'm guessing this pattern won't work. I'm trying to loop through each VMware cluster, then loop through each host in that cluster, so I'm using a -ForEach on the Describe (for each cluster) and then attempting to use a -TestCases on the It (for each host in that cluster). It's not working, presumably since both will use $_. Can someone suggest another way to do this in Pester 5? In Pester 4, I just used nested foreach loops.

Describe "Datastore count checks (host level)" -ForEach (Get-Cluster | Sort-Object) {
    
    BeforeEach {
        $strClusterName = $_.Name
        [int]$intClusterDatastoreCount = (Get-Cluster $strClusterName | Get-Datastore | Measure-Object).Count
        $objClusterHosts = Get-Cluster $strClusterName | Get-VMHost | Sort-Object Name
    }
    
    It "Host <_.Name> should have the same number of datastores as its cluster" {
        [int]$intHostDatastoreCount = ($_ | Get-Datastore | Measure-Object).Count
        $intHostDatastoreCount | Should -BeExactly $intClusterDatastoreCount
    } -TestCases $objClusterHosts
}

[hugh-martin]

With respect to the location of the BeforeEach, this pattern makes sense to me, since the BeforeEach is inside the -ForEach loop:

BeforeDiscovery {
    $collection = @("one","two","three")
}

Describe "describe" -ForEach ($collection) {
    
    BeforeEach {
        $strNum = $_
    }
    
    It "<strNum> should be three" {
        $strNum | Should -Be "three"
    }
}

However, this does not make sense to me. In this case, I would expect the BeforeEach to run just one time, since the Describe does not have a -ForEach and the BeforeEach is outside the -TestCases loop on the It statement. However, it works, and putting the BeforeEach inside the It statement definitely does not. Is this the intended behavior?

BeforeDiscovery {
    $collection = @("one","two","three")
}

Describe "describe" {
    
    BeforeEach {
        $strNum = $_
    }
    
    It "<strNum> should be three" {
        $strNum | Should -Be "three"
    } -TestCases $collection
}

[fflaten]

I agree it can feel a bit confusing. Foreach/testcases was probably designed to be used with an array of hashtables in which case you'd get variables per dictionary key in the current object (hashtable) and not rely on $_. When using hashtables it would be very clear when you would overwrite a variable since you reused a key-name. Now that you can use an array of objects directly, then $_ will be the current object (ex. the whole hashtable-object or a string in you sample).

That being said, it's easy enough to explain. When you use It -TestCases ... you actually generate x duplicate tests with different data-input (each testcase-object). BeforeEach runs before each test so it's natural to expect $_ to be the current testcase-object just like it will be inside the test. The docs should probably have a note about this.

BeforeAll however is executed once before the first test in a describe/context. At this point the latest "current object" is the current object provided in the describe/context foreach. Having that in mind, if you need to use the context/describe object in BeforeEach, you would capture it to a variable in BeforeAll, ex. BeforeAll { $myDescribeObject = $_ }; BeforeEach { Write-Host "BeforeEach does something to $myDescribeObject" }.

This is untested, but my immediate suggestion to the scenario in your previous problem would be:

BeforeDiscovery {
    $clusters = Get-Cluster | Sort-Object
}

Describe "Datastore count checks (host level)" -ForEach $clusters {

    BeforeDiscovery {
        # Setup required for TestCases / data-driven tests
        $objClusterHosts = Get-Cluster $_.Name | Get-VMHost | Sort-Object Name
    }

    BeforeAll {
        # Prepare cluster-specific values
        $strClusterName = $_.Name
        [int]$intClusterDatastoreCount = (Get-Cluster $strClusterName | Get-Datastore | Measure-Object).Count
    }

    It "Host <_.Name> should have the same number of datastores as its cluster" {
        [int]$intHostDatastoreCount = ($_ | Get-Datastore | Measure-Object).Count
        $intHostDatastoreCount | Should -BeExactly $intClusterDatastoreCount
    } -TestCases $objClusterHosts
}

[hugh-martin]

@fflaten That worked like a charm. Thank you very much! I have a couple other use cases to apply this to.

[Glober777]

Hi, I've been looking for the details regarding the duration values that are being reported by Pester v5 in its console output. For example, when I run the test, I see the output as follows:

...
[+] Some test <duration value 1> (<duration value 2> | <duration value 3>)
...

Can someone please shed some light on what these duration values actually mean?

Thanks!

[fflaten]

@Glober777 It's duration (user|framework). Duration is total duration, user is test code duration and the last is time used by the framework for mock setup etc.

See #1402

[nohwnd]

I enabled discussions for the repo. This one is not needed anymore. Closing.