Skip to content

Commit e079f77

Browse files
LarsAsplundLarsAsplund
committed
Added support for dynamic seed generation.
1 parent a3f3dca commit e079f77

34 files changed

+608
-110
lines changed

‎docs/news.d/1089.feature.rst

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Provided a unique base seed to every simulation. From that base seed any number of unique seed can derived
2+
to initialize the random number generators in the testbench. Unique seeds increase the test coverage for
3+
randomized tests over time as they are executed repeatedly by the user and the CI. The seed can be overrideen
4+
from the command line with the --seed option in order to reproduce failing tests.

‎docs/run/img/get_seed_and_runner_cfg.html

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
<div class="highlight"><pre><span></span><span class="k">constant</span><span class="w"> </span><span class="n">global_seed</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="n">get_seed</span><span class="p">(</span><span class="n">runner_cfg</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;Optional salt&quot;</span><span class="p">);</span>
2+
</pre></div>

‎docs/run/img/get_seed_wo_runner_cfg.html

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
<div class="highlight"><pre><span></span><span class="n">randomizing_process</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="k">process</span><span class="w"> </span><span class="k">is</span>
2+
<span class="w"> </span><span class="k">variable</span><span class="w"> </span><span class="n">local_seed</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">string_seed_t</span><span class="p">;</span><span class="w"> </span><span class="c1">-- string_seed_t = string(1 to 16)</span>
3+
<span class="k">begin</span>
4+
<span class="w"> </span><span class="n">get_seed</span><span class="p">(</span><span class="n">local_seed</span><span class="p">,</span><span class="w"> </span><span class="n">salt</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">randomizing_process</span><span class="na">&#39;path_name</span><span class="p">);</span>
5+
</pre></div>

‎docs/run/img/get_uniform_seed.html

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
<div class="highlight"><pre><span></span><span class="n">get_uniform_seed</span><span class="p">(</span><span class="n">seed1</span><span class="p">,</span><span class="w"> </span><span class="n">seed2</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;Optional salt&quot;</span><span class="p">);</span>
2+
3+
<span class="n">uniform</span><span class="p">(</span><span class="n">seed1</span><span class="p">,</span><span class="w"> </span><span class="n">seed2</span><span class="p">,</span><span class="w"> </span><span class="n">a_random_value</span><span class="p">);</span>
4+
<span class="n">uniform</span><span class="p">(</span><span class="n">seed1</span><span class="p">,</span><span class="w"> </span><span class="n">seed2</span><span class="p">,</span><span class="w"> </span><span class="n">another_random_value</span><span class="p">);</span>
5+
</pre></div>

‎docs/run/img/list.html

+7-1
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,22 @@
11
<div class="highlight" style="background: #002b36; color: #eee8d5;"><pre style="line-height: 125%; background: #002b36; color: #eee8d5;">> python run.py --list
2+
lib.tb_fail_on_warning.Test that fails on an assert
3+
lib.tb_fail_on_warning.Test that crashes on boundary problems
4+
lib.tb_fail_on_warning.Test that fails on VUnit check procedure
5+
lib.tb_fail_on_warning.Test that a warning passes
26
lib.tb_magic_paths.all
37
lib.tb_minimal.all
48
lib.tb_running_test_case.Test scenario A
59
lib.tb_running_test_case.Test scenario B
610
lib.tb_running_test_case.Test something else
711
lib.tb_run_all_in_same_sim.Test to_string for integer again
812
lib.tb_run_all_in_same_sim.Test to_string for boolean again
13+
lib.tb_seed.all
914
lib.tb_standalone.Test that fails on VUnit check procedure
1015
lib.tb_standalone.Test to_string for boolean
1116
lib.tb_stopping_failure.Test that fails on an assert
1217
lib.tb_stopping_failure.Test that crashes on boundary problems
1318
lib.tb_stopping_failure.Test that fails on VUnit check procedure
19+
lib.tb_stopping_failure.Test that a warning passes
1420
lib.tb_stop_level.Test that fails multiple times but doesn't stop
1521
lib.tb_with_lower_level_control.Test something
1622
lib.tb_with_lower_level_control.Test something else
@@ -19,5 +25,5 @@
1925
lib.tb_with_watchdog.Test that stalls
2026
lib.tb_with_watchdog.Test to_string for boolean
2127
lib.tb_with_watchdog.Test that needs longer timeout
22-
Listed 20 tests
28+
Listed 26 tests
2329
</pre></div>

‎docs/run/img/seed_option.html

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
<div class="highlight" style="background: #002b36; color: #eee8d5;"><pre style="line-height: 125%; background: #002b36; color: #eee8d5;">> python run.py "lib.tb_seed.Test that fails" --seed fb19f3cca859d69c</pre></div>

‎docs/run/img/tb_magic_paths_stdout.html

+5-4
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,19 @@
11
<div class="highlight" style="background: #002b36; color: #eee8d5;"><pre style="line-height: 125%; background: #002b36; color: #eee8d5;">> python run.py -v
22
Starting lib.tb_magic_paths.all
33
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_magic_paths.all_243b3c717ce1d4e82490245d1b7e8fe8797f5e94\output.txt
4+
Seed for lib.tb_magic_paths.all: 3680f2e2321cdac3
45
0 fs - default - INFO - Directory containing testbench: C:/github/vunit/docs/run/src/
56
0 fs - default - INFO - Test output directory: C:/github/vunit/docs/run/src/vunit_out/test_output/lib.tb_magic_paths.all_243b3c717ce1d4e82490245d1b7e8fe8797f5e94/
67
simulation stopped @0ms with status 0
7-
<span style="color: #008000; ">pass</span> (P=1 S=0 F=0 T=1) lib.tb_magic_paths.all (0.5 seconds)
8+
<span style="color: #008000; ">pass</span> (P=1 S=0 F=0 T=1) lib.tb_magic_paths.all (0.6 s)
89

910
==== Summary ==================================
10-
<span style="color: #008000; ">pass</span> lib.tb_magic_paths.all (0.5 seconds)
11+
<span style="color: #008000; ">pass</span> lib.tb_magic_paths.all (0.6 s)
1112
===============================================
1213
<span style="color: #008000; ">pass</span> 1 of 1
1314
===============================================
14-
Total time was 0.5 seconds
15-
Elapsed time was 0.5 seconds
15+
Total time was 0.6 s
16+
Elapsed time was 0.6 s
1617
===============================================
1718
<span style="color: #008000; ">All passed!</span>
1819
</pre></div>

‎docs/run/img/tb_minimal_stdout.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
<div class="highlight" style="background: #002b36; color: #eee8d5;"><pre style="line-height: 125%; background: #002b36; color: #eee8d5;">> python run.py
22
Starting lib.tb_minimal.all
33
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_minimal.all_42aa262c7c96c708ab3f3960f033f2328c642136\output.txt
4-
<span style="color: #008000; ">pass</span> (P=1 S=0 F=0 T=1) lib.tb_minimal.all (0.5 seconds)
4+
<span style="color: #008000; ">pass</span> (P=1 S=0 F=0 T=1) lib.tb_minimal.all (0.6 s)
55

66
==== Summary ==============================
7-
<span style="color: #008000; ">pass</span> lib.tb_minimal.all (0.5 seconds)
7+
<span style="color: #008000; ">pass</span> lib.tb_minimal.all (0.6 s)
88
===========================================
99
<span style="color: #008000; ">pass</span> 1 of 1
1010
===========================================
11-
Total time was 0.5 seconds
12-
Elapsed time was 0.5 seconds
11+
Total time was 0.6 s
12+
Elapsed time was 0.6 s
1313
===========================================
1414
<span style="color: #008000; ">All passed!</span>
1515
</pre></div>

‎docs/run/img/tb_running_test_case_stdout.html

+10-10
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,25 @@
11
<div class="highlight" style="background: #002b36; color: #eee8d5;"><pre style="line-height: 125%; background: #002b36; color: #eee8d5;">> python run.py
22
Starting lib.tb_running_test_case.Test scenario A
33
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_running_test_case.Test_scenario_A_b118fcdf82c6ba5772e038ce7455962692f50c2a\output.txt
4-
<span style="color: #008000; ">pass</span> (P=1 S=0 F=0 T=3) lib.tb_running_test_case.Test scenario A (0.5 seconds)
4+
<span style="color: #008000; ">pass</span> (P=1 S=0 F=0 T=3) lib.tb_running_test_case.Test scenario A (0.6 s)
55

6-
Starting lib.tb_running_test_case.Test scenario B
6+
(22:55:49) Starting lib.tb_running_test_case.Test scenario B
77
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_running_test_case.Test_scenario_B_8cd407ec92589901f9b4594c0f27835478242f2c\output.txt
8-
<span style="color: #008000; ">pass</span> (P=2 S=0 F=0 T=3) lib.tb_running_test_case.Test scenario B (0.5 seconds)
8+
<span style="color: #008000; ">pass</span> (P=2 S=0 F=0 T=3) lib.tb_running_test_case.Test scenario B (0.6 s)
99

10-
Starting lib.tb_running_test_case.Test something else
10+
(22:55:50) Starting lib.tb_running_test_case.Test something else
1111
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_running_test_case.Test_something_else_27dcc1aa8d44993b6b2d0b0a017fa6001b4c2aa7\output.txt
12-
<span style="color: #008000; ">pass</span> (P=3 S=0 F=0 T=3) lib.tb_running_test_case.Test something else (0.5 seconds)
12+
<span style="color: #008000; ">pass</span> (P=3 S=0 F=0 T=3) lib.tb_running_test_case.Test something else (0.6 s)
1313

1414
==== Summary ========================================================
15-
<span style="color: #008000; ">pass</span> lib.tb_running_test_case.Test scenario A (0.5 seconds)
16-
<span style="color: #008000; ">pass</span> lib.tb_running_test_case.Test scenario B (0.5 seconds)
17-
<span style="color: #008000; ">pass</span> lib.tb_running_test_case.Test something else (0.5 seconds)
15+
<span style="color: #008000; ">pass</span> lib.tb_running_test_case.Test scenario A (0.6 s)
16+
<span style="color: #008000; ">pass</span> lib.tb_running_test_case.Test scenario B (0.6 s)
17+
<span style="color: #008000; ">pass</span> lib.tb_running_test_case.Test something else (0.6 s)
1818
=====================================================================
1919
<span style="color: #008000; ">pass</span> 3 of 3
2020
=====================================================================
21-
Total time was 1.6 seconds
22-
Elapsed time was 1.6 seconds
21+
Total time was 1.8 s
22+
Elapsed time was 1.8 s
2323
=====================================================================
2424
<span style="color: #008000; ">All passed!</span>
2525
</pre></div>

‎docs/run/img/tb_seed_stdout.html

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<div class="highlight" style="background: #002b36; color: #eee8d5;"><pre style="line-height: 125%; background: #002b36; color: #eee8d5;">> python run.py
2+
Starting lib.tb_seed.Test that passes
3+
Output file: C:\repos\vunit\docs\run\src\vunit_out\test_output\lib.tb_seed.Test_that_passes_4125d67fe52dadd934f892b1209f41e7a94a39bd\output.txt
4+
Seed for lib.tb_seed.Test that passes: 8c5c0ea80b58a8ee
5+
simulation stopped @0ms with status 0
6+
<span style="color: #008000; ">pass</span> (P=1 S=0 F=0 T=2) lib.tb_seed.Test that passes (0.5 s)
7+
8+
(09:37:45) Starting lib.tb_seed.Test that fails
9+
Output file: C:\repos\vunit\docs\run\src\vunit_out\test_output\lib.tb_seed.Test_that_fails_0f173e63967af845f06d4a86c622bba76f3ffb3d\output.txt
10+
Seed for lib.tb_seed.Test that fails: fb19f3cca859d69c
11+
0 fs - default - ERROR - Something bad happened
12+
C:\repos\vunit\vunit\vhdl\core\src\core_pkg.vhd:85:7:@0ms:(report failure): Stop simulation on log level error
13+
ghdl:error: report failed
14+
ghdl:error: simulation failed
15+
<span style="color: #cb4b16; ">fail</span> (P=1 S=0 F=1 T=2) lib.tb_seed.Test that fails (0.5 s)
16+
17+
==== Summary ========================================
18+
<span style="color: #008000; ">pass</span> lib.tb_seed.Test that passes (0.5 s)
19+
<span style="color: #cb4b16; ">fail</span> lib.tb_seed.Test that fails (0.5 s)
20+
=====================================================
21+
<span style="color: #008000; ">pass</span> 1 of 2
22+
<span style="color: #cb4b16; ">fail</span> 1 of 2
23+
=====================================================
24+
Total time was 1.0 s
25+
Elapsed time was 1.0 s
26+
=====================================================
27+
<span style="color: #cb4b16; ">Some failed!</span>
28+
</pre></div>

‎docs/run/img/tb_stop_level_stdout.html

+8-11
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,23 @@
11
<div class="highlight" style="background: #002b36; color: #eee8d5;"><pre style="line-height: 125%; background: #002b36; color: #eee8d5;">> python run.py
22
Starting lib.tb_stop_level.Test that fails multiple times but doesn't stop
33
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_stop_level.Test_that_fails_multiple_times_but_doesn't_stop_d08f48d859442d0bc71e2bcdd8b429119f7cc17c\output.txt
4+
Seed for lib.tb_stop_level.Test that fails multiple times but doesn't stop: 7a0e979de335b966
45
0 fs - check - ERROR - Equality check failed - Got 17. Expected 18.
56
0 fs - check - ERROR - Equality check failed - Got 17. Expected 19.
67
FAILURE - Logger check has 2 errors
7-
C:\github\vunit\vunit\vhdl\core\src\core_pkg.vhd:84:7:@0ms:(report failure): Final log check failed
8-
C:\ghdl\bin\ghdl.exe:error: report failed
9-
in process .tb_stop_level(tb).test_runner
10-
from: vunit_lib.logger_pkg.final_log_check at logger_pkg-body.vhd:1249
11-
from: vunit_lib.run_pkg.test_runner_cleanup at run.vhd:114
12-
from: process lib.tb_stop_level(tb).test_runner at tb_stop_level.vhd:29
13-
C:\ghdl\bin\ghdl.exe:error: simulation failed
14-
<span style="color: #cb4b16; ">fail</span> (P=0 S=0 F=1 T=1) lib.tb_stop_level.Test that fails multiple times but doesn't stop (0.5 seconds)
8+
C:\github\vunit\vunit\vhdl\core\src\core_pkg.vhd:85:7:@0ms:(report failure): Final log check failed
9+
ghdl:error: report failed
10+
ghdl:error: simulation failed
11+
<span style="color: #cb4b16; ">fail</span> (P=0 S=0 F=1 T=1) lib.tb_stop_level.Test that fails multiple times but doesn't stop (0.6 s)
1512

1613
==== Summary =============================================================================
17-
<span style="color: #cb4b16; ">fail</span> lib.tb_stop_level.Test that fails multiple times but doesn't stop (0.5 seconds)
14+
<span style="color: #cb4b16; ">fail</span> lib.tb_stop_level.Test that fails multiple times but doesn't stop (0.6 s)
1815
==========================================================================================
1916
<span style="color: #008000; ">pass</span> 0 of 1
2017
<span style="color: #cb4b16; ">fail</span> 1 of 1
2118
==========================================================================================
22-
Total time was 0.5 seconds
23-
Elapsed time was 0.5 seconds
19+
Total time was 0.6 s
20+
Elapsed time was 0.6 s
2421
==========================================================================================
2522
<span style="color: #cb4b16; ">Some failed!</span>
2623
</pre></div>

‎docs/run/img/tb_stopping_failure_stdout.html

+23-20
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,43 @@
11
<div class="highlight" style="background: #002b36; color: #eee8d5;"><pre style="line-height: 125%; background: #002b36; color: #eee8d5;">> python run.py
22
Starting lib.tb_stopping_failure.Test that fails on an assert
3-
Output file: C:\repos\vunit\docs\run\src\vunit_out\test_output\lib.tb_stopping_failure.Test_that_fails_on_an_assert_f53b930e2c7649bc33253af52f8ea89a9c05f07b\output.txt
4-
C:\repos\vunit\docs\run\src\tb_stopping_failure.vhd:24:9:@0ms:(assertion error): Assertion violation
3+
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_stopping_failure.Test_that_fails_on_an_assert_f53b930e2c7649bc33253af52f8ea89a9c05f07b\output.txt
4+
Seed for lib.tb_stopping_failure.Test that fails on an assert: a866d0986a663e6b
5+
C:\github\vunit\docs\run\src\tb_stopping_failure.vhd:24:9:@0ms:(assertion error): Assertion violation
56
ghdl:error: assertion failed
67
ghdl:error: simulation failed
7-
<span style="color: #cb4b16; ">fail</span> (P=0 S=0 F=1 T=4) lib.tb_stopping_failure.Test that fails on an assert (0.5 s)
8+
<span style="color: #cb4b16; ">fail</span> (P=0 S=0 F=1 T=4) lib.tb_stopping_failure.Test that fails on an assert (0.6 s)
89

9-
(11:35:31) Starting lib.tb_stopping_failure.Test that crashes on boundary problems
10-
Output file: C:\repos\vunit\docs\run\src\vunit_out\test_output\lib.tb_stopping_failure.Test_that_crashes_on_boundary_problems_b53105615efefaa16d0cf9ee1bad37b5d3369e95\output.txt
11-
ghdl:error: index (314) out of bounds (1 to 17) at C:\repos\vunit\docs\run\src\tb_stopping_failure.vhd:26
10+
(22:55:55) Starting lib.tb_stopping_failure.Test that crashes on boundary problems
11+
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_stopping_failure.Test_that_crashes_on_boundary_problems_b53105615efefaa16d0cf9ee1bad37b5d3369e95\output.txt
12+
Seed for lib.tb_stopping_failure.Test that crashes on boundary problems: cbcb543764149e78
13+
ghdl:error: index (340) out of bounds (1 to 17) at C:\github\vunit\docs\run\src\tb_stopping_failure.vhd:26
1214
ghdl:error: simulation failed
13-
<span style="color: #cb4b16; ">fail</span> (P=0 S=0 F=2 T=4) lib.tb_stopping_failure.Test that crashes on boundary problems (0.5 s)
15+
<span style="color: #cb4b16; ">fail</span> (P=0 S=0 F=2 T=4) lib.tb_stopping_failure.Test that crashes on boundary problems (0.6 s)
1416

15-
(11:35:31) Starting lib.tb_stopping_failure.Test that fails on VUnit check procedure
16-
Output file: C:\repos\vunit\docs\run\src\vunit_out\test_output\lib.tb_stopping_failure.Test_that_fails_on_VUnit_check_procedure_717a6f8ff044e3d5fa7d7d3ec5a32971d74864dd\output.txt
17+
(22:55:55) Starting lib.tb_stopping_failure.Test that fails on VUnit check procedure
18+
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_stopping_failure.Test_that_fails_on_VUnit_check_procedure_717a6f8ff044e3d5fa7d7d3ec5a32971d74864dd\output.txt
19+
Seed for lib.tb_stopping_failure.Test that fails on VUnit check procedure: 6581894e9ea614b1
1720
0 fs - check - ERROR - Equality check failed - Got 17. Expected 18.
18-
C:\repos\vunit\vunit\vhdl\core\src\core_pkg.vhd:85:7:@0ms:(report failure): Stop simulation on log level error
21+
C:\github\vunit\vunit\vhdl\core\src\core_pkg.vhd:85:7:@0ms:(report failure): Stop simulation on log level error
1922
ghdl:error: report failed
2023
ghdl:error: simulation failed
21-
<span style="color: #cb4b16; ">fail</span> (P=0 S=0 F=3 T=4) lib.tb_stopping_failure.Test that fails on VUnit check procedure (0.5 s)
24+
<span style="color: #cb4b16; ">fail</span> (P=0 S=0 F=3 T=4) lib.tb_stopping_failure.Test that fails on VUnit check procedure (0.6 s)
2225

23-
(11:35:32) Starting lib.tb_stopping_failure.Test that a warning passes
24-
Output file: C:\repos\vunit\docs\run\src\vunit_out\test_output\lib.tb_stopping_failure.Test_that_a_warning_passes_7db91f3b27aea5f89e74e39ea51ce6d61558674e\output.txt
25-
<span style="color: #008000; ">pass</span> (P=1 S=0 F=3 T=4) lib.tb_stopping_failure.Test that a warning passes (0.4 s)
26+
(22:55:56) Starting lib.tb_stopping_failure.Test that a warning passes
27+
Output file: C:\github\vunit\docs\run\src\vunit_out\test_output\lib.tb_stopping_failure.Test_that_a_warning_passes_7db91f3b27aea5f89e74e39ea51ce6d61558674e\output.txt
28+
<span style="color: #008000; ">pass</span> (P=1 S=0 F=3 T=4) lib.tb_stopping_failure.Test that a warning passes (0.6 s)
2629

2730
==== Summary ============================================================================
28-
<span style="color: #008000; ">pass</span> lib.tb_stopping_failure.Test that a warning passes (0.4 s)
29-
<span style="color: #cb4b16; ">fail</span> lib.tb_stopping_failure.Test that fails on an assert (0.5 s)
30-
<span style="color: #cb4b16; ">fail</span> lib.tb_stopping_failure.Test that crashes on boundary problems (0.5 s)
31-
<span style="color: #cb4b16; ">fail</span> lib.tb_stopping_failure.Test that fails on VUnit check procedure (0.5 s)
31+
<span style="color: #008000; ">pass</span> lib.tb_stopping_failure.Test that a warning passes (0.6 s)
32+
<span style="color: #cb4b16; ">fail</span> lib.tb_stopping_failure.Test that fails on an assert (0.6 s)
33+
<span style="color: #cb4b16; ">fail</span> lib.tb_stopping_failure.Test that crashes on boundary problems (0.6 s)
34+
<span style="color: #cb4b16; ">fail</span> lib.tb_stopping_failure.Test that fails on VUnit check procedure (0.6 s)
3235
=========================================================================================
3336
<span style="color: #008000; ">pass</span> 1 of 4
3437
<span style="color: #cb4b16; ">fail</span> 3 of 4
3538
=========================================================================================
36-
Total time was 1.8 s
37-
Elapsed time was 1.8 s
39+
Total time was 2.4 s
40+
Elapsed time was 2.4 s
3841
=========================================================================================
3942
<span style="color: #cb4b16; ">Some failed!</span>
4043
</pre></div>

0 commit comments

Comments
 (0)