[Feature] assinging values to RB storage

[vmoens]

This PR enables assigning values to a RB.

This is slightly bc-breaking:
Previously doing storage[index] = foo was exactly equivalent to storage.set(index, foo). Now only the second will update the cursor position. This lets us assigning values without telling the buffer that we have moved the cursor.

TODO:

• Doc the feature
• test transforms (forward/inverse)

cc @wertyuilife2

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2224

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [MITIGATED] Migration to Amazon Linux 23 - issues with nvidia driver

❌ 2 New Failures, 2 Unrelated Failures

As of commit 71381a2 with merge base 47a1005 ():

NEW FAILURES - The following jobs have failed:

• Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t dc88aa1564b633062060298143548141d25a151ad5dfc4f6155f4c296b9a79a8 /exec failed with exit code 139
• Unit-tests on Linux / tests-optdeps (3.10, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 432f93a480dd7b5f7c000744d0b7654c713754f01787faae4667755b709ac72b /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Libs Tests on Linux / unittests-gym (3.9, 12.1) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}7$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1109s	58.1882ms	17.1856 Ops/s	17.9137 Ops/s	$\color{#d91a1a}-4.06\%$
test_sync	42.2688ms	36.7726ms	27.1941 Ops/s	32.5879 Ops/s	$\textbf{\color{#d91a1a}-16.55\%}$
test_async	56.1220ms	28.8487ms	34.6636 Ops/s	34.6184 Ops/s	$\color{#35bf28}+0.13\%$
test_simple	0.4420s	0.3872s	2.5828 Ops/s	2.6089 Ops/s	$\color{#d91a1a}-1.00\%$
test_transformed	0.5815s	0.5357s	1.8669 Ops/s	1.8589 Ops/s	$\color{#35bf28}+0.43\%$
test_serial	1.3122s	1.2565s	0.7958 Ops/s	0.7762 Ops/s	$\color{#35bf28}+2.53\%$
test_parallel	1.1291s	1.0685s	0.9359 Ops/s	0.9392 Ops/s	$\color{#d91a1a}-0.35\%$
test_step_mdp_speed[True-True-True-True-True]	0.1897ms	21.6572μs	46.1741 KOps/s	46.8467 KOps/s	$\color{#d91a1a}-1.44\%$
test_step_mdp_speed[True-True-True-True-False]	47.9300μs	12.9335μs	77.3188 KOps/s	75.6651 KOps/s	$\color{#35bf28}+2.19\%$
test_step_mdp_speed[True-True-True-False-True]	37.6610μs	12.6259μs	79.2021 KOps/s	78.8373 KOps/s	$\color{#35bf28}+0.46\%$
test_step_mdp_speed[True-True-True-False-False]	0.1673ms	7.6201μs	131.2318 KOps/s	128.6271 KOps/s	$\color{#35bf28}+2.03\%$
test_step_mdp_speed[True-True-False-True-True]	70.4520μs	22.5320μs	44.3812 KOps/s	43.7586 KOps/s	$\color{#35bf28}+1.42\%$
test_step_mdp_speed[True-True-False-True-False]	35.7270μs	14.0030μs	71.4133 KOps/s	69.5546 KOps/s	$\color{#35bf28}+2.67\%$
test_step_mdp_speed[True-True-False-False-True]	64.7110μs	13.8363μs	72.2736 KOps/s	72.1625 KOps/s	$\color{#35bf28}+0.15\%$
test_step_mdp_speed[True-True-False-False-False]	33.2430μs	8.8349μs	113.1876 KOps/s	111.3807 KOps/s	$\color{#35bf28}+1.62\%$
test_step_mdp_speed[True-False-True-True-True]	73.3470μs	24.2596μs	41.2208 KOps/s	41.3464 KOps/s	$\color{#d91a1a}-0.30\%$
test_step_mdp_speed[True-False-True-True-False]	41.8880μs	15.5938μs	64.1281 KOps/s	64.3867 KOps/s	$\color{#d91a1a}-0.40\%$
test_step_mdp_speed[True-False-True-False-True]	60.5130μs	13.7719μs	72.6117 KOps/s	72.2223 KOps/s	$\color{#35bf28}+0.54\%$
test_step_mdp_speed[True-False-True-False-False]	49.7040μs	8.8032μs	113.5946 KOps/s	111.5252 KOps/s	$\color{#35bf28}+1.86\%$
test_step_mdp_speed[True-False-False-True-True]	78.8180μs	25.2598μs	39.5886 KOps/s	39.5685 KOps/s	$\color{#35bf28}+0.05\%$
test_step_mdp_speed[True-False-False-True-False]	68.0380μs	16.6967μs	59.8922 KOps/s	59.3848 KOps/s	$\color{#35bf28}+0.85\%$
test_step_mdp_speed[True-False-False-False-True]	59.7120μs	14.8691μs	67.2533 KOps/s	66.9062 KOps/s	$\color{#35bf28}+0.52\%$
test_step_mdp_speed[True-False-False-False-False]	50.8250μs	10.0709μs	99.2957 KOps/s	99.7531 KOps/s	$\color{#d91a1a}-0.46\%$
test_step_mdp_speed[False-True-True-True-True]	50.0730μs	24.0166μs	41.6379 KOps/s	41.5815 KOps/s	$\color{#35bf28}+0.14\%$
test_step_mdp_speed[False-True-True-True-False]	60.2130μs	15.6141μs	64.0445 KOps/s	63.6114 KOps/s	$\color{#35bf28}+0.68\%$
test_step_mdp_speed[False-True-True-False-True]	41.2470μs	16.0076μs	62.4703 KOps/s	61.9615 KOps/s	$\color{#35bf28}+0.82\%$
test_step_mdp_speed[False-True-True-False-False]	58.5200μs	10.0388μs	99.6137 KOps/s	98.3155 KOps/s	$\color{#35bf28}+1.32\%$
test_step_mdp_speed[False-True-False-True-True]	49.4930μs	25.0709μs	39.8869 KOps/s	39.4894 KOps/s	$\color{#35bf28}+1.01\%$
test_step_mdp_speed[False-True-False-True-False]	69.3400μs	16.8013μs	59.5192 KOps/s	59.9690 KOps/s	$\color{#d91a1a}-0.75\%$
test_step_mdp_speed[False-True-False-False-True]	58.4490μs	17.2122μs	58.0983 KOps/s	57.9159 KOps/s	$\color{#35bf28}+0.31\%$
test_step_mdp_speed[False-True-False-False-False]	70.2410μs	11.2632μs	88.7849 KOps/s	88.8773 KOps/s	$\color{#d91a1a}-0.10\%$
test_step_mdp_speed[False-False-True-True-True]	62.5570μs	26.5221μs	37.7044 KOps/s	37.4246 KOps/s	$\color{#35bf28}+0.75\%$
test_step_mdp_speed[False-False-True-True-False]	71.2840μs	18.1467μs	55.1065 KOps/s	55.2705 KOps/s	$\color{#d91a1a}-0.30\%$
test_step_mdp_speed[False-False-True-False-True]	76.6140μs	17.1084μs	58.4510 KOps/s	57.9430 KOps/s	$\color{#35bf28}+0.88\%$
test_step_mdp_speed[False-False-True-False-False]	36.7590μs	11.2618μs	88.7960 KOps/s	88.8538 KOps/s	$\color{#d91a1a}-0.07\%$
test_step_mdp_speed[False-False-False-True-True]	40.6560μs	27.6700μs	36.1402 KOps/s	30.7263 KOps/s	$\textbf{\color{#35bf28}+17.62\%}$
test_step_mdp_speed[False-False-False-True-False]	67.9470μs	19.1553μs	52.2048 KOps/s	53.0900 KOps/s	$\color{#d91a1a}-1.67\%$
test_step_mdp_speed[False-False-False-False-True]	48.1710μs	17.9715μs	55.6435 KOps/s	55.4633 KOps/s	$\color{#35bf28}+0.32\%$
test_step_mdp_speed[False-False-False-False-False]	63.9290μs	12.1581μs	82.2494 KOps/s	82.3127 KOps/s	$\color{#d91a1a}-0.08\%$
test_values[generalized_advantage_estimate-True-True]	9.5934ms	9.3919ms	106.4750 Ops/s	107.1365 Ops/s	$\color{#d91a1a}-0.62\%$
test_values[vec_generalized_advantage_estimate-True-True]	37.6554ms	35.0496ms	28.5310 Ops/s	30.0954 Ops/s	$\textbf{\color{#d91a1a}-5.20\%}$
test_values[td0_return_estimate-False-False]	0.2200ms	0.1716ms	5.8266 KOps/s	5.9913 KOps/s	$\color{#d91a1a}-2.75\%$
test_values[td1_return_estimate-False-False]	26.4200ms	23.3375ms	42.8495 Ops/s	42.8169 Ops/s	$\color{#35bf28}+0.08\%$
test_values[vec_td1_return_estimate-False-False]	38.0133ms	35.3003ms	28.3284 Ops/s	29.9546 Ops/s	$\textbf{\color{#d91a1a}-5.43\%}$
test_values[td_lambda_return_estimate-True-False]	36.3389ms	33.4710ms	29.8766 Ops/s	29.7355 Ops/s	$\color{#35bf28}+0.47\%$
test_values[vec_td_lambda_return_estimate-True-False]	37.4933ms	35.3501ms	28.2885 Ops/s	30.0445 Ops/s	$\textbf{\color{#d91a1a}-5.84\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.4296ms	8.2567ms	121.1140 Ops/s	120.1513 Ops/s	$\color{#35bf28}+0.80\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	2.8866ms	2.0104ms	497.4182 Ops/s	552.4983 Ops/s	$\textbf{\color{#d91a1a}-9.97\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4833ms	0.3569ms	2.8017 KOps/s	2.8422 KOps/s	$\color{#d91a1a}-1.42\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	53.4614ms	47.0585ms	21.2501 Ops/s	23.1524 Ops/s	$\textbf{\color{#d91a1a}-8.22\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	3.6550ms	3.0303ms	330.0010 Ops/s	332.2410 Ops/s	$\color{#d91a1a}-0.67\%$
test_dqn_speed	1.5344ms	1.3269ms	753.6518 Ops/s	730.5476 Ops/s	$\color{#35bf28}+3.16\%$
test_ddpg_speed	3.5327ms	2.8599ms	349.6630 Ops/s	345.4219 Ops/s	$\color{#35bf28}+1.23\%$
test_sac_speed	9.3233ms	8.5194ms	117.3793 Ops/s	117.0273 Ops/s	$\color{#35bf28}+0.30\%$
test_redq_speed	14.6156ms	13.4285ms	74.4685 Ops/s	72.7578 Ops/s	$\color{#35bf28}+2.35\%$
test_redq_deprec_speed	14.7063ms	13.6276ms	73.3803 Ops/s	71.8917 Ops/s	$\color{#35bf28}+2.07\%$
test_td3_speed	16.6222ms	8.4936ms	117.7361 Ops/s	116.4366 Ops/s	$\color{#35bf28}+1.12\%$
test_cql_speed	38.5194ms	37.0748ms	26.9725 Ops/s	26.9791 Ops/s	$\color{#d91a1a}-0.02\%$
test_a2c_speed	10.0948ms	7.7421ms	129.1642 Ops/s	132.1702 Ops/s	$\color{#d91a1a}-2.27\%$
test_ppo_speed	8.9853ms	7.8189ms	127.8959 Ops/s	128.4550 Ops/s	$\color{#d91a1a}-0.44\%$
test_reinforce_speed	7.8736ms	6.6535ms	150.2960 Ops/s	149.0582 Ops/s	$\color{#35bf28}+0.83\%$
test_iql_speed	34.6413ms	33.1049ms	30.2070 Ops/s	29.9386 Ops/s	$\color{#35bf28}+0.90\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.3194ms	3.5601ms	280.8932 Ops/s	276.6948 Ops/s	$\color{#35bf28}+1.52\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.9630ms	0.4992ms	2.0032 KOps/s	1.9663 KOps/s	$\color{#35bf28}+1.88\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6350ms	0.4735ms	2.1119 KOps/s	2.0334 KOps/s	$\color{#35bf28}+3.86\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	4.4021ms	3.5674ms	280.3200 Ops/s	276.3138 Ops/s	$\color{#35bf28}+1.45\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.7637ms	0.5152ms	1.9411 KOps/s	2.0028 KOps/s	$\color{#d91a1a}-3.08\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6976ms	0.4722ms	2.1177 KOps/s	2.0976 KOps/s	$\color{#35bf28}+0.96\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	2.2195ms	1.7043ms	586.7524 Ops/s	589.2376 Ops/s	$\color{#d91a1a}-0.42\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	4.5091ms	1.6244ms	615.6298 Ops/s	621.8632 Ops/s	$\color{#d91a1a}-1.00\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	4.2613ms	3.6937ms	270.7330 Ops/s	273.6307 Ops/s	$\color{#d91a1a}-1.06\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.1084s	0.7041ms	1.4203 KOps/s	1.3919 KOps/s	$\color{#35bf28}+2.04\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.7532ms	0.5857ms	1.7073 KOps/s	1.6777 KOps/s	$\color{#35bf28}+1.76\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.2186ms	3.5992ms	277.8370 Ops/s	278.2373 Ops/s	$\color{#d91a1a}-0.14\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.0994ms	0.5018ms	1.9929 KOps/s	1.9393 KOps/s	$\color{#35bf28}+2.76\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6218ms	0.4745ms	2.1076 KOps/s	2.0538 KOps/s	$\color{#35bf28}+2.62\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.2557ms	3.5001ms	285.7030 Ops/s	272.2156 Ops/s	$\color{#35bf28}+4.95\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6000ms	0.4958ms	2.0169 KOps/s	1.9708 KOps/s	$\color{#35bf28}+2.34\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.6638ms	0.4826ms	2.0721 KOps/s	2.0500 KOps/s	$\color{#35bf28}+1.08\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.5835ms	3.6599ms	273.2316 Ops/s	272.8304 Ops/s	$\color{#35bf28}+0.15\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.7212ms	0.6159ms	1.6235 KOps/s	1.5946 KOps/s	$\color{#35bf28}+1.81\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8287ms	0.5959ms	1.6781 KOps/s	1.6518 KOps/s	$\color{#35bf28}+1.59\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1147s	6.0128ms	166.3132 Ops/s	168.6338 Ops/s	$\color{#d91a1a}-1.38\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	0.1165s	14.6508ms	68.2555 Ops/s	67.2764 Ops/s	$\color{#35bf28}+1.46\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	1.5397ms	1.0327ms	968.3299 Ops/s	965.6560 Ops/s	$\color{#35bf28}+0.28\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1025s	5.7222ms	174.7564 Ops/s	175.3172 Ops/s	$\color{#d91a1a}-0.32\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	14.5782ms	12.5111ms	79.9292 Ops/s	79.5834 Ops/s	$\color{#35bf28}+0.43\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	3.1603ms	1.3552ms	737.9097 Ops/s	963.2117 Ops/s	$\textbf{\color{#d91a1a}-23.39\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1044s	5.8688ms	170.3935 Ops/s	168.3958 Ops/s	$\color{#35bf28}+1.19\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	15.4933ms	12.6279ms	79.1895 Ops/s	67.5917 Ops/s	$\textbf{\color{#35bf28}+17.16\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	4.2584ms	1.2676ms	788.8630 Ops/s	828.1399 Ops/s	$\color{#d91a1a}-4.74\%$

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}5$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1175s	0.1167s	8.5670 Ops/s	8.6324 Ops/s	$\color{#d91a1a}-0.76\%$
test_sync	0.1031s	0.1014s	9.8587 Ops/s	9.5309 Ops/s	$\color{#35bf28}+3.44\%$
test_async	0.2109s	84.2739ms	11.8661 Ops/s	10.3871 Ops/s	$\textbf{\color{#35bf28}+14.24\%}$
test_single_pixels	0.1278s	0.1275s	7.8439 Ops/s	7.8986 Ops/s	$\color{#d91a1a}-0.69\%$
test_sync_pixels	85.0236ms	80.2607ms	12.4594 Ops/s	12.4018 Ops/s	$\color{#35bf28}+0.46\%$
test_async_pixels	0.1548s	68.2505ms	14.6519 Ops/s	14.4284 Ops/s	$\color{#35bf28}+1.55\%$
test_simple	0.8008s	0.7999s	1.2502 Ops/s	1.2264 Ops/s	$\color{#35bf28}+1.94\%$
test_transformed	1.0596s	1.0571s	0.9460 Ops/s	0.9385 Ops/s	$\color{#35bf28}+0.79\%$
test_serial	2.5346s	2.4786s	0.4035 Ops/s	0.4046 Ops/s	$\color{#d91a1a}-0.29\%$
test_parallel	2.4600s	2.3741s	0.4212 Ops/s	0.4256 Ops/s	$\color{#d91a1a}-1.04\%$
test_step_mdp_speed[True-True-True-True-True]	0.1010ms	33.4555μs	29.8905 KOps/s	29.2896 KOps/s	$\color{#35bf28}+2.05\%$
test_step_mdp_speed[True-True-True-True-False]	40.1330μs	19.9878μs	50.0306 KOps/s	50.1769 KOps/s	$\color{#d91a1a}-0.29\%$
test_step_mdp_speed[True-True-True-False-True]	0.1349ms	19.1519μs	52.2142 KOps/s	51.3149 KOps/s	$\color{#35bf28}+1.75\%$
test_step_mdp_speed[True-True-True-False-False]	29.4620μs	11.5176μs	86.8235 KOps/s	87.2700 KOps/s	$\color{#d91a1a}-0.51\%$
test_step_mdp_speed[True-True-False-True-True]	61.8440μs	35.1827μs	28.4231 KOps/s	27.9736 KOps/s	$\color{#35bf28}+1.61\%$
test_step_mdp_speed[True-True-False-True-False]	46.7030μs	22.0893μs	45.2709 KOps/s	45.8459 KOps/s	$\color{#d91a1a}-1.25\%$
test_step_mdp_speed[True-True-False-False-True]	0.1023ms	21.0765μs	47.4461 KOps/s	47.7042 KOps/s	$\color{#d91a1a}-0.54\%$
test_step_mdp_speed[True-True-False-False-False]	31.3720μs	13.5148μs	73.9927 KOps/s	74.3783 KOps/s	$\color{#d91a1a}-0.52\%$
test_step_mdp_speed[True-False-True-True-True]	64.1340μs	37.7706μs	26.4756 KOps/s	26.3027 KOps/s	$\color{#35bf28}+0.66\%$
test_step_mdp_speed[True-False-True-True-False]	0.1514ms	24.1030μs	41.4886 KOps/s	41.5306 KOps/s	$\color{#d91a1a}-0.10\%$
test_step_mdp_speed[True-False-True-False-True]	44.3830μs	21.1255μs	47.3361 KOps/s	47.1854 KOps/s	$\color{#35bf28}+0.32\%$
test_step_mdp_speed[True-False-True-False-False]	83.8140μs	13.3618μs	74.8404 KOps/s	74.6633 KOps/s	$\color{#35bf28}+0.24\%$
test_step_mdp_speed[True-False-False-True-True]	68.0930μs	39.0876μs	25.5835 KOps/s	25.3998 KOps/s	$\color{#35bf28}+0.72\%$
test_step_mdp_speed[True-False-False-True-False]	49.2430μs	25.5100μs	39.2002 KOps/s	39.2992 KOps/s	$\color{#d91a1a}-0.25\%$
test_step_mdp_speed[True-False-False-False-True]	47.6820μs	22.8635μs	43.7379 KOps/s	43.3924 KOps/s	$\color{#35bf28}+0.80\%$
test_step_mdp_speed[True-False-False-False-False]	31.8810μs	15.1842μs	65.8580 KOps/s	65.5647 KOps/s	$\color{#35bf28}+0.45\%$
test_step_mdp_speed[False-True-True-True-True]	56.8930μs	37.4347μs	26.7132 KOps/s	26.4089 KOps/s	$\color{#35bf28}+1.15\%$
test_step_mdp_speed[False-True-True-True-False]	40.6120μs	23.7425μs	42.1185 KOps/s	41.8205 KOps/s	$\color{#35bf28}+0.71\%$
test_step_mdp_speed[False-True-True-False-True]	51.8130μs	24.7354μs	40.4279 KOps/s	39.3069 KOps/s	$\color{#35bf28}+2.85\%$
test_step_mdp_speed[False-True-True-False-False]	45.6530μs	15.1354μs	66.0703 KOps/s	65.2509 KOps/s	$\color{#35bf28}+1.26\%$
test_step_mdp_speed[False-True-False-True-True]	63.9130μs	39.2388μs	25.4850 KOps/s	25.2994 KOps/s	$\color{#35bf28}+0.73\%$
test_step_mdp_speed[False-True-False-True-False]	42.3020μs	25.6981μs	38.9134 KOps/s	38.5835 KOps/s	$\color{#35bf28}+0.85\%$
test_step_mdp_speed[False-True-False-False-True]	54.9430μs	27.1051μs	36.8934 KOps/s	37.0126 KOps/s	$\color{#d91a1a}-0.32\%$
test_step_mdp_speed[False-True-False-False-False]	67.7840μs	17.4767μs	57.2191 KOps/s	58.8556 KOps/s	$\color{#d91a1a}-2.78\%$
test_step_mdp_speed[False-False-True-True-True]	71.1940μs	41.2004μs	24.2716 KOps/s	24.1365 KOps/s	$\color{#35bf28}+0.56\%$
test_step_mdp_speed[False-False-True-True-False]	55.9930μs	27.9558μs	35.7708 KOps/s	35.9098 KOps/s	$\color{#d91a1a}-0.39\%$
test_step_mdp_speed[False-False-True-False-True]	46.1230μs	26.6698μs	37.4955 KOps/s	36.4989 KOps/s	$\color{#35bf28}+2.73\%$
test_step_mdp_speed[False-False-True-False-False]	40.4020μs	17.1529μs	58.2993 KOps/s	58.8581 KOps/s	$\color{#d91a1a}-0.95\%$
test_step_mdp_speed[False-False-False-True-True]	55.1730μs	42.8731μs	23.3246 KOps/s	23.0284 KOps/s	$\color{#35bf28}+1.29\%$
test_step_mdp_speed[False-False-False-True-False]	52.2220μs	29.9898μs	33.3447 KOps/s	33.8118 KOps/s	$\color{#d91a1a}-1.38\%$
test_step_mdp_speed[False-False-False-False-True]	48.2230μs	28.3736μs	35.2440 KOps/s	34.5587 KOps/s	$\color{#35bf28}+1.98\%$
test_step_mdp_speed[False-False-False-False-False]	44.0220μs	18.8127μs	53.1556 KOps/s	53.1100 KOps/s	$\color{#35bf28}+0.09\%$
test_values[generalized_advantage_estimate-True-True]	25.5266ms	24.9726ms	40.0439 Ops/s	41.4161 Ops/s	$\color{#d91a1a}-3.31\%$
test_values[vec_generalized_advantage_estimate-True-True]	96.9234ms	2.8332ms	352.9531 Ops/s	368.5639 Ops/s	$\color{#d91a1a}-4.24\%$
test_values[td0_return_estimate-False-False]	97.7350μs	65.2003μs	15.3373 KOps/s	15.4697 KOps/s	$\color{#d91a1a}-0.86\%$
test_values[td1_return_estimate-False-False]	55.6692ms	54.0821ms	18.4904 Ops/s	18.5996 Ops/s	$\color{#d91a1a}-0.59\%$
test_values[vec_td1_return_estimate-False-False]	1.3971ms	1.0782ms	927.4375 Ops/s	929.3813 Ops/s	$\color{#d91a1a}-0.21\%$
test_values[td_lambda_return_estimate-True-False]	89.4433ms	85.9742ms	11.6314 Ops/s	11.7161 Ops/s	$\color{#d91a1a}-0.72\%$
test_values[vec_td_lambda_return_estimate-True-False]	1.4416ms	1.0775ms	928.1105 Ops/s	931.3892 Ops/s	$\color{#d91a1a}-0.35\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	24.7953ms	24.6078ms	40.6375 Ops/s	40.8434 Ops/s	$\color{#d91a1a}-0.50\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.9260ms	0.7093ms	1.4098 KOps/s	1.4134 KOps/s	$\color{#d91a1a}-0.25\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.8209ms	0.6862ms	1.4574 KOps/s	1.5176 KOps/s	$\color{#d91a1a}-3.97\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.6444ms	1.4625ms	683.7705 Ops/s	686.2009 Ops/s	$\color{#d91a1a}-0.35\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.7058ms	0.6789ms	1.4731 KOps/s	1.4833 KOps/s	$\color{#d91a1a}-0.69\%$
test_dqn_speed	1.8205ms	1.4842ms	673.7522 Ops/s	679.9730 Ops/s	$\color{#d91a1a}-0.91\%$
test_ddpg_speed	3.3291ms	2.9895ms	334.5052 Ops/s	337.6228 Ops/s	$\color{#d91a1a}-0.92\%$
test_sac_speed	8.9559ms	8.6360ms	115.7946 Ops/s	118.0422 Ops/s	$\color{#d91a1a}-1.90\%$
test_redq_speed	11.0362ms	10.5798ms	94.5194 Ops/s	94.7365 Ops/s	$\color{#d91a1a}-0.23\%$
test_redq_deprec_speed	11.8976ms	11.4893ms	87.0372 Ops/s	84.8070 Ops/s	$\color{#35bf28}+2.63\%$
test_td3_speed	8.6529ms	8.4446ms	118.4185 Ops/s	119.2979 Ops/s	$\color{#d91a1a}-0.74\%$
test_cql_speed	27.4719ms	26.2154ms	38.1455 Ops/s	38.5513 Ops/s	$\color{#d91a1a}-1.05\%$
test_a2c_speed	6.0864ms	5.7445ms	174.0794 Ops/s	176.4175 Ops/s	$\color{#d91a1a}-1.33\%$
test_ppo_speed	6.4652ms	6.0835ms	164.3783 Ops/s	167.0778 Ops/s	$\color{#d91a1a}-1.62\%$
test_reinforce_speed	5.5984ms	4.6991ms	212.8047 Ops/s	213.6308 Ops/s	$\color{#d91a1a}-0.39\%$
test_iql_speed	20.4532ms	19.8515ms	50.3741 Ops/s	46.5865 Ops/s	$\textbf{\color{#35bf28}+8.13\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.0815ms	4.9164ms	203.4015 Ops/s	204.0458 Ops/s	$\color{#d91a1a}-0.32\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.1493ms	0.5945ms	1.6820 KOps/s	1.6868 KOps/s	$\color{#d91a1a}-0.28\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.7089ms	0.5694ms	1.7563 KOps/s	1.7679 KOps/s	$\color{#d91a1a}-0.66\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.1553ms	4.8886ms	204.5592 Ops/s	206.5637 Ops/s	$\color{#d91a1a}-0.97\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7709ms	0.5882ms	1.7002 KOps/s	1.7071 KOps/s	$\color{#d91a1a}-0.41\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.4896ms	0.5698ms	1.7549 KOps/s	1.7879 KOps/s	$\color{#d91a1a}-1.85\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	2.3155ms	2.1247ms	470.6603 Ops/s	477.2347 Ops/s	$\color{#d91a1a}-1.38\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	5.6731ms	2.0619ms	484.9931 Ops/s	502.2892 Ops/s	$\color{#d91a1a}-3.44\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.1884ms	5.0512ms	197.9739 Ops/s	200.7512 Ops/s	$\color{#d91a1a}-1.38\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.9061ms	0.7221ms	1.3849 KOps/s	1.3939 KOps/s	$\color{#d91a1a}-0.65\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8764ms	0.6931ms	1.4428 KOps/s	1.4475 KOps/s	$\color{#d91a1a}-0.32\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.3193ms	4.8727ms	205.2248 Ops/s	205.4396 Ops/s	$\color{#d91a1a}-0.10\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.4084ms	0.5950ms	1.6808 KOps/s	1.6765 KOps/s	$\color{#35bf28}+0.25\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.7129ms	0.5681ms	1.7602 KOps/s	1.7394 KOps/s	$\color{#35bf28}+1.20\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.1271ms	4.8705ms	205.3197 Ops/s	206.4564 Ops/s	$\color{#d91a1a}-0.55\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6933ms	0.5875ms	1.7021 KOps/s	1.6979 KOps/s	$\color{#35bf28}+0.25\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.3662ms	0.5693ms	1.7567 KOps/s	1.7775 KOps/s	$\color{#d91a1a}-1.17\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.1835ms	5.0427ms	198.3067 Ops/s	199.6355 Ops/s	$\color{#d91a1a}-0.67\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.8499ms	0.7202ms	1.3886 KOps/s	1.3897 KOps/s	$\color{#d91a1a}-0.08\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8733ms	0.6968ms	1.4352 KOps/s	1.4291 KOps/s	$\color{#35bf28}+0.43\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1174s	9.4566ms	105.7463 Ops/s	137.1759 Ops/s	$\textbf{\color{#d91a1a}-22.91\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	18.9550ms	16.5080ms	60.5767 Ops/s	60.8251 Ops/s	$\color{#d91a1a}-0.41\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	1.4428ms	1.2777ms	782.6304 Ops/s	742.9536 Ops/s	$\textbf{\color{#35bf28}+5.34\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1042s	7.2463ms	138.0018 Ops/s	109.0255 Ops/s	$\textbf{\color{#35bf28}+26.58\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	18.7389ms	16.3940ms	60.9981 Ops/s	60.7670 Ops/s	$\color{#35bf28}+0.38\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	2.3102ms	1.3005ms	768.9318 Ops/s	740.6415 Ops/s	$\color{#35bf28}+3.82\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1069s	9.4676ms	105.6230 Ops/s	135.0658 Ops/s	$\textbf{\color{#d91a1a}-21.80\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	18.7894ms	16.5339ms	60.4817 Ops/s	53.5989 Ops/s	$\textbf{\color{#35bf28}+12.84\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	2.4975ms	1.4692ms	680.6510 Ops/s	663.0095 Ops/s	$\color{#35bf28}+2.66\%$