Performance comparison against a pure Numpy Simple Replay buffer Implemenation

[jamartinh]

Hi @ymd-h I have created a pure Numpy implementation of a ReplayBuffer with the same capabilities of the simple cpprb ReplayBuffer.

Sampling, get all data, batch addition and arbitrary keys between many other features,

However, for me it is unexpected the comparison results against cpprb ReplayBuffer in terms of performance:

Operation	cpprb	ReplayBuffer	Speedup
Single add	2.096183s ± 0.015288s	0.682884s ± 0.009208s	3.07x
Batch add	0.000046s ± 0.000013s	0.000008s ± 0.000002s	5.80x
Sample	0.000115s ± 0.000037s	0.000084s ± 0.000015s	1.37x
Get all	0.003825s ± 0.000253s	0.000001s ± 0.000001s	7130.60x

To which thing can we attribute these differences ?

[ymd-h]

@jamartinh

Hi, it is hard to say since I don't know your implementation and benchmark details.
(Did it take 2s per one add() call? Surprisingly slow.)

cpprb supports some features like memory compression, which has small overhead even if it is not activated (we need to check).
Additionally, ring buffer index is always atomic value because of simplicity, which may have negative impact on performance.

If you find out any problems (and send PR if possible), it will be very helpful.

---

By the way, I think major usage is PER, and I assume its bottleneck is Segment Tree.

[jamartinh]

This is averaged over 20 runs with a lot of data and repetitions