perf_hooks: add histogram.diff()

Getting the values recorded during an interval requires calling
`reset()`, which removes them for every other user of the histogram,
or copying and subtracting histograms, which silently produces a wrong
result when the source was reset in between.

Add `histogram.diff(other)`, which returns a new read-only `Histogram`
containing the values recorded after `other`, an earlier snapshot of
the histogram, was taken. Neither histogram is changed. Unlike
`subtract()`, it verifies that both histograms have the same layout,
and it throws instead of clamping when `other` contains values that
the histogram does not.

Add `histogram.resetCount`, the number of calls to `reset()` and
`subtract()`, which `snapshot()` copies. `diff()` throws when the
counts differ, so a reset between two snapshots is detected even when
every bucket has since grown past its previous count.

Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: OpenCode

Author: jasnell

doc/api/perf_hooks.md

@@ -2152,6 +2152,52 @@ added:
 Returns the number of recorded values that fall within the equivalent
 value range of the given value.
 
+### `histogram.diff(other)`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+* `other` {Histogram} An earlier snapshot of this histogram.
+* Returns: {Histogram}
+
+Returns a new {Histogram} containing the values recorded in this histogram after
+`other` was taken. Neither histogram is changed. To get the values recorded
+during each interval without calling `reset()`, compute each difference from a
+snapshot and keep that snapshot as the baseline for the next interval:
+
+```js
+const { monitorEventLoopDelay } = require('node:perf_hooks');
+
+const histogram = monitorEventLoopDelay();
+histogram.enable();
+let previous = histogram.snapshot();
+
+setInterval(() => {
+  const current = histogram.snapshot();
+  // After a reset, use everything recorded since the reset.
+  const delta = current.resetCount === previous.resetCount ?
+    current.diff(previous) : current;
+  console.log(delta.percentile(99));
+  previous = current;
+}, 10_000);
+```
+
+The `count`, `exceeds`, and bucket counts of the returned histogram are the
+differences between the two histograms. Its `min` and `max` are computed from
+the buckets of the difference, it has no EWMA state, and its `resetCount` is
+`0`.
+
+This method throws:
+
+* `ERR_INVALID_ARG_VALUE` if `other` has a different `lowest`, `highest`, or
+  `figures` configuration.
+* `ERR_INVALID_STATE` if values have been removed from this histogram since
+  `other` was taken, which is the case when the `resetCount` of the two
+  histograms differs.
+* `ERR_INVALID_ARG_VALUE` if `other` contains values that are not in this
+  histogram, for example because the histograms were passed in the wrong order.
+
 ### `histogram.exceeds`
 
 <!-- YAML
@@ -2604,7 +2650,21 @@ boundaries are equal has an infinite density.
 added: v11.10.0
 -->
 
-Resets the collected histogram data.
+Resets the collected histogram data and increments `histogram.resetCount`.
+
+### `histogram.resetCount`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+* Type: {number}
+
+The number of times values have been removed from this histogram by `reset()`
+or, for a {RecordableHistogram}, `subtract()`. A snapshot has the `resetCount`
+of its source at the time it was taken, so comparing the `resetCount` of two
+snapshots shows whether the source was reset between them. See
+[`histogram.diff()`][].
 
 ### `histogram.skewness`
 
@@ -2810,7 +2870,7 @@ added:
 
 Subtracts the values of `other` from this histogram. Both histograms should
 have compatible configurations. Bucket counts that would become negative
-are clamped to zero.
+are clamped to zero. Increments `histogram.resetCount`.
 
 ## Class: `SlidingWindowHistogram`
 
@@ -3289,6 +3349,7 @@ dns.promises.resolve('localhost');
 [Worker threads]: worker_threads.md#worker-threads
 [`'exit'`]: process.md#event-exit
 [`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options
+[`histogram.diff()`]: #histogramdiffother
 [`histogram.export()`]: #histogramexport
 [`perf_hooks.createSlidingWindowHistogram()`]: #perf_hookscreateslidingwindowhistogramoptions
 [`perf_hooks.eventLoopUtilization()`]: #perf_hookseventlooputilizationutilization1-utilization2

lib/internal/histogram.js

@@ -679,6 +679,18 @@ class Histogram {
     this[kHandle]?.reset();
   }
 
+  /**
+   * The number of times values have been removed from the histogram by
+   * `reset()` or `subtract()`.
+   * @readonly
+   * @type {number}
+   */
+  get resetCount() {
+    if (!isHistogram(this))
+      throw new ERR_INVALID_THIS('Histogram');
+    return this[kHandle]?.resetCount();
+  }
+
   /**
    * Returns a new, independent histogram containing a copy of this
    * histogram's current state. Values cannot be recorded into the returned
@@ -691,6 +703,20 @@ class Histogram {
     return new ClonedHistogram(this[kHandle].snapshot());
   }
 
+  /**
+   * Returns a new histogram containing the values recorded in this histogram
+   * after `other`, an earlier snapshot of it, was taken.
+   * @param {Histogram} other
+   * @returns {Histogram}
+   */
+  diff(other) {
+    if (!isHistogram(this))
+      throw new ERR_INVALID_THIS('Histogram');
+    if (!isHistogram(other))
+      throw new ERR_INVALID_ARG_TYPE('other', 'Histogram', other);
+    return new ClonedHistogram(this[kHandle].diff(other[kHandle]));
+  }
+
   [kClone]() {
     const handle = this[kHandle];
     return {

src/histogram-inl.h

@@ -42,6 +42,7 @@ void Histogram::Reset() {
   RwLock::ScopedWriteLock lock(mutex_);
   hdr_reset(histogram_.get());
   InvalidateRecordedSnapshot();
+  reset_count_++;
   exceeds_ = 0;
   prev_ = 0;
   ewma_mean_ = 0;
@@ -90,6 +91,11 @@ size_t Histogram::Exceeds() const {
   return exceeds_;
 }
 
+uint64_t Histogram::ResetCount() const {
+  RwLock::ScopedReadLock lock(mutex_);
+  return reset_count_;
+}
+
 int64_t Histogram::Min() const {
   RwLock::ScopedReadLock lock(mutex_);
   return hdr_min(histogram_.get());

src/histogram.cc

@@ -93,17 +93,22 @@ void CopyRecordedData(hdr_histogram* target, const hdr_histogram* source) {
 }
 }  // namespace
 
-std::shared_ptr<Histogram> Histogram::Clone() const {
-  // The layout is fixed when the histogram is created, so the copy can be
-  // allocated without holding the lock.
-  hdr_histogram* copy;
+std::shared_ptr<Histogram> Histogram::CreateWithSameLayout() const {
+  // The layout is fixed when the histogram is created, so it can be read
+  // without holding the lock.
+  hdr_histogram* histogram;
   if (hdr_init(histogram_->lowest_discernible_value,
                histogram_->highest_trackable_value,
                histogram_->significant_figures,
-               &copy) != 0) {
+               &histogram) != 0) {
     return {};
   }
-  auto clone = std::make_shared<Histogram>(HistogramPointer(copy), Options{});
+  return std::make_shared<Histogram>(HistogramPointer(histogram), Options{});
+}
+
+std::shared_ptr<Histogram> Histogram::Clone() const {
+  std::shared_ptr<Histogram> clone = CreateWithSameLayout();
+  if (!clone) return {};
 
   // Every member that holds recorded or statistical state must be copied
   // here. The recorded snapshot cache is not copied; the clone builds its own
@@ -112,6 +117,7 @@ std::shared_ptr<Histogram> Histogram::Clone() const {
   CopyRecordedData(clone->histogram_.get(), histogram_.get());
   clone->prev_ = prev_;
   clone->exceeds_ = exceeds_;
+  clone->reset_count_ = reset_count_;
   clone->ewma_alpha_ = ewma_alpha_;
   clone->ewma_mean_ = ewma_mean_;
   clone->ewma_variance_ = ewma_variance_;
@@ -121,6 +127,59 @@ std::shared_ptr<Histogram> Histogram::Clone() const {
   return clone;
 }
 
+std::shared_ptr<Histogram> Histogram::Diff(const Histogram& other,
+                                           DiffError* error) const {
+  // Counts are subtracted index by index, so both histograms must map values
+  // to the same indexes. None of these fields change after creation.
+  if (!IsCompatible(other) || histogram_->normalizing_index_offset !=
+                                  other.histogram_->normalizing_index_offset) {
+    *error = DiffError::kIncompatible;
+    return {};
+  }
+
+  std::shared_ptr<Histogram> diff = CreateWithSameLayout();
+  if (!diff) {
+    *error = DiffError::kOutOfMemory;
+    return {};
+  }
+
+  // Only the recorded values and the exceeds count carry over. EWMA and timing
+  // state cannot be subtracted.
+  uint64_t reset_count;
+  {
+    RwLock::ScopedReadLock lock(mutex_);
+    CopyRecordedData(diff->histogram_.get(), histogram_.get());
+    diff->exceeds_ = exceeds_;
+    reset_count = reset_count_;
+  }
+
+  // `diff` is not shared yet, so only the lock of `other` is needed from here
+  // on. Never holding both locks at once avoids lock ordering issues.
+  RwLock::ScopedReadLock lock(other.mutex_);
+  if (reset_count != other.reset_count_) {
+    *error = DiffError::kReset;
+    return {};
+  }
+  if (diff->exceeds_ < other.exceeds_) {
+    *error = DiffError::kNotEarlier;
+    return {};
+  }
+
+  hdr_histogram* target = diff->histogram_.get();
+  const hdr_histogram* source = other.histogram_.get();
+  for (int32_t i = 0; i < target->counts_len; i++) {
+    if (target->counts[i] < source->counts[i]) {
+      *error = DiffError::kNotEarlier;
+      return {};
+    }
+    target->counts[i] -= source->counts[i];
+  }
+  diff->exceeds_ -= other.exceeds_;
+  hdr_reset_internal_counters(target);
+  *error = DiffError::kNone;
+  return diff;
+}
+
 void Histogram::MemoryInfo(MemoryTracker* tracker) const {
   tracker->TrackFieldWithSize("histogram", GetMemorySize());
   tracker->TrackFieldWithSize("qrde_snapshot",
@@ -313,6 +372,7 @@ double Histogram::Subtract(const Histogram& other) {
     }
     hdr_reset_internal_counters(histogram_.get());
     InvalidateRecordedSnapshot();
+    reset_count_++;
     exceeds_ = (exceeds_ > other.exceeds_) ? exceeds_ - other.exceeds_ : 0;
     return static_cast<double>(dropped);
   };
@@ -1844,6 +1904,8 @@ void HistogramImpl::AddMethods(Isolate* isolate, Local<FunctionTemplate> tmpl) {
                             &fast_get_ewma_error_rate_);
   SetProtoMethodNoSideEffect(isolate, tmpl, "export", DoExport);
   SetProtoMethodNoSideEffect(isolate, tmpl, "snapshot", DoSnapshot);
+  SetProtoMethodNoSideEffect(isolate, tmpl, "diff", DoDiff);
+  SetProtoMethodNoSideEffect(isolate, tmpl, "resetCount", GetResetCount);
   SetFastMethod(isolate, instance, "reset", DoReset, &fast_reset_);
 }
 
@@ -1894,6 +1956,8 @@ void HistogramImpl::RegisterExternalReferences(
   registry->Register(GetEwmaErrorRate);
   registry->Register(DoExport);
   registry->Register(DoSnapshot);
+  registry->Register(DoDiff);
+  registry->Register(GetResetCount);
   registry->Register(fast_get_ewma_mean_);
   registry->Register(fast_get_ewma_stddev_);
   registry->Register(fast_get_ewma_error_rate_);
@@ -3054,6 +3118,39 @@ void HistogramImpl::DoSnapshot(const FunctionCallbackInfo<Value>& args) {
   if (result) args.GetReturnValue().Set(result->object());
 }
 
+void HistogramImpl::DoDiff(const FunctionCallbackInfo<Value>& args) {
+  Environment* env = Environment::GetCurrent(args);
+  HistogramImpl* histogram = HistogramImpl::FromJSObject(args.This());
+  HistogramImpl* other = HistogramImpl::FromJSObject(args[0]);
+  Histogram::DiffError error;
+  std::shared_ptr<Histogram> diff =
+      (*histogram)->Diff(*(other->histogram()), &error);
+  switch (error) {
+    case Histogram::DiffError::kNone:
+      break;
+    case Histogram::DiffError::kOutOfMemory:
+      return THROW_ERR_MEMORY_ALLOCATION_FAILED(env);
+    case Histogram::DiffError::kIncompatible:
+      return THROW_ERR_INVALID_ARG_VALUE(
+          env, "other must have the same configuration as the histogram");
+    case Histogram::DiffError::kReset:
+      return THROW_ERR_INVALID_STATE(
+          env, "Values were removed from the histogram after other was taken");
+    case Histogram::DiffError::kNotEarlier:
+      return THROW_ERR_INVALID_ARG_VALUE(
+          env, "other contains values that are not in the histogram");
+  }
+
+  BaseObjectPtr<HistogramBase> result =
+      HistogramBase::Create(env, std::move(diff));
+  if (result) args.GetReturnValue().Set(result->object());
+}
+
+void HistogramImpl::GetResetCount(const FunctionCallbackInfo<Value>& args) {
+  HistogramImpl* histogram = HistogramImpl::FromJSObject(args.This());
+  args.GetReturnValue().Set(static_cast<double>((*histogram)->ResetCount()));
+}
+
 void HistogramImpl::GetPercentilesAt(const FunctionCallbackInfo<Value>& args) {
   Environment* env = Environment::GetCurrent(args);
   HistogramImpl* histogram = HistogramImpl::FromJSObject(args.This());

src/histogram.h

@@ -65,6 +65,23 @@ class Histogram : public MemoryRetainer {
   // if the copy cannot be allocated.
   std::shared_ptr<Histogram> Clone() const;
 
+  enum class DiffError {
+    kNone,
+    kOutOfMemory,
+    // `other` has a different layout.
+    kIncompatible,
+    // Values were removed from this histogram after `other` was taken.
+    kReset,
+    // `other` contains values that this histogram does not.
+    kNotEarlier,
+  };
+
+  // Returns a new histogram containing the values recorded in this histogram
+  // after `other`, an earlier copy of it, was taken. Returns nullptr and sets
+  // `error` if the difference cannot be computed.
+  std::shared_ptr<Histogram> Diff(const Histogram& other,
+                                  DiffError* error) const;
+
   Histogram(HistogramPointer histogram, const Options& options);
   virtual ~Histogram() = default;
 
@@ -80,6 +97,7 @@ class Histogram : public MemoryRetainer {
   inline int64_t Percentile(double percentile) const;
   inline size_t Exceeds() const;
   inline size_t Count() const;
+  inline uint64_t ResetCount() const;
 
   inline uint64_t RecordDelta();
 
@@ -165,10 +183,13 @@ class Histogram : public MemoryRetainer {
   inline void UpdateEwma(double value);
   inline void InvalidateRecordedSnapshot();
   size_t GetCachedRecordedSnapshotMemorySize() const;
+  std::shared_ptr<Histogram> CreateWithSameLayout() const;
 
   HistogramPointer histogram_;
   uint64_t prev_ = 0;
   size_t exceeds_ = 0;
+  // Incremented whenever recorded values are removed by Reset() or Subtract().
+  uint64_t reset_count_ = 0;
 
   // EWMA state (active when ewma_alpha_ > 0)
   double ewma_alpha_ = 0;
@@ -242,6 +263,8 @@ class HistogramImpl {
   static void DoExport(const v8::FunctionCallbackInfo<v8::Value>& args);
   static void DoImport(const v8::FunctionCallbackInfo<v8::Value>& args);
   static void DoSnapshot(const v8::FunctionCallbackInfo<v8::Value>& args);
+  static void DoDiff(const v8::FunctionCallbackInfo<v8::Value>& args);
+  static void GetResetCount(const v8::FunctionCallbackInfo<v8::Value>& args);
 
   static void FastReset(v8::Local<v8::Value> receiver);
   static double FastGetCount(v8::Local<v8::Value> receiver);