bigtable: Slow memory leak with Direct Access enabled

[philpearl]

Client

cloud.google.com/go/bigtable v1.52.0
google.golang.org/grpc v1.83.0

Environment

Go 1.26, scratch containers on GKE

Code and Dependencies

Code is too big and ugly to list here. But it's just doing normal reads and writes to a Bigtable table

Expected behavior

Memory use doesn't keep growing

Actual behavior

Memory use does keep growing.

Screenshots

Additional context

This is very similar to github.com//issues/14582, but slower. Profiles point to similar objects being leaked. The leak is more apparent in services with low traffic - but that's perhaps because their memory usage is very small compared to the size of the leak.

Here's some profile output showing inuse_objects and inuse_space compared to a baseline profile taken about 18 hours before

Showing nodes accounting for 369747, 51.91% of 712220 total
Dropped 68 nodes (cum <= 3561)
Showing top 20 nodes out of 213
      flat  flat%   sum%        cum   cum%
    120152 16.87% 16.87%     120152 16.87%  net.JoinHostPort (inline)
     52025  7.30% 24.17%     172177 24.17%  google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
     32768  4.60% 28.78%      32768  4.60%  google.golang.org/grpc/internal/xds/balancer/outlierdetection.newCallCounter
     32768  4.60% 33.38%      32768  4.60%  google.golang.org/protobuf/internal/filedesc.(*Builder).optionsUnmarshaler
     21845  3.07% 36.44%     -10924  1.53%  google.golang.org/grpc/internal/hierarchy.SetInEndpoint
     16384  2.30% 38.74%      16384  2.30%  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).switchTo
     16384  2.30% 41.04%      16384  2.30%  google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).populateAggregateClusterLocked
     16384  2.30% 43.34%      16384  2.30%  google.golang.org/grpc/internal/xds/xdsdepmgr.newClusterWatcher
     10923  1.53% 44.88%      10923  1.53%  fmt.Sprintf
     10923  1.53% 46.41%      10923  1.53%  google.golang.org/grpc/balancer/rls/internal/keys.MakeBuilderMap
      6554  0.92% 47.33%       6554  0.92%  google.golang.org/grpc/internal/xds/clients/xdsclient.(*AnyProto).ToAny
      4681  0.66% 47.99%       4681  0.66%  reflect.unsafe_New
      4096  0.58% 48.56%       4096  0.58%  github.com/modern-go/reflect2.newUnsafeStructField
      4096  0.58% 49.14%       4096  0.58%  github.com/unravelin/core/feature-extraction-service/proto.buildFlattenStruct
      4096  0.58% 49.71%       4096  0.58%  google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString
      3641  0.51% 50.23%       3641  0.51%  encoding/json.(*decodeState).literalStore
      3641  0.51% 50.74%      24822  3.49%  encoding/json.(*decodeState).object
      3344  0.47% 51.21%       3344  0.47%  runtime.mallocgc
      2521  0.35% 51.56%      -2160   0.3%  context.(*cancelCtx).propagateCancel
      2521  0.35% 51.91%       2521  0.35%  google.golang.org/grpc/balancer/weightedtarget/weightedaggregator.(*Aggregator).Add

Showing nodes accounting for 19.54MB, 27.57% of 70.88MB total
Dropped 35 nodes (cum <= 0.35MB)
Showing top 20 nodes out of 225
      flat  flat%   sum%        cum   cum%
    6.50MB  9.18%  9.18%        9MB 12.70%  google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
    2.50MB  3.53% 12.70%     2.50MB  3.53%  net.JoinHostPort (inline)
    1.50MB  2.12% 14.83%     1.50MB  2.12%  google.golang.org/protobuf/internal/impl.consumeBytesNoZero
       1MB  1.41% 16.24%        1MB  1.41%  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).switchTo
    0.52MB  0.73% 16.97%     0.52MB  0.73%  google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile
    0.51MB  0.72% 17.69%     0.51MB  0.72%  github.com/unravelin/core/lib/lang/internal/names.map.init.0
    0.50MB  0.71% 18.40%     0.50MB  0.71%  math/rand.newSource
    0.50MB  0.71% 19.11%     0.50MB  0.71%  github.com/unravelin/core/pspdomain.map.init.1
    0.50MB  0.71% 19.81%     0.50MB  0.71%  cloud.google.com/go/bigtable/internal/metrics.NewFactory
    0.50MB  0.71% 20.52%     0.50MB  0.71%  google.golang.org/grpc/balancer/weightedtarget/weightedaggregator.(*Aggregator).Add
    0.50MB  0.71% 21.22%     0.50MB  0.71%  google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).addOrGetActiveClusterInfo
    0.50MB  0.71% 21.93%        1MB  1.41%  google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).applyRouteConfigUpdateLocked
    0.50MB  0.71% 22.63%     1.50MB  2.12%  google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).populateClusterConfigLocked
    0.50MB  0.71% 23.34%     0.50MB  0.71%  google.golang.org/protobuf/internal/impl.(*MessageInfo).makeFieldTypes
    0.50MB  0.71% 24.05%     0.50MB  0.71%  reflect.mapassign_faststr0
    0.50MB  0.71% 24.75%     0.50MB  0.71%  encoding/json.(*decodeState).literalStore
    0.50MB  0.71% 25.46%     2.50MB  3.53%  encoding/json.(*decodeState).object
    0.50MB  0.71% 26.16%     0.50MB  0.71%  github.com/modern-go/reflect2.newUnsafeStructField
    0.50MB  0.71% 26.87%     0.50MB  0.71%  github.com/unravelin/core/feature-extraction-service/proto.buildFlattenStruct
    0.50MB  0.71% 27.57%     0.50MB  0.71%  google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString