Investigate potential high memory usage with r2dbc-mssql/r2dbc-mysql instrumentation

[jasonjkeller]

Related to https://new-relic.atlassian.net/browse/NR-299687

Customer reported that they were getting an OOM with their service using SpringBoot and R2DBC Mysql.

It was confirmed that disabling the r2dbc-mysql instrumentation resolved the memory issues:

  class_transformer:
    com.newrelic.instrumentation.r2dbc-mysql:
      enabled: false

This may be related to the the use of the @NewField annotation and a known issue with how they interact with the caffeine cache when there are many instances of the annotation.

newrelic-java-agent/instrumentation/r2dbc-mssql/src/main/java/io/r2dbc/mssql/client/ReactorNettyClient_Instrumentation.java

Lines 8 to 16 in d35ffd6

	@Weave(type = MatchType.ExactClass, originalName = "io.r2dbc.mssql.client.ReactorNettyClient")
	public class ReactorNettyClient_Instrumentation {
	@NewField
	public final Connection clientConnection;
	private ReactorNettyClient_Instrumentation(Connection connection, TdsEncoder TdsEncoder, ConnectionContext context) {
	this.clientConnection = connection;
	}
	}

newrelic-java-agent/instrumentation/r2dbc-mysql/src/main/java/dev/miku/r2dbc/mysql/client/ReactorNettyClient_Instrumentation.java

Lines 10 to 18 in d35ffd6

	@Weave(type = MatchType.ExactClass, originalName = "dev.miku.r2dbc.mysql.client.ReactorNettyClient")
	class ReactorNettyClient_Instrumentation {
	@NewField
	public final Connection clientConnection;
	ReactorNettyClient_Instrumentation(Connection connection, MySqlSslConfiguration ssl, ConnectionContext context) {
	this.clientConnection = connection;
	}
	}

It may also be the underlying netty-reactor instrumentation that is the real issue as the heap dump shows a lot of cached entries for reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.

Here is the instrumentation that was loaded in the environment that was experiencing the OOM:

com.newrelic.instrumentation.rabbit-amqp-5.0.0
com.newrelic.instrumentation.jdbc-driver
com.newrelic.instrumentation.spring-webclient-5.0
com.newrelic.instrumentation.netty-4.1.16
com.newrelic.instrumentation.netty-reactor-0.9.0
com.newrelic.instrumentation.httpurlconnection
com.newrelic.instrumentation.java.logging-jdk8
com.newrelic.instrumentation.logback-classic-1.2
com.newrelic.instrumentation.jdbc-mysql-connection-8.0.11
com.newrelic.instrumentation.jdbc-mysql-8.0.11
com.newrelic.instrumentation.spring-cache-3.1.0
com.newrelic.instrumentation.r2dbc-mysql
com.newrelic.instrumentation.java.process
com.newrelic.instrumentation.jdbc-socket
com.newrelic.instrumentation.kafka-clients-spans-0.11.0.0
com.newrelic.instrumentation.spring-webflux-controller-mappings-5.0.0
com.newrelic.instrumentation.netty-4.0.8
com.newrelic.instrumentation.java.completable-future-jdk11
com.newrelic.instrumentation.java.completable-future-jdk8u40
com.newrelic.instrumentation.httpclient-4.0
com.newrelic.instrumentation.spring-webflux-5.3.0
com.newrelic.instrumentation.jdbc-generic
com.newrelic.instrumentation.kafka-clients-metrics-3.0.0

[workato-integration]

https://new-relic.atlassian.net/browse/NR-304225