Deadlock WorkloadState/BenchmarkState deadly embrace

[ajmaidak]

When running with a large number of terminals with the wikipedia benchmark I'm regularly seeing the following java deadlock:

Found one Java-level deadlock:
=============================
"main":
  waiting to lock monitor 0x00007f6dc8000be0 (object 0x00000000874f0098, a com.oltpbenchmark.WorkloadState),
  which is held by "WikipediaWorker<053>"

"WikipediaWorker<053>":
  waiting to lock monitor 0x00007f6dcc000be0 (object 0x00000000874f00c8, a com.oltpbenchmark.BenchmarkState),
  which is held by "main"

Java stack information for the threads listed above:
===================================================
"main":
	at com.oltpbenchmark.ThreadBench.runRateLimitedMultiPhase(ThreadBench.java:212)
	- waiting to lock <0x00000000874f0098> (a com.oltpbenchmark.WorkloadState)
	- locked <0x00000000874f00c8> (a com.oltpbenchmark.BenchmarkState)
	at com.oltpbenchmark.ThreadBench.runRateLimitedBenchmark(ThreadBench.java:54)
	at com.oltpbenchmark.DBWorkload.runWorkload(DBWorkload.java:646)
	at com.oltpbenchmark.DBWorkload.main(DBWorkload.java:460)
"WikipediaWorker<053>":
	at com.oltpbenchmark.BenchmarkState.getState(BenchmarkState.java:55)
	- waiting to lock <0x00000000874f00c8> (a com.oltpbenchmark.BenchmarkState)
	at com.oltpbenchmark.WorkloadState.fetchWork(WorkloadState.java:142)
	- locked <0x00000000874f0098> (a com.oltpbenchmark.WorkloadState)
	at com.oltpbenchmark.api.Worker.run(Worker.java:233)
	at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

Found 1 deadlock.

For reference this is my workload configuration file:

<?xml version="1.0"?>
<parameters>

    <!-- Connection details -->
    <type>MYSQL</type>
    <driver>com.mysql.cj.jdbc.Driver</driver>
    <url>jdbc:mysql://ajm-mysql-2:3306/benchbase?rewriteBatchedStatements=true&amp;sslMode=DISABLED</url>
    <username>admin</username>
    <password></password>
    <isolation>TRANSACTION_REPEATABLE_READ</isolation>
    <batchsize>128</batchsize>

    <!-- Scale factor is the number of wikipages *1000 -->
    <scalefactor>10</scalefactor>

    <!-- The workload -->
    <terminals>96</terminals>
    <works>
        <work>
            <time>60</time>
            <rate>30000</rate>
            <weights>1,1,7,90,1</weights>
        </work>
        <work>
            <time>60</time>
            <rate>35000</rate>
            <weights>1,1,7,90,1</weights>
        </work>
        <work>
            <time>60</time>
            <rate>40000</rate>
            <weights>1,1,7,90,1</weights>
        </work>
    </works>

    <!-- Wikipedia Procedures Declaration -->
    <transactiontypes>
        <transactiontype>
            <name>AddWatchList</name>
        </transactiontype>
        <transactiontype>
            <name>RemoveWatchList</name>
        </transactiontype>
        <transactiontype>
            <name>UpdatePage</name>
        </transactiontype>
        <transactiontype>
            <name>GetPageAnonymous</name>
        </transactiontype>
        <transactiontype>
            <name>GetPageAuthenticated</name>
        </transactiontype>
    </transactiontypes>
</parameters>

[apavlo]

@ajmaidak I think that this is the first time I have seen this issue. How many CPU cores do you have on your box?

[ajmaidak]

The VM executing benchbase has 32 vcpus, a e2-highcpu-32 machine in google cloud.

I observed the deadlock most often with >=32 terminals at rates >20000 running the wikipedia benchmark.

[apavlo]

@ajmaidak Do you see it with other benchmarks?