Geo Replica is not replicating

[ton3r]

Hi,

I've created two docker-compose cluster instances.

After both are up and running I've added the following on each

## on cluster 002
bin/pulsar-admin clusters create \
--broker-url pulsar://pulsar003.gcloud.spx:6650 \
--url http://pulsar003.gcloud.spx:8080 \
cluster-003



## on cluster 003
bin/pulsar-admin clusters create \
--broker-url pulsar://pulsar002.gcloud.spx:6650 \
--url http://pulsar002.gcloud.spx:8080 \
cluster-002

Right after that, I've executed the following on BOTH :

bin/pulsar-admin tenants create tom-replicated-tenant \
--admin-roles my-admin-role \
--allowed-clusters cluster-002,cluster-003

bin/pulsar-admin namespaces create tom-replicated-tenant/replic-ns

bin/pulsar-admin namespaces set-clusters tom-replicated-tenant/replic-ns \
--clusters cluster-002,cluster-003

Then I've created a topic on cluster-002.
Now I've tried to check this with

bin/pulsar-admin topics stats-internal tom-replicated-tenant/replic-ns/re-topic

Cluster-002 (where I've created the topic) says :

{
  "entriesAddedCounter" : 22,
  "numberOfEntries" : 22,
  "totalSize" : 86240,
  "currentLedgerEntries" : 22,
  "currentLedgerSize" : 86240,
  "lastLedgerCreatedTimestamp" : "2023-12-21T07:40:25.625Z",
  "waitingCursorsCount" : 0,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "617:21",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 617,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false,
    "underReplicated" : false
  } ],
  "cursors" : {
    "pulsar.repl.cluster-003" : {
      "markDeletePosition" : "617:-1",
      "readPosition" : "617:0",
      "waitingReadOp" : false,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 0,
      "cursorLedger" : -1,
      "cursorLedgerLastEntry" : -1,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2023-12-21T07:40:25.635Z",
      "state" : "NoLedger",
      "active" : false,
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "subscriptionHavePendingRead" : false,
      "subscriptionHavePendingReplayRead" : false,
      "properties" : { }
    },
    "re-topic_sub" : {
      "markDeletePosition" : "617:16",
      "readPosition" : "617:17",
      "waitingReadOp" : false,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 17,
      "cursorLedger" : 619,
      "cursorLedgerLastEntry" : 1,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2023-12-21T07:40:25.641Z",
      "state" : "Open",
      "active" : false,
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "subscriptionHavePendingRead" : false,
      "subscriptionHavePendingReplayRead" : false,
      "properties" : {
        "pulsar.replicated.subscription" : 1
      }
    }
  },
  "schemaLedgers" : [ ],
  "compactedLedger" : {
    "ledgerId" : -1,
    "entries" : -1,
    "size" : -1,
    "offloaded" : false,
    "underReplicated" : false
  }
}

But cluster-003 says

I have no name!@broker:/pulsar$ bin/pulsar-admin topics stats-internal tom-replicated-tenant/replic-ns/re-topic
Topic persistent://tom-replicated-tenant/replic-ns/re-topic not found

Reason: Topic persistent://tom-replicated-tenant/replic-ns/re-topic not found

What am I doing wrong? Where can I see possible exceptions or problems?

Thanks a lot.

-- Tom

[ton3r]

Was a DNS problem

[ton3r]

Them didn't resolve our DNS inside of the cluster. Now we have some other problems. But will see

 Not enough bookies to create ledger with ensembleSize=1, writeQuorumSize=1 and ackQuorumSize=1
Failed to initialize managed ledger: Not enough non-faulty bookies available error code: -6

[lhotari]

One common reason for bookies not being writable is that there's not sufficient disk space available. By default, diskUsageThreshold is 0.95 . Explained at https://bookkeeper.apache.org/docs/reference/config/#disk-utilization .
Have you checked that you aren't hitting this limit? The bookkeeper logs would tell if it switches to read only mode because of the disk space.

[ton3r]

I think it was a troubled setup in my cluster. After making cluster available over IP some things could get resolved, others seems still to try to replicate over DNS which wasn't working. After cleaning up the cluster setup and restart of each everthing seems to went fine.