BP-42: New Client API - list ledgers by nicoloboschi · Pull Request #2457 · apache/bookkeeper

nicoloboschi · 2020-10-26T15:50:05Z

Implementation of BP-42

Motivation

Enhance new client API, added methods for

listing ledgers
access to ledger metadata

Changes

Added new interfaces in "api" client package for ledgers listing
Added new method getLedgerMetadata(long ledgerId) to BookKeeper interface
Added getLedgerId() to LedgerMetadata type
-- Ledger metadata ser/deser are not changed since ledgerId is not stored inside ZK data node

Updated all tests using LedgerMetadataBuilder because ledgerId is now required in order to call build() method

Master Issue: #2422

eolivelli

great work.
I left a few comments, mostly nits

@jiazhai @sijie @Ghatage PTAL

jiazhai

+1

nicoloboschi · 2020-10-28T09:45:22Z

@eolivelli I made some changes as requested

Checks passes except for SlowBookieTest but I think it is flaky, running it locally passes

eolivelli · 2020-10-28T10:14:51Z

test failures seem related to this patch

Error:  org.apache.bookkeeper.meta.AbstractZkLedgerManagerTest.testReadLedgerMetadataSuccess
Error:    Run 1: AbstractZkLedgerManagerTest.testReadLedgerMetadataSuccess:361 expected: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}> but was: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}>
Error:    Run 2: AbstractZkLedgerManagerTest.testReadLedgerMetadataSuccess:361 expected: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}> but was: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}>
Error:    Run 3: AbstractZkLedgerManagerTest.testReadLedgerMetadataSuccess:361 expected: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}> but was: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}>
[INFO] 
Error:  org.apache.bookkeeper.meta.AbstractZkLedgerManagerTest.testUnregisterLedgerMetadataListener
Error:    Run 1: AbstractZkLedgerManagerTest.testUnregisterLedgerMetadataListener:809 expected: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}> but was: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}>
Error:    Run 2: AbstractZkLedgerManagerTest.testUnregisterLedgerMetadataListener:809 expected: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}> but was: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}>
Error:    Run 3: AbstractZkLedgerManagerTest.testUnregisterLedgerMetadataListener:809 expected: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}> but was: org.apache.bookkeeper.client.LedgerMetadataImpl<LedgerMetadata{formatVersion=3, ensembleSize=5, writeQuorumSize=3, ackQuorumSize=3, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[192.0.2.1:3181, 192.0.2.2:3181, 192.0.2.3:3181, 192.0.2.4:3181, 192.0.2.5:3181]}, customMetadata={}}>
[INFO] 
Error:  Errors: 
Error:  org.apache.bookkeeper.meta.TestLedgerMetadataSerDe.testPeggedToV3SerDe
Error:    Run 1: TestLedgerMetadataSerDe.testPeggedToV3SerDe:133 » IllegalArgument Ledger id mu...
Error:    Run 2: TestLedgerMetadataSerDe.testPeggedToV3SerDe:133 » IllegalArgument Ledger id mu...
Error:    Run 3: TestLedgerMetadataSerDe.testPeggedToV3SerDe:133 » IllegalArgument Ledger id mu...
[INFO] 
Error:  org.apache.bookkeeper.meta.TestLedgerMetadataSerDe.testStoreSystemtimeAsLedgerCtimeDisabledWithNewerVersion
Error:    Run 1: TestLedgerMetadataSerDe.testStoreSystemtimeAsLedgerCtimeDisabledWithNewerVersion:171 » IllegalArgument
Error:    Run 2: TestLedgerMetadataSerDe.testStoreSystemtimeAsLedgerCtimeDisabledWithNewerVersion:171 » IllegalArgument
Error:    Run 3: TestLedgerMetadataSerDe.testStoreSystemtimeAsLedgerCtimeDisabledWithNewerVersion:171 » IllegalArgument
[INFO] 
Error:  org.apache.bookkeeper.meta.TestLedgerMetadataSerDe.testStoreSystemtimeAsLedgerCtimeEnabledWithNewerVersion
Error:    Run 1: TestLedgerMetadataSerDe.testStoreSystemtimeAsLedgerCtimeEnabledWithNewerVersion:151 » IllegalArgument
Error:    Run 2: TestLedgerMetadataSerDe.testStoreSystemtimeAsLedgerCtimeEnabledWithNewerVersion:151 » IllegalArgument
Error:    Run 3: TestLedgerMetadataSerDe.testStoreSystemtimeAsLedgerCtimeEnabledWithNewerVersion:151 » IllegalArgument

eolivelli · 2020-10-28T10:14:58Z

rerun-failure-checks

eolivelli · 2020-10-28T10:18:18Z

rerun failure checks

eolivelli

LGTM

nicoloboschi added 2 commits October 26, 2020 16:24

BP-42: New Client API - list ledgers

6834064

Javadoc and checkstyle

4473a1f

eolivelli requested changes Oct 26, 2020

View reviewed changes

jiazhai approved these changes Oct 27, 2020

View reviewed changes

nicoloboschi added 6 commits October 27, 2020 14:21

Javadoc and checkstyle

9bc797d

Enrico's comments

ae541bf

Enrico's comments

6f81e56

Fix tests

cc61663

fix checkstyle

3cdd065

fix etcd tests

f941b0b

nicoloboschi added 2 commits October 28, 2020 11:30

fix LedgerMetadataImpl#equals

c30f5b3

fix tests

a8a9aaf

eolivelli approved these changes Oct 29, 2020

View reviewed changes

eolivelli added this to the 4.12.0 milestone Oct 29, 2020

eolivelli added area/client release/4.12.0 type/feature labels Oct 29, 2020

eolivelli assigned nicoloboschi Oct 29, 2020

eolivelli merged commit 0c84186 into apache:master Oct 29, 2020

geniusjoe mentioned this pull request Mar 17, 2026

Issue 2422: Fix SyncLedgerIterator.hasNext() failing to iterate across ZK ledger ranges #4731

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

BP-42: New Client API - list ledgers#2457

BP-42: New Client API - list ledgers#2457
eolivelli merged 10 commits into
apache:masterfrom
nicoloboschi:fix/2422/bp-42-list-ledgers

nicoloboschi commented Oct 26, 2020

Uh oh!

eolivelli left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jiazhai left a comment

Uh oh!

nicoloboschi commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 28, 2020

Uh oh!

eolivelli left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

nicoloboschi commented Oct 26, 2020

Motivation

Changes

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jiazhai left a comment

Choose a reason for hiding this comment

Uh oh!

nicoloboschi commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 28, 2020

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants