pagination: add pagination to wallet transactions #8998

Abdulkbk · 2024-08-11T15:54:24Z

Change Description

This PR adds pagination to getting transactions RPC calls by introducing index_offset and max_transactions
to specify the number of transactions to skip and the maximum number of transactions to return respectively.

Steps to Test

Steps for reviewers to follow to test the change.

Start LND
Run lncli listchaintxns -index_offset=n -max_transactions=m where n and m are any numbers
The result will be a list skipping n numbers of transactions and with m transactions

Pull Request Checklist

Testing

Your PR passes all CI checks.
Tests covering the positive and negative (error paths) are included.
Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
Commits follow the Ideal Git Commit Structure.
Any new logging statements use an appropriate subsystem and logging level.
Any new lncli commands have appropriate tags in the comments for the rpc in the proto file.
There is a change description in the release notes, or [skip ci] in the commit message for small changes.

📝 Please see our Contribution Guidelines for further guidance.

coderabbitai · 2024-08-11T15:54:28Z

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

guggero

Thanks for the PR. Looks pretty good already, have a bit of early feedback.

lnrpc/lightning.proto

guggero · 2024-08-12T07:19:38Z

lnwallet/interface.go

@@ -393,7 +393,7 @@ type WalletController interface {
 	// retrieve the transactions relevant to a specific account. When
 	// empty, transactions of all wallet accounts are returned.
 	ListTransactionDetails(startHeight, endHeight int32,
-		accountFilter string) ([]*TransactionDetail, error)
+		accountFilter string, indexOffset int, maxTnxs int) ([]*TransactionDetail, error)


nit: break before maxTnxs to avoid too long line. Also, should be spelled maxTxns?

Avoiding a long line was why I named it maxTnx, but after breaking the line, I now renamed it to maxTransactions

Looks like this is still unaddressed?

Oh I see, I unintentionally addressed this in another commit

rpcserver.go

lnwallet/btcwallet/btcwallet.go

Abdulkbk · 2024-08-12T10:18:26Z

Thanks for the PR. Looks pretty good already, have a bit of early feedback.

Thanks for taking a look. I will address the feedback you left.

Abdulkbk · 2024-08-13T22:12:41Z

Thanks for the PR. Looks pretty good already, have a bit of early feedback.

I've addressed the feedback you left, added a few changes too.

guggero

Thanks for adding the extra parameters. Still needs a few updates before I'll start manual testing. But getting there 👍

cmd/lncli/commands.go

guggero · 2024-08-16T09:09:17Z

rpcserver.go

@@ -6098,8 +6098,15 @@ func (r *rpcServer) GetTransactions(ctx context.Context,
 		endHeight = req.EndHeight
 	}

+	// To remain backward compatible, If the number of transactions was not
+	// specified, then we'll default to returning the entire transactions
+	if req.MaxTransactions == 0 {


I think this can be replaced by the use of a default value. See comment above.

I added the default value of -1 when max_transactions is not specified from the CLI. However, for RPC, the default appears to be 0, so I am checking and replacing it with -1. I used this as a reference.

Right, RPC users will have a default value of 0, makes sense then.

lnwallet/test/test_interface.go

guggero · 2024-08-16T09:13:27Z

lnwallet/interface.go

@@ -393,7 +393,7 @@ type WalletController interface {
 	// retrieve the transactions relevant to a specific account. When
 	// empty, transactions of all wallet accounts are returned.
 	ListTransactionDetails(startHeight, endHeight int32,
-		accountFilter string) ([]*TransactionDetail, error)
+		accountFilter string, indexOffset int, maxTnxs int) ([]*TransactionDetail, error)


Looks like this is still unaddressed?

lightninglabs-deploy · 2024-11-04T00:34:52Z

@Abdulkbk, remember to re-request review from reviewers when ready

yyforyongyu · 2024-11-06T03:39:59Z

Approved CI run

guggero

Thanks for the updates! Did another pass, we're definitely getting closer.

lnrpc/lightning.proto

lnwallet/btcwallet/btcwallet.go

lnrpc/lightning.proto

cmd/commands/commands.go

guggero · 2024-11-06T08:30:53Z

lnwallet/btcwallet/btcwallet.go

-	return txDetails[indexOffset:end], nil
+	txDetails = txDetails[indexOffset:end]
+
+	return txDetails, uint64(firstIdx), uint64(lastIdx), nil


Why do we need these two extra variables at all? Shouldn't firstIdx = indexOffset and lastIdx = end-1?

If my previous comment is correct, then this wouldn't be the case. However, I can replace the end variable with lastIdx, since they are the same.

guggero · 2024-11-06T08:32:46Z

lnwallet/btcwallet/btcwallet.go

+	// If maxTransactions is set to -1, then we'll return all transactions
+	// starting from the offset.
+	if maxTransactions == -1 {
+		lastIdx = len(txDetails)


This is incorrect. The last index is len(txDetails)-1. But for the slice access ([a:b]) it's correct, since b is exclusive.

My understanding is that first_index_offset and last_index_offset are 1-based indices. I am using the implementation found here as a reference, and it states here that the AddIndex starts from 1.

Also since the last_index_offset will be used as the new index_offset for the next call, wouldn't returning len(txDetails)-1 result in repeating the last transaction from the previous call as the first transaction in the new call? This is because the slice access [indexOffset: ...] is inclusive.

guggero · 2024-11-06T08:33:22Z

rpcserver.go

-		req.StartHeight, endHeight, req.Account, int(req.IndexOffset),
-		int(req.MaxTransactions),
-	)
+	transactions, firstIdx, lastIdx, err :=


nit: same here re transactions -> txns to optimize formatting.

I replaced "transactions" with "txns," but the line is still too long.

guggero · 2024-11-06T08:34:46Z

lnwallet/mock.go


-	return nil, nil
+	return nil, 0, 0, nil


These changes need to be in the previous commit to make that commit compile on its own.
A nice helper for making sure every commit compiles is:

git rebase -i origin/master --exec "make unit pkg=... case=_NONE_"

Which basically goes through each commit and compiles all unit tests (but doesn't run them), which causes all code to be compiled as well.

I squashed the last 2 commits and now they all compile successfully.

Nice helper 👍

lnwallet/test/test_interface.go

This commit adds index_offset and max_transactions to the list of parameters passed during gettransactions call. index_offset specify transactions to skip and max_transactions the total transactions returned

This commit modifies listtransactiondetails method definition to take in additional params: index_offset and maxTxn

Here we handle the offset and max_transactions parameters passed through lncli and RPC call.

Addition of first_index_offset and last_index_offset that can be used to seek further transactions from where you stopped in a previous call. We also add unit test to cover getting exactly how many transactions we requested for, and first and last index offsets are returned correctly.

Abdulkbk marked this pull request as draft August 11, 2024 15:57

guggero reviewed Aug 12, 2024

View reviewed changes

Abdulkbk marked this pull request as ready for review August 12, 2024 15:30

Abdulkbk force-pushed the trx-pagination branch 2 times, most recently from 21fb0a8 to a6609f5 Compare August 13, 2024 21:38

Abdulkbk mentioned this pull request Aug 13, 2024

Provide ability to paginate transactions similar to invoices and payments #4719

Open

guggero self-requested a review August 14, 2024 07:48

guggero reviewed Aug 16, 2024

View reviewed changes

Abdulkbk force-pushed the trx-pagination branch 2 times, most recently from 31aed37 to 594f3f1 Compare August 23, 2024 01:05

Abdulkbk force-pushed the trx-pagination branch from 594f3f1 to e4c47bb Compare October 26, 2024 16:42

Abdulkbk force-pushed the trx-pagination branch from e4c47bb to e5458b9 Compare November 5, 2024 13:34

Abdulkbk requested a review from guggero November 5, 2024 13:36

guggero reviewed Nov 6, 2024

View reviewed changes

lnrpc: modify gettransaction parameters

10dbd4f

This commit adds index_offset and max_transactions to the list of parameters passed during gettransactions call. index_offset specify transactions to skip and max_transactions the total transactions returned

Abdulkbk force-pushed the trx-pagination branch from e5458b9 to c5dea0e Compare November 7, 2024 11:59

Abdulkbk added 2 commits November 8, 2024 23:12

lnwallet+lnrpc+lntest: modify listtxn definition

940a466

This commit modifies listtransactiondetails method definition to take in additional params: index_offset and maxTxn

lnd+lncli: account for two more parameters

8423cfd

Here we handle the offset and max_transactions parameters passed through lncli and RPC call.

Abdulkbk force-pushed the trx-pagination branch 2 times, most recently from 81f9f33 to 36a59ed Compare November 10, 2024 13:49

Abdulkbk force-pushed the trx-pagination branch from 36a59ed to 650de87 Compare November 10, 2024 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pagination: add pagination to wallet transactions #8998

pagination: add pagination to wallet transactions #8998

Abdulkbk commented Aug 11, 2024

coderabbitai bot commented Aug 11, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

guggero left a comment

guggero Aug 12, 2024

Abdulkbk Aug 12, 2024

guggero Aug 16, 2024

Abdulkbk Aug 19, 2024 •

edited

Loading

Abdulkbk commented Aug 12, 2024

Abdulkbk commented Aug 13, 2024

guggero left a comment

guggero Aug 16, 2024

Abdulkbk Oct 26, 2024

guggero Nov 6, 2024

guggero Aug 16, 2024

lightninglabs-deploy commented Nov 4, 2024

yyforyongyu commented Nov 6, 2024

guggero left a comment

guggero Nov 6, 2024

Abdulkbk Nov 10, 2024 •

edited

Loading

guggero Nov 6, 2024

Abdulkbk Nov 10, 2024

guggero Nov 6, 2024

Abdulkbk Nov 10, 2024

guggero Nov 6, 2024

Abdulkbk Nov 10, 2024

pagination: add pagination to wallet transactions #8998

Are you sure you want to change the base?

pagination: add pagination to wallet transactions #8998

Conversation

Abdulkbk commented Aug 11, 2024

Change Description

Steps to Test

Pull Request Checklist

Testing

Code Style and Documentation

coderabbitai bot commented Aug 11, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

guggero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abdulkbk Aug 19, 2024 • edited Loading

Choose a reason for hiding this comment

Abdulkbk commented Aug 12, 2024

Abdulkbk commented Aug 13, 2024

guggero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightninglabs-deploy commented Nov 4, 2024

yyforyongyu commented Nov 6, 2024

guggero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abdulkbk Nov 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderabbitai bot commented Aug 11, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

Abdulkbk Aug 19, 2024 •

edited

Loading

Abdulkbk Nov 10, 2024 •

edited

Loading