Added --json flag to ec-admin running command #5332

Suvrat1629 · 2025-02-13T15:44:29Z

This PR enhances the accumulo ec-admin running command by adding a --format flag to support structured output in CSV and JSON formats. This allows for easier programmatic parsing of running compaction details.

Changes Introduced:
Added --format flag (json, csv, human [default]) to ec-admin running.
Implemented structured output formatting for better readability and parsing.
Updated runningCompactions() method to handle different output formats.
Ensured compatibility with existing behavior by keeping table format as the default.

Fixes #5205

Suvrat1629 · 2025-02-13T15:46:40Z

@kevinrr888 I hope this fixes the fix the issue with the commit history :)

kevinrr888

General comments. I suggest running mvn clean verify -Psunny before committing future changes. This will catch a lot of potential issues with new code.

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java

…min.java Co-authored-by: Kevin Rathbun <[email protected]>

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java

Concerns regarding build failures have been addressed. No further comments at this time

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java

Co-authored-by: Dave Marion <[email protected]>

dlmarion · 2025-02-20T20:38:23Z

This looks pretty good. Have you tested this locally?

Suvrat1629 · 2025-02-21T03:52:25Z

@dlmarion No, I did not understand how to setup the development environment but I wanted to contribute so I decided to open a pr regardless. Sorry about that😅

dlmarion · 2025-02-27T18:19:14Z

@Suvrat1629 - I would suggest looking at https://github.com/apache/fluo-uno to get a development environment set up so that you can test this locally.

Suvrat1629 · 2025-02-28T07:13:48Z

@dlmarion

git clone https://github.com/apache/fluo-uno.git
cd fluo-uno
./bin/uno fetch accumulo            # Fetches binary tarballs of Accumulo and its dependencies
./bin/uno setup accumulo            # Sets up Accumulo and its dependencies (Hadoop & ZooKeeper)
source <(./bin/uno env)             # Bash-specific command that sets up current shell

I ran this command and got done with the set up.
But i don't how I can test the changes that I have made locally. Where do i run the ./accumulo ec-admin command?

dlmarion · 2025-02-28T14:00:07Z

@Suvrat1629 - looking at the fluo-uno readme, it looks like you should be able to set the ACCUMULO_REPO environment variable in the uno.conf file and then fluo-uno will build Accumulo from your feature branch. See https://github.com/apache/fluo-uno/blob/main/conf/uno.conf#L46. Once everything is running, Accumulo should be running from the directory specified by the ACCUMULO_HOME variable (see https://github.com/apache/fluo-uno/blob/main/conf/uno.conf#L126). You should be able to run the accumulo ec-admin from the bin directory in $ACCUMULO_HOME.

ctubbsii · 2025-04-08T19:44:53Z

I like the idea of having a standard computer-readable output, but I'd prefer not to have CSV as an option. There are multiple different CSV standards, and they all do things slightly differently with quoting, spacing, redundant delimiters, etc. Plus, Java's String.split function is very confusing with splitting on a delimiter, making it harder to parse correctly. JSON or some other standard is fine, though.
(Note: this is a general comment, as I haven't looked at this specific PR's implementation yet)

ddanielr · 2025-04-16T16:39:33Z

I think the csv output is due to how I wrote the issue ticket. I believe a json only output would be satisfactory to close the issue ticket.

Refactored the ECAdmin command and added a test. Downside to this implementation is that it will consume more memory in the case where json output is not requested.

ddanielr

Looks good to me.
Saw the output from the test and should be able to use something like jq to correctly summarize compaction info based on table or queue.


[
  {
    "ecid": "ECID:2332d2fd-a26c-4014-a48b-9a4871baafbc",
    "addr": "ip-10-113-15-93.evoforge.org:9133",
    "kind": "USER",
    "queueName": "DCQ7",
    "ke": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU\u003d",
    "tableId": "1",
    "status": "STARTED",
    "lastUpdate": 806,
    "duration": 1,
    "numFiles": 1,
    "progress": 0.0
  }
]

Added option to print the running compactions as json. Co-authored-by: Kevin Rathbun <[email protected]> Co-authored-by: Dave Marion <[email protected]>

added formatting flags

a34562a

kevinrr888 previously requested changes Feb 13, 2025

View reviewed changes

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java Outdated Show resolved Hide resolved

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java Outdated Show resolved Hide resolved

Update server/base/src/main/java/org/apache/accumulo/server/util/ECAd…

35c4881

…min.java Co-authored-by: Kevin Rathbun <[email protected]>

ddanielr added this to the 2.1.4 milestone Feb 13, 2025

ddanielr reviewed Feb 13, 2025

View reviewed changes

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java Outdated Show resolved Hide resolved

switched to gson for json formatting

2b2907c

ddanielr reviewed Feb 13, 2025

View reviewed changes

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java Outdated Show resolved Hide resolved

reverted TExternalCompactionMap to TExternalCompactionList

ec6ea82

dlmarion reviewed Feb 20, 2025

View reviewed changes

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java Outdated Show resolved Hide resolved

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java Show resolved Hide resolved

added format validation and removed csv build up

f562a0c

dlmarion reviewed Feb 20, 2025

View reviewed changes

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java Outdated Show resolved Hide resolved

server/base/src/main/java/org/apache/accumulo/server/util/ECAdmin.java Outdated Show resolved Hide resolved

Suvrat1629 and others added 3 commits February 20, 2025 23:39

fixes format validation

e391cf0

Co-authored-by: Dave Marion <[email protected]>

json fix

fe616c4

removed a comment :')

5af65f8

formatting

7c28f1d

ctubbsii linked an issue Mar 25, 2025 that may be closed by this pull request

Add a format output flag to ./accumulo ec-admin to support easy parsing. #5205

Closed

ddanielr added the blocker This issue blocks any release version labeled on it. label Apr 17, 2025

dlmarion added 3 commits April 17, 2025 19:16

Merge branch '2.1' into formatting-flags

b33a5d4

Remove csv as a formatting option

a891398

Remove copy of running compaction map, seems unnecessary

4e68d86

dlmarion requested a review from ddanielr April 17, 2025 19:43

Added ECAdminIT, confirmed JSON output can be parsed

45e5092

Refactored the ECAdmin command and added a test. Downside to this implementation is that it will consume more memory in the case where json output is not requested.

ddanielr self-assigned this Apr 21, 2025

ddanielr approved these changes Apr 21, 2025

View reviewed changes

dlmarion merged commit 3912b36 into apache:2.1 Apr 21, 2025
8 checks passed

dlmarion mentioned this pull request Apr 21, 2025

Add a format output flag to ./accumulo ec-admin to support easy parsing. #5205

Closed

dlmarion changed the title ~~Add --format flag to ec-admin running for structured output (CSV/JSON)~~ Added --json flag to ec-admin running command Apr 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Added --json flag to ec-admin running command #5332

Added --json flag to ec-admin running command #5332

Uh oh!

Suvrat1629 commented Feb 13, 2025

Uh oh!

Suvrat1629 commented Feb 13, 2025

Uh oh!

kevinrr888 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlmarion commented Feb 20, 2025

Uh oh!

Suvrat1629 commented Feb 21, 2025

Uh oh!

dlmarion commented Feb 27, 2025

Uh oh!

Suvrat1629 commented Feb 28, 2025

Uh oh!

dlmarion commented Feb 28, 2025

Uh oh!

ctubbsii commented Apr 8, 2025

Uh oh!

ddanielr commented Apr 16, 2025

Uh oh!

ddanielr left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Added --json flag to ec-admin running command #5332

Added --json flag to ec-admin running command #5332

Uh oh!

Conversation

Suvrat1629 commented Feb 13, 2025

Uh oh!

Suvrat1629 commented Feb 13, 2025

Uh oh!

kevinrr888 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlmarion commented Feb 20, 2025

Uh oh!

Suvrat1629 commented Feb 21, 2025

Uh oh!

dlmarion commented Feb 27, 2025

Uh oh!

Suvrat1629 commented Feb 28, 2025

Uh oh!

dlmarion commented Feb 28, 2025

Uh oh!

ctubbsii commented Apr 8, 2025

Uh oh!

ddanielr commented Apr 16, 2025

Uh oh!

ddanielr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ddanielr left a comment •

edited

Loading