Add find and foreach sub commands for bulk operations #3027

vinit-chauhan · 2025-10-27T15:26:38Z

Elastic-package has been extensively used for integrations related tasks. However, most of the commands now are targeted to be run on one package at a time. We don’t have an option if we want to repeat a certain operation across multiple integrations.

This pull request adds two subcommands to the elastic-package to allow bulk operations from the elastic-package.

find
foreach

Note: Both commands are expected to be run from the integration repository.

Find subcommand

Find command adds the ability to filter and return a list of integrations based on specified criteria.

Available Filters:

Category (--categories)
Codeowner (--code-owners)
Input (--inputs)
Package Dir Name (--package-dirs) (supports glob patterns)
Package Name (--packages) (supports glob patterns)
Package Type (--package-types) ( eg., integration, input )
Spec Version (--spec-version)

You can chain multiple filters and each filter can have multiple comma-separated values.
Matching:
All filters must match.
At least one of the values match.

Currently, the find command runs sequentially and reads all the manifest files at once and keeps it in memory. We can update the code to use a buffered channel and have a producer and consumer to reduce memory footprint.

The current elastic-package and spec does not enforce the same package_name and directory name in the repo. Which leads to some integrations having different package_name and directory name.

The find command by default returns the absolute path of the package in the integration repo. However, it also provides a flag --output-info to make it return the package name or directory name.

Moreover, it also provides a flag --output to select the output format json, yaml or new line separated paths (default).

If no filter flag is provided, the command will return a list of all the integrations.

elastic-package find --inputs tcp,udp --code-owners elastic/integration-experience --packages cisco_*

Foreach subcommand

The foreach command leverages the filter registry. Therefore all the flags available in the filter are directly available to foreach commands without any code changes.

~~Additionally, foreach has 1 flag --parallel which allows the user to run commands parallelly using worker pool.~~ (Will be added in a follow-up PR)

default is 1 ( runs sequentially )

The elastic-package command you want to run goes after -- with all of its flags.

Note: You are only allowed to run allowed elastic-package subcommands ["build","check","changelog","clean","format","install","lint","test","uninstall"] (cmd/foreach.go)

elastic-package foreach --inputs tcp,udp --code-owners elastic/integration-experience -- test system -g

File changes:

internal/packages/packages.go: Added function to find the integrations repo root dir and read all manifests.
cmd/find.go: Find command implementation
cmd/foreach.go: Foreach command implementation
internal/filter/*: Filter interface and implementation for each filter flag.

Related Issues

AI Tools used

Cursor With Claude-4.5-Sonnet
Antigravity With Gemini-3-Pro

…lter

…n manifest and dir_name in integrations repo.

jrmolin · 2025-11-10T16:35:36Z

are you going to pull this out of draft?

vinit-chauhan · 2025-11-12T19:19:56Z

Removed code for parallel execution from the PR for now.
Once the required changes in elastic-package merged will raise a new PR.

mjwolf · 2025-11-12T20:07:33Z

It could be nice to add some additional output formats to the filter command. For example, one result per line output would make working with shell scripts easier. But, it's also easy to convert the json output to do this with jq, so adding multiple formats might not be critical to add.

mjwolf · 2025-11-12T20:15:12Z

internal/packages/packages.go

+		// Validate it's a package manifest
+		ok, err := isPackageManifest(path)
+		if err != nil {
+			// Log the error but continue searching


This doesn't actually log the error here. You should either remove the comment or add logging

jrmolin

there's a startling lack of testing, but it looks reasonable otherwise

jsoriano · 2025-11-12T20:22:12Z

internal/cobraext/flags.go

+	FilterOutputFlagShorthand   = "o"
+
+	FilterPackageDirNameFlagName        = "package-dirs"
+	FilterPackageDirNameFlagDescription = "package directories to filter by (comma-separated values)"


How is this flag expected to work? If I use --package-dirs ./test/packages/parallel in the elastic-package repository it doesn't find any package.

$ elastic-package filter --package-dirs ./test/packages/parallel 2025/11/12 21:20:31 INFO Found 0 matching package(s) null

package-dirs would filter packages based on the name of the package's directory. This is used to search without needing to know the package name in the manifest file.

example:

go run main.go -C ../integrations filter --package-dirs sql_input -o pkgname 2025/11/13 09:29:56 INFO Found 1 matching package(s) ["sql"]

if you want to filter all packages in one directory you can use the following command.

❯ go run main.go -C test/packages/parallel filter 2025/11/13 09:11:51 INFO Found 19 matching package(s) ["apache","apache_basic_license","auditd_manager","auth0_logsdb","aws","awsfirehose","custom_entrypoint","httpcheck","mongodb","nginx","nginx_multiple_services","oracle","otel_http_server","sql_input","system","terraform_local","ti_anomali","ti_anomali_logsdb","ti_anomali_template"]

Oh ok. So it is not possible to search in multiple directories. I guess this is fine.

does it make sense to add the example to the description so it goes to the docs?

I was trying again the command today and I struggled again to properly find packages per directory.

I find also a bit confusing to have to use -C for this, whose purpose is to change directory to do the operation, and in the case of this command limits the option to look for packages in multiple directories.

What is the use case for filtering packages per dir name with --package-dirs?

I think it would be nice to have an explicit flag to indicate where to look for packages, though at this point it would likely add more confusion.

By Default, it looks for package in current dir + 2 depth.

So you can either increase --depth/-d you can look through more directories or run the command in the directory you want to search in.

CWD: Users/vinit.chauhan/github.com/elastic/elastic-package ❯ elastic-package filter -d 4 --output-info absolute_path 2025/11/26 10:44:41 INFO Found 65 matching package(s) /Users/vinit.chauhan/github.com/elastic/elastic-package/internal/fields/testdata /Users/vinit.chauhan/github.com/elastic/elastic-package/internal/files/testdata/links /Users/vinit.chauhan/github.com/elastic/elastic-package/internal/files/testdata/testpackage /Users/vinit.chauhan/github.com/elastic/elastic-package/new_package /Users/vinit.chauhan/github.com/elastic/elastic-package/test/new_package2 /Users/vinit.chauhan/github.com/elastic/elastic-package/test/packages/benchmarks/pipeline_benchmark /Users/vinit.chauhan/github.com/elastic/elastic-package/test/packages/benchmarks/rally_benchmark

❯ cd test/packages/parallel ❯ elastic-package filter -d 4 --output-info absolute_path -v 2025/11/26 10:58:43 INFO Found 19 matching package(s) /Users/vinit.chauhan/github.com/elastic/elastic-package/test/packages/parallel/apache /Users/vinit.chauhan/github.com/elastic/elastic-package/test/packages/parallel/apache_basic_license /Users/vinit.chauhan/github.com/elastic/elastic-package/test/packages/parallel/auditd_manager /Users/vinit.chauhan/github.com/elastic/elastic-package/test/packages/parallel/auth0_logsdb

The only reason to have a flag for filtering package_directories is because package spec allows different package_name and directory name of the package.

User might want to search all Cisco packages, they can do it using following command.

CWD: .../integrations/ ❯ elastic-package filter --package-dirs 'cisco*' 2025/11/26 11:08:44 INFO Found 12 matching package(s) cisco_aironet cisco_asa cisco_duo cisco_ftd cisco_ios cisco_ise cisco_meraki cisco_meraki_metrics cisco_nexus cisco_secure_email_gateway cisco_secure_endpoint cisco_umbrella

jsoriano · 2025-11-12T20:35:14Z

cmd/foreach.go

+	errors := multierror.Error{}
+
+	for _, pkg := range filtered {
+		rootCmd := cmd.Root()


We could define another root command with the specific commands that are supported and use it here, then we could have different sets of subcommands, and maybe we could even plug it directly as subcommands of foreach.

This could be left for a future refactor too.

Something like this:

import ( "github.com/spf13/cobra" "github.com/elastic/elastic-package/internal/cobraext" ) var forEachCommands = []*cobraext.Command{ setupBuildCommand(), setupCheckCommand(), setupCleanCommand(), setupFormatCommand(), setupInstallCommand(), setupLintCommand(), setupTestCommand(), setupUninstallCommand(), } // ForEachRootCmd creates and returns root cmd for elastic-package func ForEachRootCmd() *cobra.Command { forEachCmd := &cobra.Command{ SilenceUsage: true, PersistentPreRunE: func(cmd *cobra.Command, args []string) error { return cobraext.ComposeCommandActions(cmd, args, processPersistentFlags, checkVersionUpdate, ) }, } forEachCmd.PersistentFlags().CountP(cobraext.VerboseFlagName, cobraext.VerboseFlagShorthand, cobraext.VerboseFlagDescription) forEachCmd.PersistentFlags().StringP(cobraext.ChangeDirectoryFlagName, cobraext.ChangeDirectoryFlagShorthand, "", cobraext.ChangeDirectoryFlagDescription) for _, cmd := range forEachCommands { forEachCmd.AddCommand(cmd.Command) } return forEachCmd }

That could be used here like this:

Suggested change

rootCmd := cmd.Root()

rootCmd := ForEachRootCmd()

rootCmd.SetContext(cmd.Context())

This is something that we need to do to add parallel execution. That's why I left it out of this PR, and would like to revisit it while I work on the other PR.

With that said, we do have a list of allowed sub-commands. Here

Let me know if I missed any command. - I'll also add changelog command in allow list.

Yes, we can leave this for a future refactor.

I'll also add changelog command in allow list.

Yes please.

internal/cobraext/flags.go

jsoriano · 2025-11-13T18:20:26Z

It could be nice to add some additional output formats to the filter command. For example, one result per line output would make working with shell scripts easier.

I would also prefer to have this format by default, and make json an optional alternative format.

- add build to exclude list - default output to nsv. also allow json and yaml

vinit-chauhan · 2025-11-13T20:47:03Z

Updated the code to support multiple output formats.

- Newline separated list ( default )
- JSON
- YAML

…ent-filter-registry

teresaromero · 2025-11-19T10:31:01Z

@vinit-chauhan could we add some unit testing for the filters logic? or perhaps a test for the new cmd in order to validate the output of the command with some test packages?

jsoriano

I would say to merge this and continue in follow ups, but some of the questions are related to the overall UX, that if we change later we could broke things.

So maybe we could just add some warnings, merge it, and polish this later if needed. WDYT?

jsoriano · 2025-11-25T18:23:01Z

internal/cobraext/flags.go

+	FilterOutputFlagShorthand   = "o"
+
+	FilterPackageDirNameFlagName        = "package-dirs"
+	FilterPackageDirNameFlagDescription = "package directories to filter by (comma-separated values)"


I was trying again the command today and I struggled again to properly find packages per directory.

I find also a bit confusing to have to use -C for this, whose purpose is to change directory to do the operation, and in the case of this command limits the option to look for packages in multiple directories.

What is the use case for filtering packages per dir name with --package-dirs?

I think it would be nice to have an explicit flag to indicate where to look for packages, though at this point it would likely add more confusion.

jsoriano · 2025-11-25T18:32:27Z

internal/cobraext/flags.go

+
+	FilterOutputInfoFlagName        = "output-info"
+	FilterOutputInfoFlagDescription = "output information about the packages. Available options: pkgname, dirname, absolute"
+	FilterOutputInfoFlagDefault     = "dirname"


I wonder what default is more useful, maybe the absolute path is better as it can be directly used without making any assumption?

dirname can be strange because there could be multiple of them with the same value if they are under different subdirectories.

jsoriano · 2025-11-25T18:36:09Z

README.md

+
+Use --change-directory to change the working directory and --depth to change the depth of the search.
+
+### `elastic-package foreach [flags] -- <SUBCOMMAND>`


We should list somewhere what commands are available for foreach.

jsoriano · 2025-11-25T18:36:38Z

README.md


 Use this command to download selected ingest pipelines and its referenced processor pipelines from Elasticsearch. Select data stream or the package root directories to download the pipelines. Pipelines are downloaded as is and will need adjustment to meet your package needs.

+### `elastic-package filter [flags]`


This command is looking for packages more than filtering them. Should it be called elastic-package find?

Hmm, Now that I think of this, naming it find makes more sense.

jsoriano · 2025-11-25T18:43:41Z

internal/cobraext/flags.go

+	FilterOutputFlagShorthand   = "o"
+
+	FilterOutputInfoFlagName        = "output-info"
+	FilterOutputInfoFlagDescription = "output information about the packages. Available options: pkgname, dirname, absolute"


Should we think on supporting multiple values at some point? We can add this later, but we should decide now if the output format should be different. Specially for JSON and YAML, we could list objects instead of plain values in preparation for including more data in the future.

[{"dirname":"apache"},{"dirname":"apache_tomcat"}]

Yes, we could definitely include that in future if we find any use case for that.

jsoriano · 2025-11-25T18:45:21Z

internal/filter/type.go

+}
+
+func (o *OutputOptions) validate() error {
+	validInfo := []string{"pkgname", "dirname", "absolute"}


Nit. As this is exposed as user options I would use name or package_name for the package name, and maybe absolute_path instead of absolute.

Suggested change

validInfo := []string{"pkgname", "dirname", "absolute"}

validInfo := []string{"package_name", "dir_name", "absolute_path"}

jsoriano · 2025-11-25T18:52:02Z

cmd/filter.go

+	return cobraext.NewCommand(cmd, cobraext.ContextPackage)
+}
+
+func filterCommandAction(cmd *cobra.Command, args []string) error {


We could log that these new commands are in technical preview, so we can more freely make breaking changes in follow ups.

I've added [Technical Preview] in both short and long descriptions.

Do you have any preference on where we should put the note?

vinit-chauhan · 2025-11-26T16:18:01Z

I would say to merge this and continue in follow ups, but some of the questions are related to the overall UX, that if we change later we could broke things.

So maybe we could just add some warnings, merge it, and polish this later if needed. WDYT?

Yeah, That makes sense. I'll update the code with all the review comments you made. Would you mind creating an issue where we can track the changes we need for the future?

elasticmachine · 2025-11-26T17:43:37Z

💚 Build Succeeded

Buildkite Build
Commit: 723576b

History

💚 Build #6663 succeeded e570f5e
💚 Build #6617 succeeded 89ef933
💔 Build #6613 failed d93ee4f
💚 Build #6604 succeeded 12a7bea
💔 Build #6603 failed 6196985
💚 Build #6585 succeeded 689eb72

cc @vinit-chauhan

vinit-chauhan added 30 commits October 9, 2025 14:23

initial structure of command

85d659a

fix: error skipped in edit dashboard command

17e4b39

fix: error skipped in edit dashboard command

01a681c

code refactors

010f1a4

Merge branch 'elastic:main' into cmd-filter

358f9d6

update response format.

c3fef19

move filter code to internal

3d84151

ignore .vscode dir

9a1c2aa

no args will return list of all packages

1336ff2

Merge branch 'main' of github.com:elastic/elastic-package into cmd-fi…

b588164

…lter

initial implementation for foreach.

53bd4f8

run multiple commands

2d827cc

add option to run commands in parallel

bc7a167

stash: filter-registry

ada16f7

working with subcommand flags

9123dae

Merge branch 'cmd-filter' into implement-filter-registry

423f035

created structure with working input filter

2e3721c

minor refactor; add code owner flag

8180ff4

minor refactor

f2abc11

add category flag

f72ea26

refactor code

5c4d63e

added spec version flag

d94e9b2

fix issue with unused filters applied

3cc0824

Merge branch 'main' into implement-filter-registry

78cce2b

Merge branch 'elastic:main' into implement-filter-registry

2656a22

run command in parallel

c36efd7

remove --exec to leverage -- to pass sub commands without string

0649c73

add subcommand allowlist

5e3b8b0

create map of dir_name and manifest to allow different package_name i…

df2c85b

…n manifest and dir_name in integrations repo.

add flag suffix to go files

b495ced

jsoriano mentioned this pull request Nov 11, 2025

Stop using Chdir #3066

Draft

remove parallel execution code from PR.

689eb72

vinit-chauhan force-pushed the implement-filter-registry branch from 61bba87 to 689eb72 Compare November 12, 2025 17:49

vinit-chauhan marked this pull request as ready for review November 12, 2025 19:17

vinit-chauhan requested review from jsoriano and teresaromero November 12, 2025 19:55

mjwolf reviewed Nov 12, 2025

View reviewed changes

jrmolin reviewed Nov 12, 2025

View reviewed changes

jsoriano reviewed Nov 12, 2025

View reviewed changes

vinit-chauhan added 2 commits November 13, 2025 15:37

- add changelog to allowlist

6196985

- add build to exclude list - default output to nsv. also allow json and yaml

fix: check

12a7bea

vinit-chauhan added 2 commits November 14, 2025 11:13

add example in filter command.

d93ee4f

Merge branch 'main' of github.com:elastic/elastic-package into implem…

89ef933

…ent-filter-registry

add tests for internal/filter

e570f5e

vinit-chauhan requested review from jrmolin, jsoriano and mjwolf November 25, 2025 15:23

jsoriano reviewed Nov 25, 2025

View reviewed changes

vinit-chauhan added 2 commits November 26, 2025 11:44

Address review comments

552911c

rename command to 'find'

723576b

vinit-chauhan changed the title ~~Add filter and foreach sub commands for bulk operations~~ Add find and foreach sub commands for bulk operations Nov 26, 2025

vinit-chauhan requested a review from jsoriano November 26, 2025 18:36

	rootCmd := cmd.Root()
	rootCmd := ForEachRootCmd()
	rootCmd.SetContext(cmd.Context())


		Use --change-directory to change the working directory and --depth to change the depth of the search.

		### `elastic-package foreach [flags] -- <SUBCOMMAND>`


		Use this command to download selected ingest pipelines and its referenced processor pipelines from Elasticsearch. Select data stream or the package root directories to download the pipelines. Pipelines are downloaded as is and will need adjustment to meet your package needs.

		### `elastic-package filter [flags]`

	validInfo := []string{"pkgname", "dirname", "absolute"}
	validInfo := []string{"package_name", "dir_name", "absolute_path"}

Add find and foreach sub commands for bulk operations #3027

Are you sure you want to change the base?

Add find and foreach sub commands for bulk operations #3027

Uh oh!

Conversation

vinit-chauhan commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Find subcommand

Foreach subcommand

File changes:

Related Issues

AI Tools used

Uh oh!

jrmolin commented Nov 10, 2025

Uh oh!

vinit-chauhan commented Nov 12, 2025

Uh oh!

mjwolf commented Nov 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrmolin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinit-chauhan Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsoriano commented Nov 13, 2025

Uh oh!

vinit-chauhan commented Nov 13, 2025

Uh oh!

teresaromero commented Nov 19, 2025

Uh oh!

jsoriano left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinit-chauhan Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinit-chauhan commented Nov 26, 2025

Uh oh!

elasticmachine commented Nov 26, 2025

vinit-chauhan commented Oct 27, 2025 •

edited

Loading

vinit-chauhan Nov 13, 2025 •

edited

Loading

jsoriano left a comment •

edited

Loading

vinit-chauhan Nov 26, 2025 •

edited

Loading