Add export ingest-pipelines command #2574

MichelLosier · 2025-05-06T14:19:50Z

Resolves: #1722

Adds ingest pipeline export from elasticsearch to a package with the command elastic-package export ingest-pipelines
User is prompted with a multi-selection of pipelines to export
- References to pipelines in a pipeline processor definitions will also be pulled automatically
User is prompted to select an existing data stream in the package, or the package root as destinations to export to.
- Dependent pipeline references are not prompted for, they'll be written to the same destination as selected for the parent.
Pipelines are written to yaml files unchanged

MichelLosier · 2025-05-06T14:20:45Z

cmd/export.go

 	cmd.PersistentFlags().StringP(cobraext.ProfileFlagName, "p", "", fmt.Sprintf(cobraext.ProfileFlagDescription, install.ProfileNameEnvVar))

 	return cobraext.NewCommand(cmd, cobraext.ContextPackage)
-}
-
-func exportDashboardsCmd(cmd *cobra.Command, args []string) error {


Moves export dashboards functionality into own file now that we are adding support for ingest-pipelines

MichelLosier · 2025-05-06T14:22:05Z

internal/dump/ingestpipelines.go

 )

-// IngestPipeline contains the information needed to export an ingest pipeline.
-type IngestPipeline struct {


A lot of this functionality was moved to the internal/elasticsearch package since it shares a lot of retrieval needs with the new export functionality

MichelLosier · 2025-05-09T15:05:44Z

cmd/export_dashboards.go

@@ -0,0 +1,122 @@
+// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one


This file was extracted from cmd/export.go

elasticmachine · 2025-05-09T18:59:22Z

💚 Build Succeeded

Buildkite Build
Commit: cb13e0d

History

💔 Build #5299 failed db80e29
💔 Build #5298 failed d5a089c
💔 Build #5293 failed 11a4494
💔 Build #5287 failed 1ff73e5
💔 Build #5286 failed 5ec05c4

MichelLosier · 2025-05-09T19:10:26Z

cmd/export_ingest_pipelines.go

+
+	ingestPipelineNames = slices.DeleteFunc(ingestPipelineNames, func(name string) bool {
+		// Filter out system pipelines that start with dot "." or global@
+		return strings.HasPrefix(name, ".") || strings.HasPrefix(name, "global@")


Any other pipeline types / patterns we should exclude here?

I think this is fine by now, we can modify this in the future if there is some feedback from package developers.

jsoriano

Thanks! This is looking quite good, and works well. Added some comments.

jsoriano · 2025-05-12T14:15:54Z

cmd/export_ingest_pipelines.go

+
+	ingestPipelineNames = slices.DeleteFunc(ingestPipelineNames, func(name string) bool {
+		// Filter out system pipelines that start with dot "." or global@
+		return strings.HasPrefix(name, ".") || strings.HasPrefix(name, "global@")


I think this is fine by now, we can modify this in the future if there is some feedback from package developers.

jsoriano · 2025-05-13T07:45:38Z

cmd/export_ingest_pipelines.go

+}
+
+func promptWriteLocations(pipelineNames []string, writeLocations []export.PipelineWriteLocation) (export.PipelineWriteAssignments, error) {
+


Nit. Remove empty line.

Suggested change

jsoriano · 2025-05-13T08:36:55Z

cmd/export_ingest_pipelines.go

+				Description: func(value string, index int) string {
+					idx := slices.IndexFunc(writeLocations, func(p export.PipelineWriteLocation) bool { return p.Name == value })


Won't be idx the same as index?

jsoriano · 2025-05-13T08:48:47Z

internal/dump/installedobjects_test.go

@@ -39,18 +43,20 @@ func TestDumpInstalledObjects(t *testing.T) {
 		&installedObjectsDumpSuite{
 			// To reproduce the scenario:
 			// - Start the stack with version 7.16.2.
-			// - Install apache package (1.3.4).
+			// - Install apache package (1.3.5).


Nit. Why did we need to regenerate the records? Doesn't it work the same for these cases?

jsoriano · 2025-05-13T09:01:12Z

internal/export/ingest_pipelines.go

+	yamlBytes, err := yaml.Marshal(jsonPipeline)
+	if err != nil {
+		return fmt.Errorf("marshalling ingest pipeline json to yaml failed (ID: %s): %w", pipeline.Name(), err)
+	}
+
+	// requirement: https://github.com/elastic/package-spec/pull/54
+	documentStartDashes := []byte("---\n")
+	documentBytes := append(documentStartDashes, yamlBytes...)


Nit. To avoid appending, you can use a bytes buffer. Something like this:

Suggested change

yamlBytes, err := yaml.Marshal(jsonPipeline)

if err != nil {

return fmt.Errorf("marshalling ingest pipeline json to yaml failed (ID: %s): %w", pipeline.Name(), err)

}

// requirement: https://github.com/elastic/package-spec/pull/54

documentStartDashes := []byte("---\n")

documentBytes := append(documentStartDashes, yamlBytes...)

var documentBytes bytes.Buffer

// requirement: https://github.com/elastic/package-spec/pull/54

documentBytes.WriteString("---\n")

err := yaml.NewEncoder(&documentBytes).Encode(jsonPipeline)

if err != nil {

return fmt.Errorf("marshalling ingest pipeline json to yaml failed (ID: %s): %w", pipeline.Name(), err)

}

jsoriano · 2025-05-13T09:10:24Z

internal/export/ingest_pipelines.go

+
+type PipelineWriteAssignments map[string]PipelineWriteLocation
+
+func IngestPipelines(ctx context.Context, api *elasticsearch.API, writeAssignments PipelineWriteAssignments) error {


Could we have a test case for this?

jsoriano · 2025-05-13T09:24:37Z

internal/export/ingest_pipelines.go

+		return fmt.Errorf("unmarshalling ingest pipeline failed (ID: %s): %w", pipeline.Name(), err)
+	}
+
+	yamlBytes, err := yaml.Marshal(jsonPipeline)


I wonder if we should do some formatting here. Some things:

Remove _meta, specially the managed* fields should not be needed here.

Remove version, I think this is not being used in packages.

Indentation. Looking to pipelines they use to be indented by two spaces, default marshalling seems to use four spaces.

Maybe we should make elastic-package format to automatically format these files too. But that is another story.

MichelLosier added 6 commits May 1, 2025 10:20

Move export dashboards cmd into own file

2b8dd4b

Add cmd interface for export ingest-pipelines and init es client

f1ffa7f

Implement ingest pipeline list selection

5851641

Have ingest pipelines export fetch pipelines by IDs

dd5c688

Refactor dump pipelines to share functionality with export

f65865c

Get nested pipelines

d33c88e

MichelLosier commented May 6, 2025

View reviewed changes

MichelLosier added 8 commits May 6, 2025 13:41

Filter out system pipelines for export selection

ec296da

Add GetProcessorPipelineNames method to RemotePipeline struct

6b8c718

Add package root as export option, and implement PipelineWriteLocation

a6e1c14

Write parent pipelines to yaml files

721eba5

Fix ingest_pipeline dir name

748c597

Write pipeline processor pipeline deps

5ec05c4

Remove logging lines

1ff73e5

Update installedobjects_test to handle bulk pipeline requests

11a4494

MichelLosier commented May 9, 2025

View reviewed changes

MichelLosier added 4 commits May 9, 2025 09:36

Apply formatting

d5a089c

Add triple-dash document start to pipeline yaml

5e8cbb0

Remove log line

db80e29

Update readme

cb13e0d

MichelLosier marked this pull request as ready for review May 9, 2025 19:09

MichelLosier commented May 9, 2025

View reviewed changes

jsoriano reviewed May 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add export ingest-pipelines command #2574

Add export ingest-pipelines command #2574

MichelLosier commented May 6, 2025 •

edited

Loading

MichelLosier May 6, 2025

MichelLosier May 6, 2025

MichelLosier May 9, 2025

elasticmachine commented May 9, 2025

MichelLosier May 9, 2025

jsoriano May 12, 2025

jsoriano left a comment

jsoriano May 12, 2025

jsoriano May 13, 2025

jsoriano May 13, 2025

jsoriano May 13, 2025

jsoriano May 13, 2025

jsoriano May 13, 2025

jsoriano May 13, 2025

		@@ -0,0 +1,122 @@
		// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one

		}

		func promptWriteLocations(pipelineNames []string, writeLocations []export.PipelineWriteLocation) (export.PipelineWriteAssignments, error) {

		Description: func(value string, index int) string {
		idx := slices.IndexFunc(writeLocations, func(p export.PipelineWriteLocation) bool { return p.Name == value })


		type PipelineWriteAssignments map[string]PipelineWriteLocation

		func IngestPipelines(ctx context.Context, api *elasticsearch.API, writeAssignments PipelineWriteAssignments) error {

Add export ingest-pipelines command #2574

Are you sure you want to change the base?

Add export ingest-pipelines command #2574

Conversation

MichelLosier commented May 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 9, 2025

💚 Build Succeeded

History

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichelLosier commented May 6, 2025 •

edited

Loading