added aws prometheus remote write exporter #1

amanbrar1999 · 2020-11-03T18:05:58Z

Description:
This PR introduces a new exporter that leverages the existing Prometheus Remote Write Exporter in the core collector repo, and adds AWS Sigv4 authentication logic to it.

The idea here is to add an interceptor to the http client that is used to initialize the Prometheus Remote Write Exporter struct. The interceptor handles the Sigv4 logic, and the rest of the logic is identical to the core exporter.

The way this is designed, any changes in the core exporter's config will need to be duplicated here, however any other changes should be fine

This PRs size is significantly bloated by the go module files

Link to tracking Issue:
No direct issue for this exists but it is related to vendor specific code discussion: open-telemetry/oteps#141

Testing:
Unit tests have been added for the factory and config. Testing the authentication logic has been done manually by setting up a pipeline to communicate with AWS services.

Documentation:
A thorough README.md is included, as well as all example configuration in testdata/config.yaml

-- cc @alolita

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.32.0 to 1.33.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.32.0...v1.33.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/golang/protobuf](https://github.com/golang/protobuf) from 1.4.2 to 1.4.3. - [Release notes](https://github.com/golang/protobuf/releases) - [Commits](golang/protobuf@v1.4.2...v1.4.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [go.opencensus.io](https://github.com/census-instrumentation/opencensus-go) from 0.22.4 to 0.22.5. - [Release notes](https://github.com/census-instrumentation/opencensus-go/releases) - [Commits](census-instrumentation/opencensus-go@v0.22.4...v0.22.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#1457) Bumps [go.opencensus.io](https://github.com/census-instrumentation/opencensus-go) from 0.22.4 to 0.22.5. - [Release notes](https://github.com/census-instrumentation/opencensus-go/releases) - [Commits](census-instrumentation/opencensus-go@v0.22.4...v0.22.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

) Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.11.1 to 1.11.2. - [Release notes](https://github.com/klauspost/compress/releases) - [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml) - [Commits](klauspost/compress@v1.11.1...v1.11.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.19.2 to 0.19.3. - [Release notes](https://github.com/kubernetes/client-go/releases) - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.19.2...v0.19.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

) Bumps [gopkg.in/DataDog/dd-trace-go.v1](https://github.com/DataDog/dd-trace-go) from 1.26.0 to 1.27.1. - [Release notes](https://github.com/DataDog/dd-trace-go/releases) - [Commits](DataDog/dd-trace-go@v1.26.0...v1.27.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.34.9 to 1.35.20. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.34.9...v1.35.20) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.32.0 to 0.34.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.32.0...v0.34.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.35.17 to 1.35.20. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.35.17...v1.35.20) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…or (#1472) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.35.2 to 1.35.20. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.35.2...v1.35.20) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

#1452) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.35.19 to 1.35.20. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.35.19...v1.35.20) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ter (#1477) Bumps [go.opencensus.io](https://github.com/census-instrumentation/opencensus-go) from 0.22.4 to 0.22.5. - [Release notes](https://github.com/census-instrumentation/opencensus-go/releases) - [Commits](census-instrumentation/opencensus-go@v0.22.4...v0.22.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/golang/protobuf](https://github.com/golang/protobuf) from 1.4.2 to 1.4.3. - [Release notes](https://github.com/golang/protobuf/releases) - [Commits](golang/protobuf@v1.4.2...v1.4.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.34.9 to 1.35.20. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.34.9...v1.35.20) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.32.0 to 1.33.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.32.0...v1.33.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Bogdan Drutu <[email protected]>

Bumps [github.com/hashicorp/go-hclog](https://github.com/hashicorp/go-hclog) from 0.14.0 to 0.14.1. - [Release notes](https://github.com/hashicorp/go-hclog/releases) - [Commits](hashicorp/go-hclog@v0.14.0...v0.14.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [go.opencensus.io](https://github.com/census-instrumentation/opencensus-go) from 0.22.4 to 0.22.5. - [Release notes](https://github.com/census-instrumentation/opencensus-go/releases) - [Commits](census-instrumentation/opencensus-go@v0.22.4...v0.22.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Removes the config option "namespace" and hardcodes "otel" instead, which is what the Datadog backend expects. Additionally, stops treating the 'running' metric as a special case that didn't use the configured namespace and makes it be processsed like a regular metric.

* Add metric filter option to metrics transform processor * Address review comments and add new_name regex substitution

Signed-off-by: Bogdan Drutu <[email protected]>

…rcleCI (#1486)

…1487) Signed-off-by: Bogdan Drutu <[email protected]>

* Move test as per CR comments * Fix getting cached value * Remove unneeded dependencies and config sections

Fix an error for the config file part.

alolita

lgtm. version number hardcoded - we should figure out how to make this dynamically pull the latest stable release

Signed-off-by: Bogdan Drutu <[email protected]>

* initial commit of ecs translation * reverted trace ID change * added unit tests * addressed comments and unit test issues * fixed breaking API changes

amanbrar1999 · 2020-11-05T02:56:10Z

PR has been made upstream, closing this

Adds a Cloud Foundry metric receiver which reads metrics from Cloud Foundry Reverse Log Proxy Gateway. More details available in the `README.md`. `make gotidy` seems to have made plenty of subtle changes to `go.sum` files, not sure if this is normal. This PR contains the overall structure, documentation, implementation for config and factory, but does NOT contain the implementation of the receiver and does not enable the component, as that will come in separate PRs later. **Link to tracking Issue:** open-telemetry#5320 **Testing:** Unit tests. Manual testing was performed against Tanzu Application Service (TAS) versions 2.7, 2.8 and 2.11. Considered adding an integration test with mocked HTTP servers acting as endpoints where the HTTP server would provide a constant response (prerecorded from the real TAS traffic), but not sure if mocks would make more sense. **Documentation:** `README.md` and `doc.go` for the new receiver module were added.

* add vcenter vSAN collection * checkpoint on getting property collection working * checkpoint before integration test * dual receivers under root receiver pointer * checkpoint before updated mdatagen * use syslog receiver rather than tcplogreceiver * getting more performance counter refinements * remove unneccessary component addition * try to fix go.mod resolution issues * try to fix go.mod resolution issues pt 2 * addlicense * fix go.mod by fixing require directive * add readme for metrics * update readme * fix go.mod referring nonexistent version * add performance manager tests * more tests * add more attributes to virtual machines and host systems * add more attributes to virtual machines and host systems * spike changelog entry * fix go.mod in both places * fix go.mod in configschema * add // import github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vmwarevcenterreceiver to imports * add quotations * add to receiver lifecycle * remove extra go generate direction * fix typo of utilizaiton in metric description * small changes to interval id in performance queries to be more consistent * PR feedback including omitting company name prefix * PR feedback to not fail starting the component on potential network failures * minor grammar correction in vcenter readme * update expected metrics * update host_effective attribute value * remove PerformanceInterval customizability * add to codeowners * fix indentation on merge conflict * fix changelog entry place so its in the new components section * update to be on 0.49.0 of the collector' * add PR number to changelog * regenerate with newer version of mdatagen * move error log if unable to connect on start to receiver.Start() rather than scraper.Start() * fix test cases from last commit * minor update to config with tests * fix metric description * use utc for host vsan collection as well * update comments of public facing methods * return errors on getting clusters to the scraper errors * PR feedback #1 * instantiate new client if client is nil * update all descriptions to have punctuation * three more descs * move ensureReceiver up to once we validated as a config * some more PR feedback * looking into race conditions * run go tidy * fix import order and remove unneccessary mutex * remove mutex from struct * refactor client to responsible for knowing if the vsan endpoints are reachable * fix integration test referencing old var * change metrics.metrics => metrics.settings, update client pr feedback * remove vSAN collection temporarily * remove extra metric attributes for vSAN * remove vsan specific variables * clean up host PerfCounter disk latency metrics and fix some descriptions to better reflect interval * add 20s interval to extended documentation as needed * mdatagen fixes * add integration test metric scrape * fix import order * go up to 0.49.1 * gotidy * add replace directive for semconv * gotidy fixes * fix component not being on 0.50.0 * update to v0.50.1-0.20220429151328-041f39835df7 * use newer mdatagen * remove any logging functionality change && update documentation * fix integration test from flattening of config * fix scraper start not erroring if connection cannot be established * make scrapertest less flaky * format test json * Apply suggestions from code review Co-authored-by: Daniel Jaglowski <[email protected]> * adjust metric definition for vcenter.host.disk.throughput * remove comment and move pm level 2 metrics to appropriate section * try to be respective of datacenters * fix only vCenter server functionality * try building out a mock server for test coverage * make goporto * fix build issues * use latest mdatagen * add newlines to ends of xml recordings * fix integration test * moved around scrapererrors because now the receiver is datacenter dependent * try and do an audit of performance metrics and requests/responses * update testdata with correct units * make tidy * make tidy * update collector version * fix local testing code including modules * remove deprecated use of commonponenterror * pr feedback; add method of collection recording, return poweredOn/poweredOff VMs * remove content.json * fix description change in scraper_test.go * update collector version * bump replaced module; rebuild load tests * fix alibaba version auto localizing Co-authored-by: Daniel Jaglowski <[email protected]>

dependabot bot and others added 25 commits November 2, 2020 15:07

Run Auto Assign workflow once draft PRs are ready for review (#1481)

9321514

Signed-off-by: Bogdan Drutu <[email protected]>

Add metric filter option to metrics transform processor (#1447)

e3e304a

* Add metric filter option to metrics transform processor * Address review comments and add new_name regex substitution

Remove unnecessary dependencies from tracegen (#1480)

c2f0ce4

Signed-off-by: Bogdan Drutu <[email protected]>

amanbrar1999 requested review from alolita and JasonXZLiu November 3, 2020 18:05

JasonXZLiu approved these changes Nov 3, 2020

View reviewed changes

tigrannajaryan and others added 2 commits November 3, 2020 14:04

Increase memory limits for TestLog10kDPS/OTLP to avoid failures on Ci…

4be0f7c

…rcleCI (#1486)

Upgrade otel collector core, fix remove of the Set Array/Map values (#…

60de58c

…1487) Signed-off-by: Bogdan Drutu <[email protected]>

albertvaka and others added 3 commits November 3, 2020 14:37

[datadog exporter] Move ProcessMetrics test to metrics package (#1488)

db507b3

* Move test as per CR comments * Fix getting cached value * Remove unneeded dependencies and config sections

Update aws xray receiver readme (#1485)

3b7afff

Fix an error for the config file part.

Add @kbrockhoff as an Approver (#1492)

59a1114

alolita approved these changes Nov 4, 2020

View reviewed changes

bogdandrutu and others added 5 commits November 4, 2020 16:01

Add @andrewhsu as triager to the collector (#1496)

95011c7

Signed-off-by: Bogdan Drutu <[email protected]>

Skip processing empty metric slice in Stackdriver exporter (#1494)

507ad41

Improved ECS attribute and origin translation in awsxrayexporter (#1428)

86af421

* initial commit of ecs translation * reverted trace ID change * added unit tests * addressed comments and unit test issues * fixed breaking API changes

added aws prometheus remote write exporter

5485adb

added copyrights

a124c20

amanbrar1999 force-pushed the add-aws-cortex-exporter branch from c6e7e6c to a124c20 Compare November 5, 2020 01:29

amanbrar1999 requested a review from anuraaga as a code owner November 5, 2020 01:29

updated documentation

66682ed

amanbrar1999 closed this Nov 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added aws prometheus remote write exporter #1

added aws prometheus remote write exporter #1

amanbrar1999 commented Nov 3, 2020 •

edited

Loading

alolita left a comment

amanbrar1999 commented Nov 5, 2020

added aws prometheus remote write exporter #1

added aws prometheus remote write exporter #1

Conversation

amanbrar1999 commented Nov 3, 2020 • edited Loading

alolita left a comment

Choose a reason for hiding this comment

amanbrar1999 commented Nov 5, 2020

amanbrar1999 commented Nov 3, 2020 •

edited

Loading