Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
1fba1f9
improvement: Refactor CloudTrail log processing, support batching(com…
haihongren Nov 16, 2021
2ab02b2
improvement: externalized BATCH_SIZE_FACTOR, added S3_CLOUDTRAIL_LOG_…
haihongren Nov 18, 2021
58cab14
added PermissionsBoundary and FunctionRole optional parameters
haihongren Jul 18, 2022
5301a04
updated _get_batch_size_factor to fallback to original value set in v…
haihongren Jul 27, 2022
be09657
Increment version
nedl86 Nov 5, 2021
b66eb93
Merge pull request #39 from newrelic/bump-1.1.1-to-1.1.2
danybmx Aug 22, 2022
b8722d8
Bump version to 1.1.3
danybmx Aug 22, 2022
23a0f1b
Merge branch 'master' into role_permission_boundary
danybmx Aug 22, 2022
eee8ddb
fix: Moving the requirements back to the src folder
danybmx Aug 24, 2022
cfd81ec
fix: update the version to 1.1.2 in the template as well
danybmx Aug 24, 2022
c5a2106
Merge pull request #40 from newrelic/fix-for-sam-packaging
danybmx Aug 24, 2022
b9c9401
docs: Adding developer docs and makefile
danybmx Aug 25, 2022
843681d
Merge pull request #41 from newrelic/adding-developers-guidelines
danybmx Aug 25, 2022
cea060d
Merge branch 'master' into role_permission_boundary
danybmx Aug 25, 2022
8378914
Merge pull request #37 from haihongren/role_permission_boundary
danybmx Aug 25, 2022
73db604
Update README.md (#22)
melissaklein24 Aug 25, 2022
380f52a
Updated vulnerable packages (#30)
William-Hill Aug 25, 2022
2265011
Upgrade dependencies versions because vulnerability issues (#42)
danybmx Aug 25, 2022
43f616a
Merge branch 'master' into Zayeem/master
danybmx Aug 25, 2022
71c2c8e
Bump version to 1.1.5
danybmx Aug 25, 2022
a6c9cdf
Update template to include CloudTrails config and AdditionalAttributes
danybmx Aug 26, 2022
0e40e7f
Merge pull request #43 from newrelic/add_additional_attributes
danybmx Aug 26, 2022
51fa316
Upgrade vulnerable dependencies (#53)
luckslovez Apr 13, 2023
f18a243
feat: bump Python runtime version to v3.11
jcsobrino Nov 16, 2023
53476af
Merge pull request #59 from jcsobrino/bump-python-runtime-3.11
jcsobrino Dec 15, 2023
d54671e
Issue with CloudTrail digest fixed
nr-sivakumar Jan 22, 2024
f7efc58
Fix for library vulnerabilities
nr-dsharma Feb 1, 2024
67f1869
Merge pull request #69 from nr-dsharma/dsharma_vulnerabilites_fix
jcsobrino Feb 2, 2024
d90c26b
Merge pull request #67 from Sivakumar3695/cloud_trail_digest_issue_fix
jcsobrino Feb 2, 2024
adce9da
fix: Fix for serverless python requirements file path
SivaKumarP127 Apr 23, 2024
3cbcfe9
Merge pull request #70 from SivaKumarP127/serverless_requirements_pat…
jcsobrino Apr 29, 2024
c29711e
version bump for aiohttp, idna & request library
nr-dsharma Jun 12, 2024
67fd3e7
version bump for aiohttp
nr-dsharma Jun 18, 2024
6e35e90
Merge pull request #75 from nr-dsharma/NR-279406_vulnerabilties_fix
danybmx Jun 18, 2024
2d8efbf
fixing vulnerabilities
hrai-nr Jul 15, 2025
d552c8d
fixing build issue
hrai-nr Jul 29, 2025
1a845b0
fixing vulnerabilities
hrai-nr Jul 29, 2025
2cf0315
fixing CVEs
hrai-nr Nov 26, 2025
a9d2a1e
Merge pull request #91 from newrelic/NR_479632
hrai-nr Dec 9, 2025
2c7ad39
feat: Upgrade Lambda runtime to Python 3.12
voorepreethi Dec 22, 2025
b49f595
Merge pull request #93 from newrelic/upgrade-python3.12
rohit-bandlamudi-nr Jan 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,6 @@ packaged.yml
# SAM
.aws-sam/
#VS Code
.vscode/
.vscode/
#JetBrains
.idea
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
## Version 1.3.0 (2025-12-22)
- Upgraded runtime from Python 3.11 to Python 3.12
- Updated to use Amazon Linux 2023 base (via Python 3.12 runtime)
- Python 3.11 deprecation date: June 30, 2026
- Python 3.12 support extends until October 31, 2028
- No code changes required - all dependencies compatible with Python 3.12

## Previous Versions
- updated to last release 1.1.1
- updated semantic version
- updated urls and lables
Expand Down
66 changes: 66 additions & 0 deletions DEVELOPER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Developer

This project uses [SAM](https://aws.amazon.com/serverless/sam/) tool for build,
run locally, package and publish.

There is a serverless file to play locally as well, but here we will focus on
the [SAM](https://aws.amazon.com/serverless/sam/) way of doing it.

## Requirements

- Make

### Local run / build

- Docker
- AWS Account
- AWS Profile configured on your computer, remember that AWS uses the `default`
profile unless you specify it. You can specify usign `AWS_PROFILE` enviroment
variable.
- [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-reference.html#serverless-sam-cli)

### Package

- An AWS S3 bucket where the package will be uploaded

### Deploy

- AWS Account and enough permissions to do the deploy, generated template will
need IAM capabilities.

### Publishing

- For personal publishing is mandatory to update the function name on the
`template.yml` file to avoid collide with the New Relic official release of
this application.

## Building it locally

This will generate an image of the lambda application using a docker container
that you will be able to run locally.

Just run `make build`.

## Running locally

You should have built the image locally as mentioned in the previous step.

Then you need to have a "sample" event of a file in an S3 bucket so we can use
it, we provide a sample one in the test/mock.json but it wouldn't work if you
haven't access to the given S3 bucket.

Then just run `LICENSE_KEY=<YOUR_NEW_RELIC_LICENSE_KEY> TEST_FILE="./test/mock.json" make run` to run it locally.

## Packaging

Run `BUCKET=<S3_BUCKET_NAME> REGION=<S3_BUCKET_AWS_REGION> make package`

## Deploying

Run `REGION=<THE_REGION_YOU_WANT> STACK_NAME=<THE_STACK_NAME_YOU_WANT> make deploy`

## Publishing

Run `REGION=<SAR_AWS_REGION> make publish` to publish your package. Remember to
update the function name before publishing it to do not collide with the New
Relic official application.
43 changes: 43 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
build:
sam build --use-container

run: check-run-env
sam local invoke "NewRelicLogIngestionFunction" -e $(TEST_FILE)

package: check-package-env
sam package --output-template-file packaged.yml --s3-bucket $(BUCKET) --region $(REGION)

deploy: check-deploy-env
sam deploy --template-file packaged.yml --stack-name $(STACK_NAME) --region $(REGION)

publish: check-publish-env
sam publish --template packaged.yml --region $(REGION)

check-run-env:
ifndef LICENSE_KEY
$(error LICENSE_KEY is undefined)
endif
ifndef TEST_FILE
$(error TEST_FILE is undefined)
endif

check-package-env:
ifndef REGION
$(error REGION is undefined)
endif
ifndef BUCKET
$(error BUCKET is undefined)
endif

check-deploy-env:
ifndef REGION
$(error REGION is undefined)
endif
ifndef STACK_NAME
$(error STACK_NAME is undefined)
endif

check-publish-env:
ifndef REGION
$(error REGION is undefined)
endif
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[![Community Project header](https://github.com/newrelic/open-source-office/raw/master/examples/categories/images/Community_Project.png)](https://github.com/newrelic/open-source-office/blob/master/examples/categories/index.md#community-project)
[![Community Plus header](https://github.com/newrelic/opensource-website/raw/master/src/images/categories/Community_Plus.png)](https://opensource.newrelic.com/oss-category/#community-plus)

# AWS Lambda for sending logs from S3 to New Relic

Expand All @@ -22,5 +22,11 @@ Contributions to improve s3-log-ingestion-lambda are encouraged! Keep in mind wh

To execute our corporate CLA, which is required if your contribution is on behalf of a company, or if you have any questions, please drop us an email at [email protected].

## Developers

For more information about how to contribute from the developer point of view,
we recommend you to take a look to the [DEVELOPER.md](./DEVELOPER.md) that
contains most of the info you'll need.

## License
`s3-log-ingestion-lambda` is licensed under the [Apache 2.0](http://apache.org/licenses/LICENSE-2.0.txt) License. The s3-log-ingestion-lambda also uses source code from third party libraries. Full details on which libraries are used and the terms under which they are licensed can be found in the third party notices docume
`s3-log-ingestion-lambda` is licensed under the [Apache 2.0](http://apache.org/licenses/LICENSE-2.0.txt) License. The s3-log-ingestion-lambda also uses source code from third party libraries. Full details on which libraries are used and the terms under which they are licensed can be found in the third party notices document
21 changes: 0 additions & 21 deletions requirements.txt

This file was deleted.

15 changes: 10 additions & 5 deletions serverless.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,32 +22,37 @@ service: ${env:SERVICE_NAME}

provider:
name: aws
runtime: python3.8
runtime: python3.12
iamRoleStatements:
- Effect: "Allow"
Action:
- "s3:GetObject"
- "s3:GetObject"
Resource: "arn:aws:s3:::${env:S3_BUCKET_NAME}/*"

plugins:
- serverless-python-requirements

custom:
pythonRequirements:
fileName: ./src/requirements.txt
dockerizePip: non-linux

functions:
NewRelic-s3-log-ingestion:
handler: src/handler.lambda_handler
environment:
environment:
LICENSE_KEY: ${env:LICENSE_KEY}
LOG_TYPE: ${env:LOG_TYPE}
DEBUG_ENABLED: ${env:DEBUG_ENABLED}
S3_CLOUD_TRAIL_LOG_PATTERN: ${env:S3_CLOUD_TRAIL_LOG_PATTERN}
S3_IGNORE_PATTERN: ${env:S3_IGNORE_PATTERN}
BATCH_SIZE_FACTOR: ${env:BATCH_SIZE_FACTOR}
ADDITIONAL_ATTRIBUTES: ${env:ADDITIONAL_ATTRIBUTES}

events:
- s3:
bucket: ${env:S3_BUCKET_NAME}
event: s3:ObjectCreated:*
rules:
- prefix: ${env:S3_PREFIX, ""}
existing: true
existing: true
Loading