[request]: Serialize/Deserialize of models for Lambda events

[nmoutschen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue, please leave a comment

Tell us about your request

Add support for deserialization of structs for use with Lambda events. This is similar to #211, but for event sources that integrate with Lambda.

Alternatively, we could provide a official set of Lambda events, but this would be hard to do without an authoritative repository of event schemas for Lambda.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

I am currently writing a Lambda function in Rust for a GitHub sample that's invoked from a DynamoDB streams to showcase how to do parallel processing within a Lambda function.

Lambda's Event Source Mapping takes care of polling events from the DynamoDB stream and invoking the Lambda function with a batch. Lambda sends a JSON objects resembling aws_sdk_dynamodbstreams::output::GetRecordsOutput without NextShardIterator.

Are you currently working around this issue?

Right now, I've recreated most of it by making my own structs copying the Go Lambda SDK's DynamoDBEvent structure.

Additional context

N/A

Attachments

• Pull Request (in progress)

[Velfi]

Thanks for submitting this, I'll ask the rest of the team what they think about it.

[calavera]

You might want to take a look at this project: https://github.com/LegNeato/aws-lambda-events

[spf5000]

I wanted to piggy-back off this as I'm not sure the best place to put generic requests for serde serialize/deserialize support. I'm currently looking for ways to deserialize the items (aka HashMap<String, AttributeValue>) from DynamoDB after exporting the table to s3 and downloading the files locally. I don't currently see an easy way to deserialize these items from S3 into items/structs using the existing serde_json or serde_dynamo crates today.

Maybe there is something I'm missing here, but this would be extremely valuable to be able to deserialize the DynamoDB table exports for scripts in Rust while learning the language and experimenting with the SDK.

[liberwang1013]

I have same problems. My situation is that I want to parse cloudwatch event detail field into aws config ConfigurationItemChangeNotification, ConfigurationItem is really complicated struct, serde remote type is not a good way to do it

[Velfi]

The biggest issue is what this will do to crate sizes and compile times. I wrote about this in an FAQ entry.

Why don't the SDK service crates implement serde::Serialize or serde::Deserialize for any types?

1. Compilation time: serde makes heavy use of several crates (proc-macro2, quote, and syn) that are very expensive to compile. Several service crates are already quite large and adding a serde dependency would increase compile times beyond what we consider acceptable. When we last checked, adding serde derives made compilation 23% slower.

2. Misleading results: We can't use serde for serializing requests to AWS or deserializing responses from AWS because both sides of that process would require too much customization. Adding serialize/deserialize impls for operations has the potential to confuse users when they find it doesn't actually capture all the necessary information (like headers and trailers) sent in a request or received in a response.

In the future, we may add serde support behind a feature gate. However, we would only support this for operation Input and Output structs with the aim of making SDK-related tests easier to set up and run.

It's possibly that in a future version of Rust that supports "poll-based dependency resolution" we could add support for serde without affecting compile times for those that don't use it. We're not compiler experts and are definitely open to suggestions, but it's important that any proposed solution weighs the needs of all users.

related topics:
https://internals.rust-lang.org/t/towards-a-second-edition-of-the-compiler/5582
https://users.rust-lang.org/t/why-cant-rustc-only-compile-code-that-is-actually-used/64601

[dcormier]

It would be nice to have a crate feature to enable deserialization for those of us who need it. It's a fairly common way to do that in the Rust community. We are currently impacted by its absence, and are resorting to using types from the aws_lambda_events crate.

[MathieuDuponchelle]

I have a similar need for serializing transcripts received using the transcribestreaming API. If I'm not mistaken those events are received as JSON in the first place, so my ideal solution would be a hook for direct access to the received messages.

[landonxjames]

You should be able to get access directly to the not-yet-deserialized JSON with an Interceptor using the read_before_deserialization hook: https://docs.rs/aws-smithy-runtime-api/latest/aws_smithy_runtime_api/client/interceptors/trait.Intercept.html#method.read_before_deserialization

[MathieuDuponchelle]

You should be able to get access directly to the not-yet-deserialized JSON with an Interceptor using the read_before_deserialization hook: https://docs.rs/aws-smithy-runtime-api/latest/aws_smithy_runtime_api/client/interceptors/trait.Intercept.html#method.read_before_deserialization

Oh interesting thanks :)

[MathieuDuponchelle]

@landonxjames this was promising but unfortunately I can't get anything useful out of it, I filed #1212 with what I tried :)

[MathieuDuponchelle]

In my understanding the interceptor code can only be used to inspect the initial response, not follow up messages

[MathieuDuponchelle]

@landonxjames I ended up using serde_with to copy the relevant structures over to types I can implement Serialize for, this is of course not optimal, if I missed a way to actually intercept the events in the event stream please let me know.