Change list of match ids to set for json_handler.py (#395)

* Change list of match ids to set for json_handler.py Increase the speed of the json_handler by migrating from a list to a set. Move from O(n) to O(1) * Update CHANGELOG.md * Bump Version 0.65 -> 0.66 * Include optimisation for composite json matches --------- Co-authored-by: Matteo Figus <[email protected]>
awslabs · Jan 26, 2024 · 3efdceb · 3efdceb
1 parent 2d7fd40
commit 3efdceb
Show file tree

Hide file tree

Showing 3 changed files with 10 additions and 4 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,11 @@
 # Change Log
 
+## v0.66
+
+- [#395](https://github.com/awslabs/amazon-s3-find-and-forget/issues/395):
+  Increase the speed of the json_handler by migrating from a list to a set.
+  Move from O(n) to O(1)
+
 ## v0.65
 
 - [#393](https://github.com/awslabs/amazon-s3-find-and-forget/issues/393): Fix

diff --git a/backend/ecs_tasks/delete_files/json_handler.py b/backend/ecs_tasks/delete_files/json_handler.py
@@ -52,7 +52,7 @@ def delete_matches_from_json_file(input_file, to_delete, compressed=False):
             for column in to_delete:
                 if column["Type"] == "Simple":
                     record = get_value(column["Column"], parsed)
-                    if record and record in column["MatchIds"]:
+                    if record and record in set(column["MatchIds"]):
                         should_delete = True
                         break
                 else:
@@ -61,7 +61,7 @@ def delete_matches_from_json_file(input_file, to_delete, compressed=False):
                         record = get_value(col, parsed)
                         if record:
                             matched.append(record)
-                    if matched in column["MatchIds"]:
+                    if tuple(matched) in set(map(tuple, column["MatchIds"])):
                         should_delete = True
                         break
             if should_delete:

diff --git a/templates/template.yaml b/templates/template.yaml
@@ -1,6 +1,6 @@
 AWSTemplateFormatVersion: "2010-09-09"
 Transform: AWS::Serverless-2016-10-31
-Description: Amazon S3 Find and Forget (uksb-1q2j8beb0) (version:v0.65) (tag:main)
+Description: Amazon S3 Find and Forget (uksb-1q2j8beb0) (version:v0.66) (tag:main)
 
 Parameters:
   AccessControlAllowOriginOverride:
@@ -206,7 +206,7 @@ Conditions:
 Mappings:
   Solution:
     Constants:
-      Version: 'v0.65'
+      Version: 'v0.66'
 
 Resources:
   TempBucket: