Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SG-38119 Release v3.8.0. Make payload optimization default #366

Merged
merged 3 commits into from
Feb 12, 2025
Merged

SG-38119 Release v3.8.0. Make payload optimization default #366

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e9e998d
Make payload optimization default
carlos-villavicencio-adsk Feb 7, 2025
99a57bb
Packaging for v3.8.0
carlos-villavicencio-adsk Feb 7, 2025
6b31a01
Update HISTORY.rst
carlos-villavicencio-adsk Feb 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion HISTORY.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,17 @@ Flow Production Tracking Python API Changelog

Here you can see the full list of changes between each Python API release.

v3.8.0 (2024 Feb 7)
===================

- Extend the payload optimizations to the ``in`` and ``not_in`` filters and
the ``update`` method.
- The payload optimization is now enabled by default.
It can be disabled with the ``SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION``
environment variable.

v3.7.0 (2024 Dec 9)
===========================
===================
- Remove unnecessary data in the payload when combining related queries before sending it to the server.
This would improve overall performance decreasing network latency and server processing.
See documentation for more information.
Expand Down
6 changes: 3 additions & 3 deletions docs/reference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -950,13 +950,13 @@ Stores the number of milliseconds to wait between request retries. By default,
In the case that both this environment variable and the config's ``rpc_attempt_interval`` property are set, the value in ``rpc_attempt_interal`` will be used.


SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION
SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION
=======================================

.. note:: (v3.7.0) This is an experimental feature. Feel free to disable this feature if you are experiencing any issues.

When set to ``1``, this environment variable will enable the entity optimization feature.
This feature is disabled by default and is used to reduce the payload size made to the server when retrieving entities
When set to ``1``, this environment variable will disable the entity optimization feature.
This feature is enabled by default and is used to reduce the payload size made to the server when retrieving entities
improving overall performance by decreasing network latency and server processing.

For example, a ``find`` call like this:
Expand Down
2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

setup(
name='shotgun_api3',
version='3.7.0',
version='3.8.0',
description='Flow Production Tracking Python API',
long_description=readme,
author='Autodesk',
Expand Down
22 changes: 11 additions & 11 deletions shotgun_api3/shotgun.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ def _is_mimetypes_broken():

SG_TIMEZONE = SgTimezone()

SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION = False
SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION = False

NO_SSL_VALIDATION = False
"""
Expand All @@ -118,7 +118,7 @@ def _is_mimetypes_broken():

# ----------------------------------------------------------------------------
# Version
__version__ = "3.7.0"
__version__ = "3.8.0"

# ----------------------------------------------------------------------------
# Errors
Expand Down Expand Up @@ -652,9 +652,9 @@ def __init__(self,
raise ValueError("Value of SHOTGUN_API_RETRY_INTERVAL must be positive, "
"got '%s'." % self.config.rpc_attempt_interval)

global SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION
if os.environ.get("SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION", "0").strip().lower() == "1":
SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION = True
global SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION
if os.environ.get("SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION", "0").strip().lower() == "1":
SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION = True

self._connection = None

Expand Down Expand Up @@ -1136,12 +1136,12 @@ def _add_project_param(self, params, project_entity):
def _translate_update_params(
self, entity_type, entity_id, data, multi_entity_update_modes
):
global SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION
global SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION

def optimize_field(field_dict):
if SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION:
return {k: _get_type_and_id_from_value(v) for k, v in field_dict.items()}
return field_dict
if SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION:
return field_dict
return {k: _get_type_and_id_from_value(v) for k, v in field_dict.items()}

full_fields = self._dict_to_list(
data,
Expand Down Expand Up @@ -4493,9 +4493,9 @@ def _translate_filters_simple(sg_filter):

# Payload optimization: Do not send a full object
# just send the `type` and `id` when combining related queries
global SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION
global SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION
if (
SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION
not SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION
and condition["path"] != "id"
and condition["relation"] in ["is", "is_not", "in", "not_in"]
and isinstance(values[0], dict)
Expand Down
10 changes: 6 additions & 4 deletions tests/test_unit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,7 @@ def test_invalid(self):

self.assertRaises(api.ShotgunError, api.shotgun._translate_filters, filters, "all")

@mock.patch.dict(os.environ, {"SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION": "1"})
def test_related_object(self):
filters = [
[
Expand All @@ -434,10 +435,11 @@ def test_related_object(self):
}
],
}
api.Shotgun("http://server_path", "script_name", "api_key", connect=False)
result = api.shotgun._translate_filters(filters, "all")
self.assertEqual(result, expected)

@mock.patch.dict(os.environ, {"SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION": "1"})
@mock.patch("shotgun_api3.shotgun.SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION", False)
def test_related_object_entity_optimization_is(self):
filters = [
[
Expand Down Expand Up @@ -487,7 +489,7 @@ def test_related_object_entity_optimization_is(self):
result = api.shotgun._translate_filters(filters, "all")
self.assertEqual(result, expected)

@mock.patch.dict(os.environ, {"SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION": "1"})
@mock.patch("shotgun_api3.shotgun.SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION", False)
def test_related_object_entity_optimization_in(self):
filters = [
[
Expand Down Expand Up @@ -561,7 +563,7 @@ def test_related_object_update_entity(self):
result = sg._translate_update_params(entity_type, entity_id, data, multi_entity_update_modes)
self.assertEqual(result, expected)

@mock.patch.dict(os.environ, {"SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION": "1"})
@mock.patch("shotgun_api3.shotgun.SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION", False)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we want to keep the patch here? Instead of removing the line since it's enabled by default?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems since the module shotgun is loaded in memory, the global module variable value is kept. So this mocking reverts the value to the original value.

def test_related_object_update_optimization_entity(self):
entity_type = "Anything"
entity_id = 999
Expand Down Expand Up @@ -612,7 +614,7 @@ def test_related_object_update_optimization_entity(self):
result = sg._translate_update_params(entity_type, entity_id, data, multi_entity_update_modes)
self.assertEqual(result, expected)

@mock.patch.dict(os.environ, {"SHOTGUN_API_ENABLE_ENTITY_OPTIMIZATION": "1"})
@mock.patch("shotgun_api3.shotgun.SHOTGUN_API_DISABLE_ENTITY_OPTIMIZATION", False)
def test_related_object_update_optimization_entity_multi(self):
entity_type = "Asset"
entity_id = 6626
Expand Down