Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Rename agent to connector #3165

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

[WIP] Rename agent to connector #3165

wants to merge 11 commits into from

Conversation

pingsutw
Copy link
Member

@pingsutw pingsutw commented Mar 4, 2025

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Summary by Bito

This pull request systematically renames 'Agent' to 'Connector' across the entire codebase for consistent terminology and improved maintainability. Changes span core backend modules, CLI commands, webhook components, and multiple plugins including Airflow, AWS SageMaker, BigQuery, and k8sdataservice. This terminology standardization reduces ambiguity, clarifies module responsibilities, and enhances overall code structure and maintainability.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 5

@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 4, 2025

Code Review Agent Run #a98aa0

Actionable Suggestions - 2
  • plugins/flytekit-bigquery/flytekitplugins/bigquery/agent.py - 1
    • Class name mismatch in registration call · Line 97-97
  • tests/flytekit/unit/extend/test_agent.py - 1
Review Details
  • Files reviewed - 26 · Commit Range: 4647880..4647880
    • flytekit/clis/sdk_in_container/serve.py
    • flytekit/exceptions/system.py
    • flytekit/extras/webhook/agent.py
    • flytekit/sensor/sensor_engine.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/agent.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/boto3_agent.py
    • plugins/flytekit-aws-sagemaker/tests/test_boto3_agent.py
    • plugins/flytekit-aws-sagemaker/tests/test_inference_agent.py
    • plugins/flytekit-bigquery/flytekitplugins/bigquery/__init__.py
    • plugins/flytekit-bigquery/flytekitplugins/bigquery/agent.py
    • plugins/flytekit-bigquery/tests/test_agent.py
    • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/agent.py
    • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_agent.py
    • plugins/flytekit-kf-pytorch/tests/test_elastic_task.py
    • plugins/flytekit-mmcloud/flytekitplugins/mmcloud/agent.py
    • plugins/flytekit-mmcloud/tests/test_mmcloud.py
    • plugins/flytekit-openai/flytekitplugins/openai/batch/agent.py
    • plugins/flytekit-openai/flytekitplugins/openai/chatgpt/agent.py
    • plugins/flytekit-openai/tests/chatgpt/test_agent.py
    • plugins/flytekit-openai/tests/openai_batch/test_agent.py
    • plugins/flytekit-perian/flytekitplugins/perian_job/agent.py
    • plugins/flytekit-slurm/flytekitplugins/slurm/function/agent.py
    • plugins/flytekit-snowflake/flytekitplugins/snowflake/agent.py
    • plugins/flytekit-snowflake/tests/test_agent.py
    • tests/flytekit/unit/extend/test_agent.py
    • tests/flytekit/unit/sensor/test_sensor_engine.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 4, 2025

Changelist by Bito

This pull request implements the following key changes.

Key Change Files Impacted
Feature Improvement - Connector Service Refactor

base_connector.py - Introduced new connector base and executor classes replacing legacy agent implementations.

utils.py - Updated documentation and function names to reference connectors instead of agents; established alias for get_agent_secret.

__init__.py - Replaced agent references with connector in module imports for webhook components.

task.py - Changed inheritance from SyncAgentExecutorMixin to SyncConnectorExecutorMixin to reflect renaming.

base_sensor.py - Updated base sensor to use AsyncConnectorExecutorMixin instead of the deprecated agent mixin.

sensor_engine.py - Refactored sensor engine to adopt connector registry and base classes for uniform connector management.

test_agent.py - Modified test functions and imports to use connector terminology.

__init__.py - Renamed BotoAgent and SageMakerEndpointAgent to BotoConnector and SageMakerEndpointConnector.

boto3_mixin.py - Renamed Boto3AgentMixin to Boto3ConnectorMixin to align with connector naming.

boto3_task.py - Updated inheritance to SyncConnectorExecutorMixin, reflecting the core renaming.

task.py - Revised task class to use AsyncConnectorExecutorMixin for proper connector behavior.

workflow.py - Modified documentation in parameters to reference connectors instead of agents.

test_boto3_agent.py - Updated test cases by replacing agent references with connector counterparts.

test_boto3_mixin.py - Adjusted mixin tests and string references from agent to connector.

test_inference_agent.py - Updated test cases to use connector references, corrected S3 paths, and updated function names.

serve.py - Deprecated 'agent' command in favor of a new 'connector' command with enhanced options.

base_agent.py - Replaced legacy agent base classes with connector aliases for streamlined code.

test_inference_task.py - Replaced agent references with connector in S3 input location.

test_inference_workflow.py - Updated S3 output path from agent to connector.

__init__.py - Renamed BigQueryAgent to BigQueryConnector in module init.

task.py - Switched to AsyncConnectorExecutorMixin; class inheritance updated.

test_agent.py - Modified tests to replace agent with connector and update function calls.

__init__.py - Updated import from agent to connector.

test_task.py - Patched K8sManager import to reference connector.

test_elastic_task.py - Updated docstring from agent to connector process.

__init__.py - Renamed MMCloudAgent to MMCloudConnector.

test_mmcloud.py - Replaced agent with connector references and updated instance assertions and async call patterns.

__init__.py - Renamed BatchEndpointAgent and ChatGPTAgent to connector counterparts.

task.py - Switched to AsyncConnectorExecutorMixin in batch task.

task.py - Replaced SyncAgentExecutorMixin with SyncConnectorExecutorMixin in ChatGPTTask.

test_agent.py - Renamed test functions to use connector terminology and updated function calls.

__init__.py - Updated import: replaced PerianAgent with PerianConnector.

task.py - Replaced AsyncAgentExecutorMixin with AsyncConnectorExecutorMixin and updated error messaging.

setup.py - Updated description to state connector instead of agent.

__init__.py - Renamed SnowflakeAgent to SnowflakeConnector.

task.py - Switched to AsyncConnectorExecutorMixin in task implementation.

test_snowflake.py - Updated database parameter from FLYTEAGENT to FLYTE in test configuration.

__init__.py - Replaced DatabricksAgent with DatabricksConnector in module init.

task.py - Updated mixin inheritance to AsyncConnectorExecutorMixin; modified messaging and updated PysparkFunctionTask.

pydoclint-errors-baseline.txt - Removed outdated references to base_agent and agent docstrings.

test_serve.py - Added test for connector prometheus port and replaced agent command with connector command.

test_serve.py - Updated CLI test to invoke 'serve connector' instead of 'serve agent'.

test_agent.py - Replaced agent references with connector in imports and test functions.

test_sensor_engine.py - Replaced sensor agent calls with connector calls for create, get, and delete operations.

test_snowflake.py - Updated Snowflake URI to replace FLYTEAGENT with FLYTE.

Other Improvements - Connector Exception Update

system.py - Renamed exception from FlyteAgentNotFound to FlyteConnectorNotFound for consistency.

@@ -94,4 +94,4 @@ def delete(self, resource_meta: BigQueryMetadata, **kwargs):
client.cancel_job(resource_meta.job_id, resource_meta.project, resource_meta.location)


AgentRegistry.register(BigQueryAgent())
AgentRegistry.register(BigQueryConnector())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Class name mismatch in registration call

The class name BigQueryConnector is being used in the registration call, but based on the surrounding code context, it appears the class is actually named BigQueryAgent. This mismatch will likely cause runtime errors.

Code suggestion
Check the AI-generated fix before applying
Suggested change
AgentRegistry.register(BigQueryConnector())
AgentRegistry.register(BigQueryAgent())

Code Review Run #a98aa0


Should Bito avoid suggestions like this for future reviews? (Manage Rules)

  • Yes, avoid them

pingsutw added 3 commits March 4, 2025 11:37
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 4, 2025

Code Review Agent Run #71e148

Actionable Suggestions - 2
  • plugins/flytekit-mmcloud/tests/test_mmcloud.py - 2
Review Details
  • Files reviewed - 49 · Commit Range: 4647880..4383c12
    • flytekit/clis/sdk_in_container/serve.py
    • flytekit/exceptions/system.py
    • flytekit/extend/backend/base_agent.py
    • flytekit/extend/backend/base_connector.py
    • flytekit/extend/backend/connector_service.py
    • flytekit/extend/backend/utils.py
    • flytekit/extras/webhook/__init__.py
    • flytekit/extras/webhook/task.py
    • flytekit/sensor/base_sensor.py
    • flytekit/sensor/sensor_engine.py
    • plugins/flytekit-airflow/flytekitplugins/airflow/__init__.py
    • plugins/flytekit-airflow/flytekitplugins/airflow/connector.py
    • plugins/flytekit-airflow/flytekitplugins/airflow/task.py
    • plugins/flytekit-airflow/tests/test_agent.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/__init__.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/boto3_mixin.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/boto3_task.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/task.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/workflow.py
    • plugins/flytekit-aws-sagemaker/tests/test_boto3_agent.py
    • plugins/flytekit-aws-sagemaker/tests/test_boto3_mixin.py
    • plugins/flytekit-aws-sagemaker/tests/test_inference_agent.py
    • plugins/flytekit-aws-sagemaker/tests/test_inference_task.py
    • plugins/flytekit-aws-sagemaker/tests/test_inference_workflow.py
    • plugins/flytekit-bigquery/flytekitplugins/bigquery/__init__.py
    • plugins/flytekit-bigquery/flytekitplugins/bigquery/task.py
    • plugins/flytekit-bigquery/tests/test_agent.py
    • plugins/flytekit-mmcloud/flytekitplugins/mmcloud/__init__.py
    • plugins/flytekit-mmcloud/tests/test_mmcloud.py
    • plugins/flytekit-openai/flytekitplugins/openai/__init__.py
    • plugins/flytekit-openai/flytekitplugins/openai/batch/task.py
    • plugins/flytekit-openai/flytekitplugins/openai/chatgpt/task.py
    • plugins/flytekit-openai/tests/chatgpt/test_agent.py
    • plugins/flytekit-perian/flytekitplugins/perian_job/__init__.py
    • plugins/flytekit-perian/flytekitplugins/perian_job/task.py
    • plugins/flytekit-perian/setup.py
    • plugins/flytekit-slurm/flytekitplugins/slurm/function/agent.py
    • plugins/flytekit-snowflake/flytekitplugins/snowflake/__init__.py
    • plugins/flytekit-snowflake/flytekitplugins/snowflake/task.py
    • plugins/flytekit-snowflake/tests/test_snowflake.py
    • plugins/flytekit-spark/flytekitplugins/spark/__init__.py
    • plugins/flytekit-spark/flytekitplugins/spark/connector.py
    • plugins/flytekit-spark/flytekitplugins/spark/task.py
    • tests/flytekit/clis/sdk_in_container/test_serve.py
    • tests/flytekit/unit/cli/pyflyte/test_serve.py
    • tests/flytekit/unit/extras/webhook/test_agent.py
    • tests/flytekit/unit/extras/webhook/test_end_to_end.py
    • tests/flytekit/unit/sensor/test_sensor_engine.py
    • tests/flytekit/unit/types/structured_dataset/test_snowflake.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

@@ -110,12 +110,12 @@ def say_hello0(name: str) -> str:
return f"Hello, {name}."

task_spec = get_serializable(OrderedDict(), serialization_settings, say_hello0)
agent = AgentRegistry.get_agent(task_spec.template.type)
connector = ConnectorRegistry.get_connector(task_spec.template.type)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing import for ConnectorRegistry

It appears that AgentRegistry has been renamed to ConnectorRegistry in this PR. This change should be reflected in the imports as well. Currently, the import statement for ConnectorRegistry is missing, which could lead to runtime errors.

Code suggestion
Check the AI-generated fix before applying
 @@ -9,6 +9,7 @@
  from flyteidl.core.execution_pb2 import TaskExecution
  from flytekitplugins.mmcloud import MMCloudConnector, MMCloudConfig, MMCloudTask
  from flytekitplugins.mmcloud.utils import async_check_output, flyte_to_mmcloud_resources
 +from flytekit.extend.backend.base_connector import ConnectorRegistry
  from flytekit import Resources, task
  from flytekit.configuration import DefaultImages, ImageConfig, SerializationSettings
  from flytekit.extend import get_serializable

Code Review Run #71e148


Should Bito avoid suggestions like this for future reviews? (Manage Rules)

  • Yes, avoid them


assert isinstance(agent, MMCloudAgent)
assert isinstance(connector, MMCloudConnector)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing import for MMCloudConnector

The assertion is checking for MMCloudConnector but there's no import for this class. The import statement should be updated to include MMCloudConnector instead of or in addition to MMCloudAgent.

Code suggestion
Check the AI-generated fix before applying
 @@ -10,6 +10,7 @@
  from flyteidl.core.execution_pb2 import TaskExecution
  from flytekitplugins.mmcloud import MMCloudConnector, MMCloudConfig, MMCloudTask
  from flytekitplugins.mmcloud.utils import async_check_output, flyte_to_mmcloud_resources
 +from flytekit.extend.backend.base_connector import ConnectorRegistry
  from flytekit import Resources, task
  from flytekit.configuration import DefaultImages, ImageConfig, SerializationSettings
  from flytekit.extend import get_serializable

Code Review Run #71e148


Should Bito avoid suggestions like this for future reviews? (Manage Rules)

  • Yes, avoid them

Signed-off-by: Kevin Su <[email protected]>
@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 5, 2025

Code Review Agent Run #c2bc44

Actionable Suggestions - 0
Review Details
  • Files reviewed - 13 · Commit Range: 4383c12..4ee1710
    • flytekit/clis/sdk_in_container/serve.py
    • flytekit/extend/backend/base_agent.py
    • flytekit/sensor/sensor_engine.py
    • plugins/flytekit-airflow/flytekitplugins/airflow/connector.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/__init__.py
    • plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/connector.py
    • plugins/flytekit-bigquery/flytekitplugins/bigquery/connector.py
    • plugins/flytekit-mmcloud/flytekitplugins/mmcloud/connector.py
    • plugins/flytekit-openai/flytekitplugins/openai/batch/connector.py
    • plugins/flytekit-perian/flytekitplugins/perian_job/connector.py
    • plugins/flytekit-snowflake/flytekitplugins/snowflake/connector.py
    • plugins/flytekit-spark/flytekitplugins/spark/connector.py
    • tests/flytekit/unit/cli/pyflyte/test_serve.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

Signed-off-by: Kevin Su <[email protected]>
@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 10, 2025

Code Review Agent Run #64ed85

Actionable Suggestions - 0
Review Details
  • Files reviewed - 3 · Commit Range: 4ee1710..e2f1bf9
    • flytekit/extend/backend/base_connector.py
    • pydoclint-errors-baseline.txt
    • tests/flytekit/unit/extend/test_connector.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

Signed-off-by: Kevin Su <[email protected]>
Copy link

codecov bot commented Mar 10, 2025

Codecov Report

Attention: Patch coverage is 87.25100% with 32 lines in your changes missing coverage. Please review.

Project coverage is 75.30%. Comparing base (0560327) to head (02f5bdf).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
flytekit/extend/backend/base_connector.py 86.79% 18 Missing and 10 partials ⚠️
flytekit/extend/backend/connector_service.py 73.33% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3165      +/-   ##
==========================================
- Coverage   78.74%   75.30%   -3.45%     
==========================================
  Files         321      214     -107     
  Lines       27055    22261    -4794     
  Branches     2901     2901              
==========================================
- Hits        21305    16764    -4541     
+ Misses       4947     4647     -300     
- Partials      803      850      +47     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 10, 2025

Code Review Agent Run #d82c2b

Actionable Suggestions - 0
Review Details
  • Files reviewed - 33 · Commit Range: e2f1bf9..24de61d
    • Dockerfile.agent
    • flytekit/clis/sdk_in_container/metrics.py
    • flytekit/core/array_node.py
    • flytekit/core/context_manager.py
    • flytekit/core/interface.py
    • flytekit/core/python_function_task.py
    • flytekit/core/worker_queue.py
    • flytekit/image_spec/__init__.py
    • flytekit/image_spec/default_builder.py
    • flytekit/image_spec/image_spec.py
    • flytekit/models/interface.py
    • flytekit/remote/executions.py
    • flytekit/remote/metrics.py
    • flytekit/remote/remote.py
    • flytekit/types/file/file.py
    • plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/pod_template.py
    • plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py
    • plugins/flytekit-kf-pytorch/tests/test_shared.py
    • plugins/flytekit-slurm/flytekitplugins/slurm/__init__.py
    • plugins/flytekit-slurm/flytekitplugins/slurm/function/task.py
    • plugins/flytekit-slurm/flytekitplugins/slurm/script/agent.py
    • plugins/flytekit-slurm/flytekitplugins/slurm/script/task.py
    • plugins/flytekit-slurm/flytekitplugins/slurm/ssh_utils.py
    • plugins/flytekit-slurm/setup.py
    • plugins/flytekit-slurm/tests/test_slurm_shell_task.py
    • plugins/flytekit-spark/flytekitplugins/spark/task.py
    • plugins/flytekit-spark/setup.py
    • tests/flytekit/integration/remote/workflows/basic/flytefile.py
    • tests/flytekit/unit/core/image_spec/test_image_spec.py
    • tests/flytekit/unit/core/test_async.py
    • tests/flytekit/unit/core/test_context_manager.py
    • tests/flytekit/unit/models/test_interface.py
    • tests/flytekit/unit/types/file/test_file.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

Signed-off-by: Kevin Su <[email protected]>
@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 10, 2025

Code Review Agent Run #5dcdce

Actionable Suggestions - 0
Review Details
  • Files reviewed - 1 · Commit Range: 24de61d..449811b
    • flytekit/extend/backend/base_connector.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 11, 2025

Code Review Agent Run #003d61

Actionable Suggestions - 0
Review Details
  • Files reviewed - 3 · Commit Range: 449811b..1617c06
    • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_task.py
    • plugins/flytekit-spark/tests/test_connector.py
    • tests/flytekit/unit/extend/test_connector.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

Signed-off-by: Kevin Su <[email protected]>
@flyte-bot
Copy link
Contributor

flyte-bot commented Mar 12, 2025

Code Review Agent Run #c18704

Actionable Suggestions - 0
Review Details
  • Files reviewed - 1 · Commit Range: 1617c06..bdf12a8
    • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/__init__.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

  • /review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by Bito Logo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants