fix: Resolve 502 deployment failure by upgrading Azure AI and agent dependencies

[Kanchan-Microsoft]

Purpose

This pull request updates several dependencies across the project, primarily to support newer versions of Azure AI and related frameworks, and to improve compatibility and stability. The main focus is on upgrading the azure-ai-projects package and restructuring agent framework dependencies.

Dependency upgrades and restructuring:

• Upgraded azure-ai-projects to version 2.0.0b3 in both content-gen/infra/vscode_web/requirements.txt and content-gen/src/backend/requirements.txt to ensure consistency and access to new features.
• Replaced the single agent-framework dependency with agent-framework-azure-ai and agent-framework-core, both pinned to version 1.0.0b260114, in content-gen/src/backend/requirements.txt for better modularity and clarity.
• Added opentelemetry-semantic-conventions-ai==0.4.13 to support OpenTelemetry tracing required by the agent framework.

Azure SDK adjustments:

• Added an explicit version pin for azure-search-documents (>=11.4.0) in content-gen/src/backend/requirements.txt for improved dependency management; removed the same from content-gen/scripts/requirements-post-deploy.txt to prevent redundancy.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information

[github-actions]

🎉 This PR is included in version 2.2.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀