Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New Feature: Agent Names captured in traces and logs to aggregate per agent stats #10174

Open
evmin opened this issue Jan 13, 2025 · 4 comments · Fixed by #10255
Open

New Feature: Agent Names captured in traces and logs to aggregate per agent stats #10174

evmin opened this issue Jan 13, 2025 · 4 comments · Fixed by #10255
Assignees
@TaoChenOSU @dmytrostruk
Labels
agents experimental Associated with an experimental feature

Comments

@evmin
Copy link

evmin commented Jan 13, 2025

name: Feature request
about: Tracing agent names

For the internal audit I would like to be able to trace agent token usage statistics.

  • Invocation within a chat
  • Within the chat across multiple invocations
  • Across multiple chats and invocations

I am OK to rely on Kusto SQL, but I need the agent names to be captured in traces and logs automatically.
@TaoChenOSU TaoChenOSU self-assigned this Jan 13, 2025
@crickman crickman added agents experimental Associated with an experimental feature labels Jan 14, 2025
@TaoChenOSU TaoChenOSU mentioned this issue Jan 14, 2025
Open
4 tasks
@sphenry
Copy link
Member

sphenry commented Jan 21, 2025

Adding @dmytrostruk , it might make sense that this should be at the Kernel level

@sphenry sphenry removed the triage label Jan 21, 2025
@TaoChenOSU TaoChenOSU mentioned this issue Jan 22, 2025
Merged
4 tasks
@TaoChenOSU TaoChenOSU linked a pull request Jan 22, 2025 that will close this issue
Python: Add agent invocation spans #10255
Merged
4 tasks
@markwallace-microsoft
Copy link
Member

@dmytrostruk can you handle making the equivalent .NET changes

@markwallace-microsoft markwallace-microsoft moved this to Sprint: Planned in Semantic Kernel Jan 22, 2025
moonbox3 added a commit that referenced this issue Jan 22, 2025
@TaoChenOSU @moonbox3
Python: Add agent invocation spans (#10255)
6ecfdcf 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Address feature request:
#10174

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
This pull request introduces a new decorator for tracing agent
invocations and applies it to various methods in the
`ChatCompletionAgent` and `OpenAIAssistantAgent` classes. Additionally,
it includes new unit tests to ensure the decorator is correctly applied
and functions as expected.

### Key Changes:

**Decorator Implementation:**
* Added a new `trace_agent_invocation` decorator in
`python/semantic_kernel/utils/telemetry/agent_diagnostics/decorators.py`
to trace agent invocations using OpenTelemetry.

**Decorator Application:**
* Applied the `trace_agent_invocation` decorator to the `invoke` and
`invoke_stream` methods in `ChatCompletionAgent`
(`python/semantic_kernel/agents/chat_completion/chat_completion_agent.py`).
[[1]](diffhunk://#diff-0f0a27c107368504c4347c88528b7b4234dcead1919005bcae13f5d16d6cf26dR19)
[[2]](diffhunk://#diff-0f0a27c107368504c4347c88528b7b4234dcead1919005bcae13f5d16d6cf26dR80)
[[3]](diffhunk://#diff-0f0a27c107368504c4347c88528b7b4234dcead1919005bcae13f5d16d6cf26dR134)
* Applied the `trace_agent_invocation` decorator to the `invoke` and
`invoke_stream` methods in `OpenAIAssistantAgent`
(`python/semantic_kernel/agents/open_ai/open_ai_assistant_base.py`).
[[1]](diffhunk://#diff-70e75c57136d3a90d045af4b8d59fb8e8101251d7c5126458bb5d9e6c36556a4R46)
[[2]](diffhunk://#diff-70e75c57136d3a90d045af4b8d59fb8e8101251d7c5126458bb5d9e6c36556a4R608)
[[3]](diffhunk://#diff-70e75c57136d3a90d045af4b8d59fb8e8101251d7c5126458bb5d9e6c36556a4R861)

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Evan Mattson <[email protected]>
@TaoChenOSU TaoChenOSU reopened this Jan 22, 2025
@TaoChenOSU
Copy link
Contributor

TaoChenOSU commented Jan 22, 2025 •
edited
Loading

Reopening for .Net

@dmytrostruk We may need to similar sample in .Net to verify the feature: #10184

Please let me know if you want to split the work or have a discussion.

@dmytrostruk dmytrostruk moved this from Sprint: Planned to Sprint: In Progress in Semantic Kernel Jan 23, 2025
@crickman
Copy link
Contributor

crickman commented Jan 27, 2025 •
edited
Loading

Including the agent-id along with the name may be something to consider. Name is optional, but Id is required. Name is also not enforced as unique. May want to revisit the Python changes as well.

@dmytrostruk dmytrostruk mentioned this issue Jan 28, 2025
Merged
4 tasks
github-merge-queue bot pushed a commit that referenced this issue Jan 29, 2025
@dmytrostruk
.Net: Updated agent logs to include agent name (#10330)
c311183 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

Related: #10174

This PR updates logging messages in Agent Framework to include agent
name. `UnnamedAgent` is used as a default value when the agent's name is
not specified to provide meaningful context in logs.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TaoChenOSU TaoChenOSU

@dmytrostruk dmytrostruk

Labels
agents experimental Associated with an experimental feature
Projects
Semantic Kernel
Status: Sprint: In Progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Python: Add agent invocation spans TaoChenOSU/semantic-kernel-fork
6 participants
@evmin @sphenry @TaoChenOSU @dmytrostruk @crickman @markwallace-microsoft