Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion content/academy/evaluate/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ If production behavior matches expectations, you can keep scaling with more conf
lede: "Test whether your judge prompt agrees with how you would label cases, and iterate until it does.",
},
{
href: "/guides/cookbook/evaluation_of_rag_with_ragas",
href: "/resources/engineering/evaluation-of-rag-with-ragas",
topic: "RAG evaluation with RAGAS",
lede: "Score retrieval and generation quality for a RAG application.",
},
Expand Down
2 changes: 1 addition & 1 deletion content/academy/japan/evaluate.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ _アップロードされた PDF から構造化フィールド (ベンダー、
title="その他のガイド"
guides={[
{
href: "/guides/cookbook/evaluation_of_rag_with_ragas",
href: "/resources/engineering/evaluation-of-rag-with-ragas",
topic: "RAGAS を使った RAG 評価",
lede: "RAG アプリケーションのリトリーバルと生成の品質を採点します。",
},
Expand Down
9 changes: 2 additions & 7 deletions content/blog/2024-04-python-decorator.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@

## Design decisions

1. The decorator reuses the low-level SDK to create traces and asynchronously batch them to the Langfuse API. This implementation was derived from the PostHog SDKs and is [tested](/guides/cookbook/langfuse_sdk_performance_test) to have little to no impact on the performance of your application.
1. The decorator reuses the low-level SDK to create traces and asynchronously batch them to the Langfuse API. This implementation was derived from the PostHog SDKs and is [tested](/resources/engineering/langfuse-sdk-performance-test) to have little to no impact on the performance of your application.
2. The decorator maintains a call stack internally that keeps track of nested function calls to reflect the observation hierarchy in the trace.
3. To be async-safe, the decorator leverages [Python Contextvars](https://docs.python.org/3/library/contextvars.html) for managing its state.
4. The same `observe()` decorator is used to create a trace (outermost decorated function) and to add spans to the trace (inner decorated functions). This way, functions can be used in multiple traces without needing to be strictly a "trace" or a "span" function.
Expand Down Expand Up @@ -213,7 +213,7 @@
- Langfuse OpenAI SDK Wrapper for automatic instrumentation of OpenAI calls
- LiteLLM Proxy for standardization of 100+ models on the OpenAI API

Have a look at [this cookbook](/guides/cookbook/integration_litellm_proxy) to see an end-to-end example – we really think you'll like this stack and there are lots of teams in the Langfuse Community who built on top of it.
Have a look at [this cookbook](/integrations/gateways/litellm#openai-sdk-wrapper) to see an end-to-end example – we really think you'll like this stack and there are lots of teams in the Langfuse Community who built on top of it.

## Thank you

Expand All @@ -229,12 +229,7 @@
<Card title="Docs" href="/docs/sdk/python/decorators" icon={<BookOpen />} />
<Card
title="Example notebook"
href="/docs/sdk/python/example"
icon={<FileCode />}
/>
<Card
title="Video introduction"
href="/guides/videos/introducing-python-decorator"
icon={<VideoIcon />}
/>
</Cards>

Check warning on line 235 in content/blog/2024-04-python-decorator.mdx

View check run for this annotation

Claude / Claude Code Review

[quality] Unused VideoIcon import left after Card removal

In content/blog/2024-04-python-decorator.mdx, the PR removes the 'Video introduction' Card (icon={<VideoIcon />}, linking to the deleted /guides/videos/introducing-python-decorator page) but leaves the now-unused `Video as VideoIcon` import from lucide-react on line 226. Trim it to `import { FileCode, BookOpen } from "lucide-react";`.
Comment on lines 232 to 235

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 In content/blog/2024-04-python-decorator.mdx, the PR removes the 'Video introduction' Card (icon={}, linking to the deleted /guides/videos/introducing-python-decorator page) but leaves the now-unused Video as VideoIcon import from lucide-react on line 226. Trim it to import { FileCode, BookOpen } from "lucide-react";.

Extended reasoning...

The PR deletes the Video introduction <Card> at the end of this blog post, which was the only place using <VideoIcon /> (it linked to /guides/videos/introducing-python-decorator, a page also removed in this PR). However, the import statement on line 226 was left unchanged: import { FileCode, BookOpen, Video as VideoIcon } from "lucide-react";.

A grep for VideoIcon across the file confirms it now appears exactly once — in that import statement — with zero remaining JSX usages. Note that the '&lt;Video src=... /&gt;' tags elsewhere in the file (e.g. lines 44 and 56) are unrelated: those reference the project's custom MDX video-embed component, not the lucide-react icon that gets aliased to VideoIcon here. So there is no other consumer that would make this import still necessary.

FileCode and BookOpen are both still referenced by the two remaining <Card> elements, so only VideoIcon needs to be dropped. The fix is a one-line change:

- import { FileCode, BookOpen, Video as VideoIcon } from "lucide-react";
+ import { FileCode, BookOpen } from "lucide-react";

Step-by-step proof:

  1. Before this PR, the file had three Cards: Docs, Example notebook, and Video introduction (the last using icon={<VideoIcon />}).
  2. This PR deletes the Video introduction Card block entirely (visible in the diff), leaving only the Docs and Example notebook Cards.
  3. The import line itself was not touched by the diff, so Video as VideoIcon is still imported.
  4. Since no JSX in the file references VideoIcon anymore, it is dead code — an artifact of the Card removal rather than an intentional import.

This is a pure MDX cleanup: an unused import in an MDX frontmatter-adjacent JS import doesn't break the build, doesn't affect rendering, and doesn't fail any CI check (H1 check, Prettier, link-check are all unaffected). It's a straightforward nit a maintainer would want cleaned up during review, but it does not block merging.

4 changes: 2 additions & 2 deletions content/blog/2024-07-ai-agent-observability-with-langfuse.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ You do not need a specific framework to build AI agents, and Langfuse is deliber

LangGraph ([GitHub](https://github.com/langchain-ai/langgraph)) is an open-source framework by the LangChain team for building complex, stateful, multi-agent applications. LangGraph includes built-in persistence to save and resume state, which enables error recovery and human-in-the-loop workflows.

LangGraph agents can be [monitored with Langfuse](/guides/cookbook/integration_langgraph), and the agent graph view renders the LangGraph structure automatically.
LangGraph agents can be [monitored with Langfuse](/integrations/frameworks/langgraph), and the agent graph view renders the LangGraph structure automatically.

<Video
src="https://static.langfuse.com/docs-videos/langgraph-overview.mp4"
Expand Down Expand Up @@ -355,7 +355,7 @@ Following the Academy's loop, agent evaluation runs in two places. **Offline**,

If you want to get started with building AI agents and monitoring them with Langfuse, here are the best places to begin:

- **Build and trace an agent:** Follow our [end-to-end example](/guides/cookbook/integration_langgraph) of building a simple agent with LangGraph and tracking it with Langfuse.
- **Build and trace an agent:** Follow our [end-to-end example](/integrations/frameworks/langgraph) of building a simple agent with LangGraph and tracking it with Langfuse.
- **Learn the concepts:** Work through the [Langfuse Academy](/academy) to understand the AI engineering loop from tracing to evaluation.
- **Evaluate your agents:** Read the [AI agent evaluation guide](/resources/engineering/ai-agent-evaluation) and set up your first dataset and experiment.
- **Explore all integrations:** Browse the full list of [supported integrations](/integrations) to find the right setup for your stack.
4 changes: 2 additions & 2 deletions content/blog/2024-08-what-is-langchain.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,15 @@ LangSmith is a closed-source, commercial offering by the company behind the Lang

The answer is, **[LangGraph](https://www.langchain.com/langgraph) is a framework for developing AI agents**. LangGraph [(GitHub)](https://github.com/langchain-ai/langgraph) represents LLM applications as graphs with nodes and edges. Each node represents a task, and an edge describes the flow of data.

**Using Langfuse with LangGraph**: For observability into LangGraph Agents, you can integrate [LangServe with Langfuse](/guides/cookbook/integration_langgraph). Using the open-source Langfuse library, you can gain detailed traces and evaluate your LangChain agents.
**Using Langfuse with LangGraph**: For observability into LangGraph Agents, you can integrate [LangGraph with Langfuse](/integrations/frameworks/langgraph). Using the open-source Langfuse library, you can gain detailed traces and evaluate your LangChain agents.

### What is LangServe?

The answer is, **[LangServe](https://python.langchain.com/v0.2/docs/langserve/) is a library that helps deploy LangChain applications via a REST API**.

Using LangServe will allow you to create a scalable Python web server for your LangChain application that you can then deploy and scale out to your Cloud provider of choice, such as GCP or Replit.

**Logging and Debugging for LangServe**: You can integrate LangServe with Langfuse for [open-source observability](/integrations/frameworks/langserve). You can use the Langfuse UI to debug, analyze, and iterate on your LangServe application.
**Logging and Debugging for LangServe**: You can integrate LangServe with Langfuse for [open-source observability](/integrations/frameworks/langchain). You can use the Langfuse UI to debug, analyze, and iterate on your LangServe application.

## Langfuse - an alternative to LangChain?

Expand Down
4 changes: 2 additions & 2 deletions content/blog/2024-10-opentelemetry-for-llm-observability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ OpenTelemetry supports various exporters and backends, making it flexible and ad

This post is a high-level overview of the challenges of LLM observability and the current state of using OpenTelemetry (OTel) for LLMOps.

OTel is geared towards general observability, and **traces are a great standardized way to capture LLM application data** – we have recorded a [webinar](/guides/videos/webinar-observability-llm-systems) on this. While we are excited about OTel and the roadmap towards it across LLMOps tools, non-OTel LLMOps tools are preferred by many teams. This post explores why this is the case and how OTel can address these challenges in the future.
OTel is geared towards general observability, and **traces are a great standardized way to capture LLM application data** – we have recorded a [webinar](/resources/engineering/webinar-observability-llm-systems) on this. While we are excited about OTel and the roadmap towards it across LLMOps tools, non-OTel LLMOps tools are preferred by many teams. This post explores why this is the case and how OTel can address these challenges in the future.

_Example trace of our [public demo](/docs/demo)_

Expand Down Expand Up @@ -145,4 +145,4 @@ We are committed to OTel and are happy to contribute to the SIG. We will continu

If you want to get started with tracing your AI applications with Langfuse today, check out our [quickstart guide](/docs/get-started) on how to use Langfuse with multiple LLM building frameworks like [Langchain](/integrations/frameworks/langchain) or [LlamaIndex](/integrations/frameworks/llamaindex).

If you are curious about why Traces are a good fit for LLM observability, check out our [webinar](/guides/videos/webinar-observability-llm-systems) on the topic.
If you are curious about why Traces are a good fit for LLM observability, check out our [webinar](/resources/engineering/webinar-observability-llm-systems) on the topic.
2 changes: 1 addition & 1 deletion content/blog/2025-03-19-ai-agent-comparison.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ If you want the short answer before the details:
<Video src="https://static.langfuse.com/docs-videos/langgraph-overview.mp4" gifStyle aspectRatio={1546 / 1080} />
_[Example trace in Langfuse](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/a8b0cc9e-da3b-485f-a642-35431a6f9289)_

[How to trace LangGraph agents with Langfuse →](/guides/cookbook/integration_langgraph)
[How to trace LangGraph agents with Langfuse →](/integrations/frameworks/langgraph)

Developers who prefer to model AI tasks as stateful workflows gravitate toward LangGraph. If your application demands robust task decomposition, parallel branching, or the ability to inject custom logic at specific stages, LangGraph's explicit approach is a good fit.

Expand Down
2 changes: 1 addition & 1 deletion content/blog/2025-11-12-evals.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ For apps beyond one-shot queries, extend the basics. The methods stay the same;

### RAG pipelines [#rag]

Because a RAG system has a retrieval step and a generation step, measure them separately: retrieval relevance and precision for the document-fetching stage, then generation-side metrics on the answer. The two metrics that carry most RAG evaluations are **faithfulness** (does the answer stick to the retrieved context?) and **answer relevance** (does the answer actually address the question?). We cover judge design and runnable code for both in [RAG faithfulness evaluation](/resources/engineering/rag-faithfulness-evaluation) and [answer relevance evaluation](/resources/engineering/answer-relevance-evaluation), and when ungrounded output is your main worry, [hallucination detection](/resources/engineering/hallucination-detection) walks through a taxonomy and evaluator setup. Langfuse also integrates with [RAGAS](/guides/cookbook/evaluation_of_rag_with_ragas) for specialized RAG metrics.
Because a RAG system has a retrieval step and a generation step, measure them separately: retrieval relevance and precision for the document-fetching stage, then generation-side metrics on the answer. The two metrics that carry most RAG evaluations are **faithfulness** (does the answer stick to the retrieved context?) and **answer relevance** (does the answer actually address the question?). We cover judge design and runnable code for both in [RAG faithfulness evaluation](/resources/engineering/rag-faithfulness-evaluation) and [answer relevance evaluation](/resources/engineering/answer-relevance-evaluation), and when ungrounded output is your main worry, [hallucination detection](/resources/engineering/hallucination-detection) walks through a taxonomy and evaluator setup. Langfuse also integrates with [RAGAS](/resources/engineering/evaluation-of-rag-with-ragas) for specialized RAG metrics.

### AI agents [#agents]

Expand Down
4 changes: 2 additions & 2 deletions content/blog/launch-week-1.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ const res = await openai.chat.completions.create({
});
```

We [launched](/changelog/2024-04-21-openai-integration-JS-SDK) a new wrapper for the OpenAI JS SDK. This integration, designed to enable easier monitoring of OpenAI API usage, features seamless observability with enhancements like automatic tracking of prompts, completions, and API errors, as well as insights into model usage and costs. After a soft launch that gathered user feedback for improvements, the integration is now fully available, complete with comprehensive [documentation](/integrations/model-providers/openai-js) and an [example notebook](/guides/cookbook/js_integration_openai).
We [launched](/changelog/2024-04-21-openai-integration-JS-SDK) a new wrapper for the OpenAI JS SDK. This integration, designed to enable easier monitoring of OpenAI API usage, features seamless observability with enhancements like automatic tracking of prompts, completions, and API errors, as well as insights into model usage and costs. After a soft launch that gathered user feedback for improvements, the integration is now fully available, complete with comprehensive [documentation](/integrations/model-providers/openai-js) and an example notebook.

### Day 1: PostHog Integration

Expand Down Expand Up @@ -74,7 +74,7 @@ We're excited to introduce the [**LLM Playground**](/docs/playground) to Langfus
title="Decorator Integration"
/>

We're happy to share that the Decorator-based integration for Python now supports all Langfuse features and is the recommended way to use Langfuse in Python. The decorator makes integrating with Langfuse so much easier. Head over to the [Python Decorator docs](/docs/sdk/python/decorators) to learn more. All inputs, outputs, timings are captured automatically, and it works with all other [Langfuse integrations](/integrations) (LangChain, LlamaIndex, OpenAI SDK, ...). To celebrate this milestone, we wrote a [blog post](/blog/2024-04-python-decorator) on the technical details and created the [example notebook](/guides/cookbook/example_decorator_openai_langchain) shown in the video as it demonstrates what's really cool about the decorator. Thanks again to [@lshalon](https://github.com/lshalon) and [@AshisGhosh](https://github.com/AshisGhosh) for your contributions to this!
We're happy to share that the Decorator-based integration for Python now supports all Langfuse features and is the recommended way to use Langfuse in Python. The decorator makes integrating with Langfuse so much easier. Head over to the [Python Decorator docs](/docs/sdk/python/decorators) to learn more. All inputs, outputs, timings are captured automatically, and it works with all other [Langfuse integrations](/integrations) (LangChain, LlamaIndex, OpenAI SDK, ...). To celebrate this milestone, we wrote a [blog post](/blog/2024-04-python-decorator) on the technical details and created the example notebook shown in the video as it demonstrates what's really cool about the decorator. Thanks again to [@lshalon](https://github.com/lshalon) and [@AshisGhosh](https://github.com/AshisGhosh) for your contributions to this!

### Day 4: Datasets v2

Expand Down
2 changes: 1 addition & 1 deletion content/changelog/2024-02-27-llama-index-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ PS: If you are interested in an integration with LlamaIndex.TS, add your upvote/
/>
<span>
_Based on the [LlamaIndex
Cookbook](/guides/cookbook/integration_llama_index)._
integration](/integrations/frameworks/llamaindex)._
</span>

### 📚 More details
Expand Down
4 changes: 2 additions & 2 deletions content/changelog/2024-03-05-uptrain-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ import { FileCode } from "lucide-react";

<Cards num={3}>
<Card
title="Uptrain + Langfuse Cookbook"
href="/guides/cookbook/evaluation_with_uptrain"
title="External evaluation pipelines cookbook"
href="/guides/cookbook/example_external_evaluation_pipelines"
icon={<FileCode />}
/>
</Cards>
Expand Down
2 changes: 1 addition & 1 deletion content/changelog/2024-04-21-openai-integration-JS-SDK.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ import { FileCode } from "lucide-react";
/>
<Card
title="Example Notebook"
href="/guides/cookbook/js_integration_openai"
href="/integrations/model-providers/openai-js"
icon={<FileCode />}
/>
</Cards>
7 changes: 1 addition & 6 deletions content/changelog/2024-04-24-decorator.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ On Day 3 of [Launch Week 1](/blog/launch-week-1), we're happy to share that the

The decorator makes integrating with Langfuse so much easier. Head over to the [Python Decorator docs](/docs/sdk/python/decorators) to learn more. All inputs, outputs, timings are captured automatically, and it works with all other [Langfuse integrations](/integrations) (LangChain, LlamaIndex, OpenAI SDK, ...).

To celebrate this milestone, we wrote a [blog post](/blog/2024-04-python-decorator) on the technical details and created the [example notebook](/guides/cookbook/example_decorator_openai_langchain) shown in the video as it demonstrates what's really cool about the decorator. Let us know what you think in the [GitHub discussion](https://github.com/orgs/langfuse/discussions/1009), and stay tuned for more updates during [Langfuse Launch Week 1](/blog/launch-week-1) 🚀
To celebrate this milestone, we wrote a [blog post](/blog/2024-04-python-decorator) on the technical details and created the example notebook shown in the video as it demonstrates what's really cool about the decorator. Let us know what you think in the [GitHub discussion](https://github.com/orgs/langfuse/discussions/1009), and stay tuned for more updates during [Langfuse Launch Week 1](/blog/launch-week-1) 🚀

Thanks again to [@lshalon](https://github.com/lshalon) and [@AshisGhosh](https://github.com/AshisGhosh) for your contributions to this!

Expand All @@ -42,9 +42,4 @@ import { Book, FileCode } from "lucide-react";
href="/docs/sdk/python/example"
icon={<FileCode />}
/>
<Card
title="Rap battle example notebook"
href="/guides/cookbook/example_decorator_openai_langchain"
icon={<FileCode />}
/>
</Cards>
2 changes: 1 addition & 1 deletion content/changelog/2024-08-08-openai-structured-outputs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ import { FileCode, BookOpen } from "lucide-react";
<Cards num={3}>
<Card
title="Cookbook: OpenAI Structured Outputs"
href="/guides/cookbook/integration_openai_structured_output"
href="/integrations/model-providers/openai-py#structured-output"
icon={<FileCode />}
/>
</Cards>
4 changes: 2 additions & 2 deletions content/changelog/2024-08-21-openai-multi-modal-traces.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Langfuse now supports rendering multi-modal traces. In addition to text, you may

## How to trace vision input in Langfuse?

If you use the [OpenAI Python SDK](/integrations/model-providers/openai-py), you can follow our [OpenAI cookbook](/guides/cookbook/integration_openai_sdk) which includes an example for tracing images in Langfuse.
If you use the [OpenAI Python SDK](/integrations/model-providers/openai-py), you can follow our [OpenAI cookbook](/integrations/model-providers/openai-py) which includes an example for tracing images in Langfuse.

<Cards num={3}>
<Card
Expand All @@ -26,7 +26,7 @@ If you use the [OpenAI Python SDK](/integrations/model-providers/openai-py), you
/>
<Card
title="Cookbook: OpenAI Integration (Python)"
href="/guides/cookbook/integration_openai_sdk"
href="/integrations/model-providers/openai-py"
icon={<FileCode />}
/>
</Cards>
2 changes: 1 addition & 1 deletion content/changelog/2025-02-14-trace-graph-view.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ This feature is currently in beta. We welcome your feedback and encourage you to
- [Example trace](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/8ed12d68-353f-464f-bc62-720984c3b6a0?display=graph)
- [Agent Graphs Documentation](/docs/tracing-features/agent-graphs)
- [LangGraph](https://langchain-ai.github.io/langgraph/)
- [LangGraph + Langfuse cookbook](/guides/cookbook/integration_langgraph)
- [LangGraph + Langfuse cookbook](/integrations/frameworks/langgraph)
Original file line number Diff line number Diff line change
Expand Up @@ -413,7 +413,7 @@ Cost depends on the judge model and the size of the inputs being evaluated. A ty
<Details>
<Summary>Can I use LLM-as-a-Judge for RAG evaluation?</Summary>

Yes. LLM-as-a-Judge is particularly effective for RAG pipelines. You can evaluate faithfulness (is the answer grounded in the retrieved context?), relevance (does the answer address the question?), and completeness (does the answer cover all relevant information?). Langfuse also integrates with [RAGAS](/guides/cookbook/evaluation_of_rag_with_ragas) for specialized RAG evaluation metrics.
Yes. LLM-as-a-Judge is particularly effective for RAG pipelines. You can evaluate faithfulness (is the answer grounded in the retrieved context?), relevance (does the answer address the question?), and completeness (does the answer cover all relevant information?). Langfuse also integrates with [RAGAS](/resources/engineering/evaluation-of-rag-with-ragas) for specialized RAG evaluation metrics.

</Details>

Expand Down
Loading