LangChain support for Orchestration Client

[kay-schmitteckert]

Describe the Problem

The Orchestration Client already simplifies the process of developing and kickstarting GenAI projects as well as communicating with foundation models. Now, LangChain support for OpenAI is available, and instead of writing a separate wrapper for each vendor, the idea is to proceed with a LangChain wrapper for the Orchestration Client, which would broadly cover everything.

Propose a Solution

LangChain wrapper for Orchestration client, e.g.:

import { LLM, type BaseLLMParams } from "@langchain/core/language_models/llms";
import type { CallbackManagerForLLMRun } from "@langchain/core/callbacks/manager";

import { OrchestrationClient, OrchestrationModuleConfig, ChatMessages } from "@sap-ai-sdk/orchestration";

export interface CustomLLMInput extends BaseLLMParams {
    deploymentId: string;
    resourceGroup?: string;
    modelName: string;
    modelParams?: {};
    modelVersion?: string;
}

export class GenerativeAIHubCompletion extends LLM {
    deploymentId: string;
    resourceGroup: string;
    modelName: string;
    modelParams: {};
    modelVersion: string;

    constructor(fields: CustomLLMInput) {
        super(fields);
        this.deploymentId = fields.deploymentId;
        this.resourceGroup = fields.resourceGroup || "default";
        this.modelName = fields.modelName;
        this.modelParams = fields.modelParams || {};
        this.modelVersion = fields.modelVersion || "latest";
    }

    _llmType() {
        return "Generative AI Hub - Orchestration Service";
    }

    async _call(
        prompt: string,
        options: this["ParsedCallOptions"],
        runManager: CallbackManagerForLLMRun
    ): Promise<string> {
       
        // Configuration & Prompt
        const llmConfig = {
            model_name: this.modelName,
            model_params: this.modelParams,
            model_version: this.modelVersion
        };

        const messages: ChatMessages = [{ role: "user", content: "{{?prompt}}?" }];
        const config: OrchestrationModuleConfig = {
            templating: {
                template: [{ role: "user", content: "{{?prompt}}" }]
            },
            llm: llmConfig
        };

        // Orchestration Client
        const orchestrationClient = new OrchestrationClient(config, {
            resourceGroup: this.resourceGroup
            //deploymentId: this.deploymentId
        });

        // Call the orchestration service.
        const response = await orchestrationClient.chatCompletion({
            inputParams: { prompt }
        });
        // Access the response content.
        return response.getContent();
    }
}

Describe Alternatives

No response

Affected Development Phase

Getting Started

Impact

Inconvenience

Timeline

No response

Additional Context

No response

[jjtang1985]

Thank you very much for raising up this feature request.
Orchestration service contains a list of the following modules in a pipeline:

• LLM access itself via templating module
  • The harmonised API allows using LLMs from different vendors
• Content filtering
• Data Masking
• Grounding (coming soon)
• ...

Therefore, it's more than a service for LLM access.

If we want to make an adapter like what we do for our LangChain package, users would be able to use the original LangChain APIs with SAP GenAI Hub.

However, LangChain knows nothing about other orchestration modules beyonds the LLM access, so we need some API extensions for e.g.,:

• configuring content filtering
• response/error handling for content filtering, with a new API design, which might not look like a LangChain client?

This seems to be a big epic.
I would like to understand more details of the use cases, so we might be able to split the task.

[kay-schmitteckert]

Hey @jjtang1985,

Thanks for your reply. Since we define how to initialize and call the model, we can expose the api of the orchestration model via the constructor of the langchain wrapper and just pass it through.

export class GenerativeAIHubCompletion extends LLM {
    
    orchestrationClient: OrchestrationClient;

    constructor(config: OrchestrationModuleConfig, deploymentConfig?: ResourceGroupConfig | undefined) {
        super();
        this.orchestrationClient = new OrchestrationClient(config, deploymentConfig);
        ...
    }
    ...
}

What are your thoughts on this?

-Kay

[jjtang1985]

Thanks for the example.

Since we define how to initialize and call the model, we can expose the api of the orchestration model via the constructor of the langchain wrapper and just pass it through.

This is a valid design for initialisation and sending the request. We'll then think about the response parsing.

This seems to be a big epic.
I would like to understand more details of the use cases, so we might be able to split the task.

Maybe I should rephrase my original question.
What orchestration modules would you be interested in the first place:

1. llm module and templating module
2. content filtering module
3. data masking module
4. grounding module (will come soon)

I would assume #1 would be our first task, which is the foundation of all LLM accesses.
Would it be already helpful or would you expect that we should support all the orchestration modules in the first place.
I'm asking, because I want to split the task.

Best, Junjie

[kay-schmitteckert]

Yes, I think model access would be a valid starting point

[jjtang1985]

We've just implemented a PoC: #573
The sample code can be found here: https://github.com/SAP/ai-sdk-js/blob/main/sample-code/src/langchain-orchestration.ts

We plan to extend it in Q2

[tomfrenken]

Hi @kay-schmitteckert, in v1.9.0 we've released the orchestration langchain client as part of the langchain package, feel free to try it and give feedback 👍

[g-kevorkian]

Hi colleagues, thanks a lot for releasing this! I am currently part of Kay's team and have started experimenting with the langchain orchestration client. Here's a bit of feedback.

Context: Our current aim in using the orchestration client is to take advantage of grounding including in help.sap.com content, while keeping our existing code-based langgraph agent (at least initially) so we're looking for a drop-in replacement for the AzureOpenAiChatClient.

• Setup: (minor point) The documentation gives a procedure with Postman or curl to create a deployment for orchestration but this is much easier done with the AI launchpad.
• Instantiation : in most scenarios, a resourceGroup will have to be specified. While this is documented somewhere and alluded to the documentation, there is no direct code sample on how to do this so it is not immediately clear from reading the doc. The given code sample does not specify any resource group so the fallback is default and while this might work for you, it most likely will not work for most people making use of the SDK. Fortunately there is a type definition for OrchestrationClient and one can also debug to figure out why deployments are not getting picked up (it is not immediate though).
• Substitution :
  • one of the idea being using a langchain (thus abstracted) client is that we can easily substitute it for another in a langchain/langgraph setup. The fact that the langchain orchestration client requires a template to be specified for the templating module turns out to be impractical in that regard. In our current setup, we instantiate the AzureOpenAiChatClient once, and without any prompt, and use it in different steps with everything being passed as messages. With the orchestration client constraint of having the template specified in addition to the messages being passed as history, the only option I found was to hack the _generate method (which is quite ugly and also quite workflow-specific)
  • a smaller problem when trying to use the OrchestrationClient in our code in place of the AzureOpenAiChatClient is a problem with types. The OrchestrationCallOptions type used by the langchain client references the AI SDK tool type instead of the langchain one. I ended up defining

    type ToolDefinitionLike = ToolDefinition | ChatCompletionTool

    so I could write signatures like:

    private async createLLMAsk<LLM extends BaseLanguageModel<any, BaseLanguageModelCallOptions&{'tools': ToolDefinitionLike[]}>>(llm: LLM, embedding: AzureOpenAiEmbeddingClient) {

    as the assignment to an unparameterised BaseLanguageModelwould not work
• Limitations: the stated limitation of tool messages not being supported is almost a deal breaker (but I suspect it can be circumvented by fiddling with history - haven't tried this yet). One of the reasons for using langchain/langgraph will typically be to create code-based agents that involve tools. I stumbled upon this actually by hitting an exception in the code.
• Gounding (with help.sap.com) : In the absence of streaming (which I understand is still in the works) grounding support becomes the primary motivation for us to use the orchestration client but so far I haven't managed to make it work. Is specific configuration required? I read somewhere about the ext.ai.sap.com/document-grounding label to be put on the resourceGroup. Is that the missing part? Can this be done on an existing resource group?
  Update: I managed to add the label to an existing resource group by doing a PATCH request (this bruno collection was useful - it seems editing the labels is not possible via the AI Launchpad UI). Then I also got to run the help.sap.com query from that collection and do the same via the lang chain orchestration client. I wonder if specifying the label on the resource group is still a requirement (but I will leave it there)

This is early feedback, I may have more later but I hope it is useful