Improve performance adding integrations from editor sidebar #1828

andresgutgon · 2025-10-17T16:07:10Z

What?

Some improvements on connection a integration (tool or trigger)

TODO

Improve loading of third-party integrations by caching the first page
Remove N+1 requests on list integrations by avoiding getting triggers and tools
Add connection to another account on existing integrations
Add invalidate integration cache to backoffice

andresgutgon · 2025-10-17T16:13:59Z

packages/core/src/services/copilot/latte/tools/settings/createIntegration.ts

 const createIntegrationLatte = defineLatteTool(
  async ({ name, app: appName }, { workspace, user }) => {
-    const appResult = await getApp({ name: appName })
+    const appResult = await getApp({ name: appName, withConfig: true })


@csansoon is this the one that needs config

We use to get the tools here before, so yes

andresgutgon · 2025-10-17T16:14:44Z

packages/core/src/services/copilot/latte/tools/triggers/fetchFullConfigSchema.ts

 }): PromisedResult<ConfigurablePropWithRemoteOptions[]> {
  try {
+    // TODO: Cache this on Redis to improve performance on tool retrieval
    const { data: component } = await pipedream.components.retrieve(componentId)


This could also be cached in Redis to improve tool retrieval at request time. Not doing in this PR tho

andresgutgon · 2025-10-17T16:15:34Z

packages/core/src/services/history/revertDocumentVersion.ts

+          title: `Revert changes for "${oldDocumentPath}"`,
+          description: `Reverted changes of "${oldDocumentPath}" made in version ${changedCommit.title}`,
+        },
+      })


I dont know what happened with Prettier @geclos

andresgutgon · 2025-10-17T17:52:02Z

...ocumentUuid]/_components/DocumentEditor/Editor/SidebarArea/Tools/ActiveIntegration/index.tsx

      : undefined
-  const { data: pipedreamApp } = usePipedreamApp(appNameSlug)
+  const { data: pipedreamApp } = usePipedreamApp(appNameSlug, {
+    withConfig: false,


This makes the data retrieved from the integration provider (Pie...) leaner. They have a large property with a lot of JSON that we don't need in most cases when using listing tools.

andresgutgon · 2025-10-17T17:53:30Z

...ts/DocumentEditor/Editor/SidebarArea/Tools/ConnectToolsModal/ConnectPipedreamModal/index.tsx


-        <PipedreamAppCard app={app} />
+        {!isLoadingAppData && !hasTools ? (
+          <Alert


This is the tradeoff with not fetching the N+1 integrations + tools all the time. We can't do the filtering by tool or trigger so there will be false positives in the UI so we let users know this integration does not have tools in this case. Sorry

uff... That sounds frustrating.

Pipedream actually had a filtering option in their sdk before updating it. They FORGOT to add it back in the new version and just created an issue.

We can use that option when is ready in the SDK

andresgutgon · 2025-10-17T17:54:34Z

...ocumentUuid]/_components/DocumentEditor/Editor/SidebarArea/Tools/ConnectToolsModal/index.tsx

-  toolCount?: number
-}) {
-  if (type === 'UnConnectedPipedreamApp') {
-    const tools = pluralize(toolCount ?? 0, 'available tool', 'available tools')


No counter of tools in the connect integration modal. This is the other tradeoff. Happy to do it because we get faster first listing of tools.

andresgutgon · 2025-10-17T17:56:08Z

...ocumentUuid]/_components/DocumentEditor/Editor/SidebarArea/Tools/ConnectToolsModal/index.tsx

-            },
-          }) satisfies SearchableOptionItem<ToolType>,
-      )
+    const availableApps: SearchableOptionItem<ToolType>[] = pipedreamApps.map(


Now we list all integrations before we filter out already connected integrations. No need. Users will see, for example, Slack on their already connected and they can connect from a new account, or they can click on connect in the non-connected integrations. This is better

Why is this better? Please explain

Is a product decision. Is better because you can see always all the apps available even the ones you already have connected. This way you have 2 places to make the connection.

That can add confusion to the user, dont you think?

andresgutgon · 2025-10-17T17:56:47Z

...mmitUuid]/documents/[documentUuid]/_components/DocumentTabs/DocumentTriggers/Modal/index.tsx

  const { data: pipedreamData, isLoading: loadingComponents } = usePipedreamApp(
    (integrationForTrigger as PipedreamIntegration | undefined)?.configuration
      .appName,
+    { withConfig: true },


I think this fat trigger is needed for the configuration panel on the triggers modal

andresgutgon · 2025-10-17T17:56:57Z

...ments/[documentUuid]/_components/DocumentTabs/DocumentTriggers/Modal/TriggerTypeSelector.tsx

-  const { data, isLoading } = usePipedreamApp(integration.configuration.appName)
+  const { data, isLoading } = usePipedreamApp(
+    integration.configuration.appName,
+    { withConfig: true },


I think this fat trigger is needed for the configuration panel on the triggers modal

andresgutgon · 2025-10-17T17:57:30Z

...Uuid]/_components/DocumentTabs/DocumentTriggers/Settings/IntegrationTriggers/TriggerList.tsx

  trigger: DocumentTrigger<DocumentTriggerType.Integration>
  integration: PipedreamIntegration
-  component?: PipedreamComponent
+  component?: LightPipedreamComponent<T>


When withConfig: false we have a new type without the fat JSON property

learningbizz · 2025-10-20T07:52:31Z

apps/web/src/app/(admin)/backoffice/integrations/_components/InvalidateAppCache.tsx

+    { nameSlug: string; timestamp: Date }[]
+  >([])
+
+  const { execute: searchApps, isPending: isSearching } = useLatitudeAction(


nitpick: not a fan of leaving actions within a component, I suggest we create a store for all of the actions for invalidating the cache for cleaner code.

This is a one direction only action. No need to use a store

if it's a single instance it's fine, when it's time to DRY moving to stores is correct

learningbizz · 2025-10-20T07:55:41Z

apps/web/src/app/(admin)/backoffice/integrations/_components/InvalidateAppCache.tsx

+  const handleInvalidate = async (nameSlug: string) => {
+    const [result, _error] = await invalidateApp({ nameSlug })
+    if (result) {
+      setInvalidatedApps((prev) => [


This could also go within the store, including the useEffects, and it would be cleaner than poluting a component with all this logic

learningbizz · 2025-10-20T08:02:23Z

apps/web/src/app/api/integrations/pipedream/apps/route.ts

-    const { searchParams } = new URL(request.url)
-    const query = searchParams.get('query') || undefined
-    const cursor = searchParams.get('cursor') || undefined
-    const withTriggers = searchParams.get('withTriggers')


why are we removing here the trigger and tools?

This is the N+1 queries that we're not doing now when fetching the applications. We're not fetching all their triggers and tools

learningbizz · 2025-10-20T08:13:36Z

packages/core/src/constants.ts

 }

+// Light version of AppDto without configurableProps for reduced payload
+export type LightPipedreamComponent<T extends PipedreamComponentType> = Omit<


Light is not comprehensible for me, would rename to more literal AppWithoutConfig

learningbizz · 2025-10-20T08:14:41Z

docker-compose.local.yml

@@ -1,5 +1,3 @@
-version: '3.9'


This should not be removed IMO, could potentially break the docker compose although I never use it

We don't have in the other docker compose
https://github.com/latitude-dev/latitude-llm/blob/main/docker-compose.yml

And docker was complaining that version is legacy

learningbizz · 2025-10-20T08:36:07Z

packages/core/src/services/integrations/pipedream/cache/apps.ts

+  return getOrSet<CachedAppsResponse>(cacheKey, fetchApps, ONE_DAY_IN_SECONDS)
+}
+
+export type CachedAppResponse<C extends boolean> = C extends true


This type is also used in the frontend store, why duplicate it? We can add it in constants to share

csansoon · 2025-10-20T08:52:25Z

apps/web/src/stores/pipedreamApp.ts

+type MaybeAppDto<C extends boolean> = C extends true
+  ? AppDto
+  : C extends false
+    ? LightAppDto
+    : never
+type AppResponse<C extends boolean> =
+  | { data: MaybeAppDto<C>; ok: true }


Could we just update AppDto to have tools and triggers optional? That way we will have to check for them in the UI and decide whether to show each UI depending on if the attribute is there or not.
Also, this way we don't have multiple types for the same thing, it is confusing: App, AppDto, LightAppDto, MaybeAppDto<C>...

andresgutgon force-pushed the fix/improve-performance-adding-integrations-from-editor-sidebar branch from ad97287 to 9406650 Compare October 17, 2025 16:10

andresgutgon commented Oct 17, 2025

View reviewed changes

andresgutgon requested review from csansoon and geclos October 17, 2025 16:22

andresgutgon force-pushed the fix/improve-performance-adding-integrations-from-editor-sidebar branch 2 times, most recently from de1ccb6 to 71471c8 Compare October 17, 2025 17:49

andresgutgon commented Oct 17, 2025

View reviewed changes

andresgutgon force-pushed the fix/improve-performance-adding-integrations-from-editor-sidebar branch from 71471c8 to 8ed554f Compare October 18, 2025 10:57

Improve performance adding integrations from editor sidebar

5243ce7

andresgutgon force-pushed the fix/improve-performance-adding-integrations-from-editor-sidebar branch from 8ed554f to 5243ce7 Compare October 18, 2025 11:53

learningbizz reviewed Oct 20, 2025

View reviewed changes

geclos approved these changes Oct 20, 2025

View reviewed changes

learningbizz approved these changes Oct 20, 2025

View reviewed changes

csansoon reviewed Oct 20, 2025

View reviewed changes

csansoon approved these changes Oct 20, 2025

View reviewed changes

andresgutgon merged commit 7a7c6ab into main Oct 20, 2025
7 checks passed

andresgutgon deleted the fix/improve-performance-adding-integrations-from-editor-sidebar branch October 20, 2025 09:01

github-actions bot locked and limited conversation to collaborators Oct 20, 2025

Improve performance adding integrations from editor sidebar #1828

Improve performance adding integrations from editor sidebar #1828

Uh oh!

Conversation

andresgutgon commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

TODO

Uh oh!

Choose a reason for hiding this comment

Uh oh!

learningbizz Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

learningbizz Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

learningbizz Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

andresgutgon commented Oct 17, 2025 •

edited

Loading

learningbizz Oct 20, 2025 •

edited

Loading

learningbizz Oct 20, 2025 •

edited

Loading

learningbizz Oct 20, 2025 •

edited

Loading