Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Cal.com Copilot #12855

Closed
wants to merge 12 commits into from
Closed

feat: Cal.com Copilot #12855

wants to merge 12 commits into from

Conversation

maxprilutskiy
Copy link
Contributor

@maxprilutskiy maxprilutskiy commented Dec 18, 2023

Fixes #13341

Hey @keithwillcode @baileypumfleet,

As agreed, submitting a PR with what we have done during the hackathon for you to take a look. :)

P.S. Commiting the work as-is: likely has redundancies/inefficiencies/uncovered edge cases (since it's a hackathon PoC).

Cal.com Copilot

Cal.com Copilot helps new users get started with the platform faster and provides a much more personalized first experience, using AI.

Built by @vrcprl and @maxprilutskiy, during the #OSSHack hackathon in NYC.


Our solution improves the onboarding flow and creates a personalized set of events for users, offering more choices to schedule new calls.

Previously, the app randomly generated three events at the end of the user onboarding process. Now, based on the information provided by users during onboarding, the app generates several personalized events.

These include:

  • Title
  • Description
  • Location (in-person, by phone, online), along with specific details such as phone numbers or addresses if provided
  • Meeting durations depend on the nature of the meeting (for example, a dinner with friends might last about 2 hours, while a quick chat usually doesn't take more than 15 minutes)
  • Minimum notice requirements; if the user specifies different notice times for various event types (e.g., 2 days for in-person and 2 hours for online), the app will configure accordingly
  • Automatically identifying personal events (like a call with mom) and hiding them from the list of events visible to business partners

Demo

Check out our demo 😉

Cal.com Copilot Demo Loom

https://www.loom.com/share/3ca21a10254443b9a01bdc90e78d3a5d

What's next

Opportunities for further improvements:

  • Add support for more features (settings) that can be configured based on user input;
  • Use this solution to create a single event based on the user’s input (aka use co-pilot outside of the onboarding flow);
  • Make Copilot configurable, e.g. turn on/off depending on the presence of the openai key;
  • Migrate from openai api to another solution: openai can sometimes be unstable, but there are other alternatives out there;
  • Improve not only the actual loading speed, but also the perceived loading speed via specific design decisions;
  • Add user input validation;
  • Add needed translations;
  • Make the prompts respect the zod schema, by either fine-tuning the LLM model or providing an extensive set of input <> output examples within the LLM prompts themselves;
  • Make the prompts much more robust by testing them against hundreds of potential user inputs;
  • Implement a JSON validator, as there will never be 100% certainty openai returns a valid JSON;
  • Implement fallback mechanism if OpenAI returns invalid response or is down;
  • Ideally, the feature must also be connected to product analytics tool you're using (posthog?), to make sure we arent' releasing this blindly.

Rollout considerations

We recommend rolling this out in phases (e.g. 20 => 60 => 100), to make sure Copilot not just gives new users more reasons to be booked, but those bookings actually happen and the variety of new event types doesn't overwhelm the user.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link

vercel bot commented Dec 18, 2023

@maxprilutskiy is attempting to deploy a commit to the cal Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions bot added the ❗️ .env changes contains changes to env variables label Dec 18, 2023
Copy link
Contributor

github-actions bot commented Dec 18, 2023

Thank you for following the naming conventions! 🙏 Feel free to join our discord and post your PR link.

Copy link
Contributor

📦 Next.js Bundle Analysis for @calcom/web

This analysis was generated by the Next.js Bundle Analysis action. 🤖

⚠️ Global Bundle Size Increased

Page Size (compressed)
global 163.58 KB (🟡 +515 B)
Details

The global bundle is the javascript bundle that loads alongside every page. It is in its own category because its impact is much higher - an increase to its size means that every page on your website loads slower, and a decrease means every page loads faster.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

If you want further insight into what is behind the changes, give @next/bundle-analyzer a try!

Fifty-six Pages Changed Size

The following pages changed size from the code in this PR compared to its base branch:

Page Size (compressed) First Load % of Budget (350 KB)
/[user]/[type] 248.41 KB 411.98 KB 117.71% (🟢 -0.50%)
/[user]/[type]/embed 248.44 KB 412.02 KB 117.72% (🟢 -0.51%)
/apps/[slug]/[...pages] 542.48 KB 706.06 KB 201.73% (🟢 -0.83%)
/apps/categories 245.03 KB 408.61 KB 116.74% (🟢 -0.15%)
/apps/categories/[category] 249.69 KB 413.27 KB 118.08% (🟢 -0.15%)
/auth/forgot-password 101.66 KB 265.24 KB 75.78% (🟡 +0.21%)
/auth/login 119.04 KB 282.62 KB 80.75% (🟡 +0.28%)
/auth/setup 150.23 KB 313.8 KB 89.66% (🟡 +0.29%)
/auth/verify 103.29 KB 266.86 KB 76.25% (🟢 -11.08%)
/availability 382.74 KB 546.32 KB 156.09% (🟢 -11.24%)
/availability/troubleshoot 270.31 KB 433.89 KB 123.97% (🟢 -0.15%)
/bookings/[status] 314.87 KB 478.45 KB 136.70% (🟡 +0.51%)
/d/[link]/[slug] 248.4 KB 411.98 KB 117.71% (🟢 -0.46%)
/enterprise 245.48 KB 409.06 KB 116.88% (🟢 -0.16%)
/event-types 541.71 KB 705.29 KB 201.51% (🟢 -0.47%)
/event-types/[type] 417.68 KB 581.26 KB 166.08% (🟢 -0.52%)
/getting-started/[[...step]] 401.13 KB 564.71 KB 161.35% (🟢 -0.16%)
/more 244.66 KB 408.24 KB 116.64% (🟢 -0.16%)
/org/[orgSlug]/[user]/[type] 248.58 KB 412.16 KB 117.76% (🟢 -0.52%)
/org/[orgSlug]/[user]/[type]/embed 248.61 KB 412.19 KB 117.77% (🟢 -0.52%)
/org/[orgSlug]/team/[slug]/[type] 248.43 KB 412.01 KB 117.72% (🟢 -0.46%)
/settings/admin 251.31 KB 414.89 KB 118.54% (🟢 -0.15%)
/settings/admin/apps 264.49 KB 428.07 KB 122.31% (🟢 -0.15%)
/settings/admin/apps/[category] 264.48 KB 428.06 KB 122.30% (🟢 -0.14%)
/settings/admin/flags 255.06 KB 418.64 KB 119.61% (🟢 -0.16%)
/settings/admin/impersonation 251.57 KB 415.15 KB 118.61% (🟢 -0.15%)
/settings/admin/oAuth 263.32 KB 426.89 KB 121.97% (🟢 -0.15%)
/settings/admin/oAuth/oAuthView 91.05 KB 254.63 KB 72.75% (🟢 -0.14%)
/settings/admin/organizations 252.61 KB 416.18 KB 118.91% (🟢 -0.15%)
/settings/admin/users 253.52 KB 417.1 KB 119.17% (🟢 -0.15%)
/settings/billing 251.52 KB 415.1 KB 118.60% (🟢 -0.15%)
/settings/developer/api-keys 255.92 KB 419.5 KB 119.86% (🟢 -0.16%)
/settings/developer/webhooks 255.88 KB 419.46 KB 119.85% (🟢 -0.15%)
/settings/developer/webhooks/[id] 257.23 KB 420.81 KB 120.23% (🟢 -0.14%)
/settings/developer/webhooks/new 257.28 KB 420.86 KB 120.25% (🟢 -0.15%)
/settings/my-account/calendars 263.13 KB 426.71 KB 121.92% (🟢 -0.15%)
/settings/my-account/conferencing 263.43 KB 427.01 KB 122.00% (🟢 -0.15%)
/settings/organizations/appearance 263.91 KB 427.49 KB 122.14% (🟢 -3.28%)
/settings/organizations/billing 251.55 KB 415.13 KB 118.61% (🟢 -0.16%)
/settings/organizations/members 380.27 KB 543.85 KB 155.39% (🟢 -11.29%)
/settings/organizations/teams/other 252.38 KB 415.96 KB 118.85% (🟢 -0.16%)
/settings/organizations/teams/other/[id]/members 258.38 KB 421.96 KB 120.56% (🟢 -0.15%)
/settings/organizations/teams/other/[id]/profile 456.69 KB 620.27 KB 177.22% (🟢 -0.51%)
/settings/security/impersonation 256.6 KB 420.18 KB 120.05% (🟢 -0.15%)
/settings/security/password 265.38 KB 428.96 KB 122.56% (🟢 -0.14%)
/settings/security/sso 260.55 KB 424.13 KB 121.18% (🟢 -0.45%)
/settings/security/two-factor-auth 260.44 KB 424.02 KB 121.15% (🟢 -0.15%)
/settings/teams 251.05 KB 414.63 KB 118.47% (🟢 -0.16%)
/settings/teams/[id]/billing 251.55 KB 415.13 KB 118.61% (🟢 -0.16%)
/settings/teams/[id]/profile 457.51 KB 621.09 KB 177.45% (🟢 -0.51%)
/settings/teams/[id]/sso 261.07 KB 424.65 KB 121.33% (🟢 -0.45%)
/team/[slug]/[type] 248.4 KB 411.98 KB 117.71% (🟢 -0.46%)
/team/[slug]/[type]/embed 248.43 KB 412.01 KB 117.72% (🟢 -0.46%)
/teams 244.93 KB 408.51 KB 116.72% (🟢 -0.16%)
/video/[uid] 244.52 KB 408.09 KB 116.60% (🟡 +3.48%)
/workflows/[workflow] 396.42 KB 560 KB 160.00% (🟢 -0.28%)
Details

Only the gzipped size is provided here based on an expert tip.

First Load is the size of the global bundle plus the bundle for the individual page. If a user were to show up to your website and land on a given page, the first load size represents the amount of javascript that user would need to download. If next/link is used, subsequent page loads would only need to download that page's bundle (the number in the "Size" column), since the global bundle has already been downloaded.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

The "Budget %" column shows what percentage of your performance budget the First Load total takes up. For example, if your budget was 100kb, and a given page's first load size was 10kb, it would be 10% of your budget. You can also see how much this has increased or decreased compared to the base branch of your PR. If this percentage has increased by 20% or more, there will be a red status indicator applied, indicating that special attention should be given to this. If you see "+/- <0.01%" it means that there was a change in bundle size, but it is a trivial enough amount that it can be ignored.

@keithwillcode keithwillcode added Low priority Created by Linear-GitHub Sync event types Created by Linear-GitHub Sync labels Dec 19, 2023
@keithwillcode keithwillcode changed the title Cal.com Copilot Hackathon PoC feat: Cal.com Copilot Dec 19, 2023
Copy link
Contributor

github-actions bot commented Jan 3, 2024

This PR is being marked as stale due to inactivity.

@github-actions github-actions bot added the Stale label Jan 3, 2024
@keithwillcode keithwillcode requested a review from a team January 8, 2024 15:44
@@ -2,7 +2,7 @@
"typescript.tsdk": "node_modules/typescript/lib",
"editor.formatOnSave": false,
"editor.codeActionsOnSave": {
"source.fixAll.eslint": true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this changed?

})
);
await new Promise((resolve, reject) => {
createEventTypesWithCopilot.mutate(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that we should delete default event types, copilot should be optional no?


let responseContent = llmResponse?.choices?.[0]?.message?.content || "[]";

console.log("responseContent", responseContent);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
console.log("responseContent", responseContent);

@exception
Copy link
Contributor

For this PR to move forward, we'd need the option for AI Copilot or the default event types.
I believe this should also be done in the background (or at least have a notice) as it slows the onboarding process (while AI compute is happening).

cc @keithwillcode @PeerRich

@keithwillcode keithwillcode added this to the v3.9 milestone Jan 10, 2024
@keithwillcode keithwillcode added the community Created by Linear-GitHub Sync label Jan 11, 2024
@keithwillcode keithwillcode requested a review from a team January 20, 2024 14:26
Copy link
Contributor

Hey there, there is a merge conflict, can you take a look?

@github-actions github-actions bot added the 🚨 merge conflict This PR has a merge conflict that has to be addressed label Jan 23, 2024
@keithwillcode keithwillcode marked this pull request as draft January 25, 2024 21:51
@keithwillcode
Copy link
Contributor

Moving back to draft as there are implementation changes we are going to make

Copy link
Contributor

github-actions bot commented Feb 9, 2024

This PR is being marked as stale due to inactivity.

@github-actions github-actions bot added Stale and removed Stale labels Feb 9, 2024
Copy link
Contributor

This PR is being marked as stale due to inactivity.

@github-actions github-actions bot added the Stale label Feb 25, 2024
@github-actions github-actions bot removed the Stale label Mar 27, 2024
@keithwillcode keithwillcode modified the milestones: v3.9, v4.0 Mar 27, 2024
@keithwillcode
Copy link
Contributor

Closing for now until we re-prioritize this

@dosubot dosubot bot removed this from the v4.0 milestone Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community Created by Linear-GitHub Sync ❗️ .env changes contains changes to env variables event types Created by Linear-GitHub Sync Low priority Created by Linear-GitHub Sync 🚨 merge conflict This PR has a merge conflict that has to be addressed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CAL-2982] Cal.com Copilot
4 participants