Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/packages/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -907,7 +907,7 @@ Word-level transcripts (whisper output) are grouped into readable caption cues o
npx hyperframes telemetry status
```

Telemetry collects command names, render performance, render checkpoint/error names, aggregate browser diagnostic counts, browser initialization duration and tween count, aggregate video extraction workload counts (such as extracted frame count and VFR preflight count), example choices, and system info — including a coarse environment fingerprint (OS, kernel string, CPU/memory shape, sandbox runtime such as gVisor or Docker, and the *name* of a coding agent driving the CLI when one is detected, e.g. `claude_code` / `codex` / `cursor`). The agent name is derived from the existence of well-known environment variables; their values are never read. Telemetry redacts local paths and URL query strings from render error/checkpoint messages and does **not** collect project names, video content, environment variable values, or personally identifiable information. Disable with `HYPERFRAMES_NO_TELEMETRY=1` or the command above.
Telemetry collects command names, render performance, render checkpoint/error names, aggregate browser diagnostic counts, browser initialization duration and tween count, aggregate video extraction workload counts (such as extracted frame count and VFR preflight count), example choices, and system info — including a coarse environment fingerprint (OS, kernel string, CPU/memory shape, sandbox runtime such as gVisor or Docker, and the *name* of a coding agent driving the CLI when one is detected, e.g. `claude_code` / `codex` / `cursor`). The agent name is derived from the existence of well-known environment variables; their values are never read. Telemetry redacts local paths and URL query strings from render error/checkpoint messages and does **not** collect project names, video content, or environment variable values. It collects no personally identifiable information until you sign in: when you authenticate with `hyperframes auth login`, your HeyGen account email (or your username, if your account has no email) is linked to your usage so CLI activity can be associated with your account (and your prior anonymous usage is stitched to it). Nothing else personal is collected, and this only happens after you choose to sign in. Disable all telemetry with `HYPERFRAMES_NO_TELEMETRY=1` or the command above.

See [Feedback Collection](/guides/feedback) for how the periodic post-render prompt and Studio feedback bar work, what data they collect, and how to opt out.

Expand Down
40 changes: 40 additions & 0 deletions packages/cli/src/commands/auth/login.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,17 @@ vi.mock("../../auth/index.js", async (orig) => {
return { ...actual, AuthClient: MockAuthClient };
});

// Spy on the telemetry the login flow emits, so we can assert the identity is
// attributed on success. login.ts imports these via a dynamic import of
// telemetry/index.js; the mock intercepts it.
const telemetry = vi.hoisted(() => ({
trackAuthLoginStarted: vi.fn(),
trackAuthLoginCompleted: vi.fn(),
trackAuthLoginFailed: vi.fn(),
identifyUser: vi.fn(),
}));
vi.mock("../../telemetry/index.js", () => telemetry);

const ENV_KEYS = ["HEYGEN_API_KEY", "HYPERFRAMES_API_KEY", "HEYGEN_CONFIG_DIR"] as const;

describe("auth login --api-key rollback", () => {
Expand All @@ -46,6 +57,7 @@ describe("auth login --api-key rollback", () => {
process.env["HEYGEN_CONFIG_DIR"] = dir;
verifyState.reject = false;
verifyState.user = { email: "alice@example.com" };
for (const fn of Object.values(telemetry)) fn.mockClear();
// process.exit throws so we can assert the post-rollback state.
vi.spyOn(process, "exit").mockImplementation(((code?: string | number | null) => {
throw new Error(`process.exit:${code ?? 0}`);
Expand Down Expand Up @@ -158,6 +170,34 @@ describe("auth login --api-key rollback", () => {
expect(onDisk.future_credential).toEqual({ token: "owned_by_other_cli" });
});

it("attributes a successful login to the account email", async () => {
verifyState.user = { email: "alice@example.com", username: "alice" };
await runLogin("hg_goodkey456");
expect(telemetry.identifyUser).toHaveBeenCalledWith("alice@example.com");
expect(telemetry.trackAuthLoginCompleted).toHaveBeenCalledWith("api_key", "alice@example.com");
});

it("falls back to username when the account has no email", async () => {
verifyState.user = { username: "alice" };
await runLogin("hg_goodkey456");
expect(telemetry.identifyUser).toHaveBeenCalledWith("alice");
expect(telemetry.trackAuthLoginCompleted).toHaveBeenCalledWith("api_key", "alice");
});

it("does not identify when the identity probe returns nothing", async () => {
verifyState.user = {}; // verified key, but no identity fields
await runLogin("hg_goodkey456");
expect(telemetry.identifyUser).not.toHaveBeenCalled();
expect(telemetry.trackAuthLoginCompleted).toHaveBeenCalledWith("api_key", undefined);
});

it("records a rejected key as failed and never identifies", async () => {
verifyState.reject = true;
await expect(runLogin("hg_badkey123")).rejects.toThrow(/process\.exit:1/);
expect(telemetry.identifyUser).not.toHaveBeenCalled();
expect(telemetry.trackAuthLoginFailed).toHaveBeenCalledWith("api_key", "rejected");
});

it("preserves an unknown/foreign top-level key across a successful re-login", async () => {
// Cross-CLI invariant end-to-end: a key heygen-cli (or a future
// version) wrote must survive a hyperframes-cli login round-trip.
Expand Down
53 changes: 38 additions & 15 deletions packages/cli/src/commands/auth/login.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,18 +99,14 @@ async function runOAuthLogin(): Promise<void> {

// fallow-ignore-next-line complexity
async function reportIdentity(): Promise<void> {
const { trackAuthLoginCompleted, trackAuthLoginFailed } =
const { trackAuthLoginCompleted, trackAuthLoginFailed, identifyUser } =
await import("../../telemetry/index.js");
const credential = await tryResolveCredential();
if (!credential) {
trackAuthLoginFailed("oauth", "no_credential");
console.error(c.warn("Sign-in completed but no credential was persisted."));
process.exit(1);
}
// A resolvable credential IS the success signal: the tokens are on disk and
// usable. The `/v3/users/me` probe below only fetches a display name, so its
// outcome is cosmetic and does not gate completion.
trackAuthLoginCompleted("oauth");
// Wire the refresh hook here too — a freshly-minted token shouldn't
// need it, but a fast IdP-side rotation (or a misconfigured short
// TTL) shouldn't punish the user with a hard failure when the
Expand All @@ -124,20 +120,42 @@ async function reportIdentity(): Promise<void> {
// `auth status` can show "Logged in as ..." without re-hitting
// /v3/users/me. Best-effort — a persist failure never fails the login.
await persistUserInfo(user);
// Attribute this install to the signed-in account (and stitch its prior
// anonymous usage) before recording completion, so the completed event
// carries the identity. Both no-op under the telemetry opt-out.
const id = identityKey(user);
if (id) identifyUser(id);
trackAuthLoginCompleted("oauth", id);
const identity = userDisplayName(toStoredUserInfo(user)) ?? "(unknown user)";
console.log(c.success(`✓ Signed in as ${identity}.`));
} catch (err) {
// Don't roll back — the OAuth tokens are valid on disk; this is a
// transient verify-side issue. The identity probe failed, so any
// stale user block from a prior login (possibly a DIFFERENT account)
// is cleared so `auth status` can't surface the wrong identity.
// transient verify-side issue. The credential is persisted and usable, so
// the sign-in still COMPLETED; we just have no resolved identity to
// attribute it to. The stale user block from a prior login (possibly a
// DIFFERENT account) is cleared so `auth status` can't surface it.
await clearUserInfoBestEffort();
trackAuthLoginCompleted("oauth");
console.error(
c.warn(`Signed in. Identity check failed (transient): ${(err as Error).message}`),
);
}
}

/**
* The stable key we associate this install with in telemetry after sign-in.
* `/v3/users/me` exposes no opaque user_id, so we key on the HeyGen account
* EMAIL — the canonical account identifier and the reliable join key back to
* billing — falling back to username only when the account exposes no email.
* (Username is NOT a privacy win — HeyGen usernames are frequently email-shaped
* — it is purely a fallback so an emailless account is still attributable.)
* The privacy notice (showTelemetryNotice) and docs/packages/cli.mdx disclose
* both, so keep them in sync with whatever this returns.
*/
function identityKey(user: UserInfo): string | undefined {
return user.email ?? user.username;
}

/** Project the API `/v3/users/me` view onto the on-disk identity block. */
function toStoredUserInfo(user: UserInfo): StoredUserInfo {
const out: StoredUserInfo = {};
Expand Down Expand Up @@ -179,7 +197,7 @@ async function clearUserInfoBestEffort(): Promise<void> {

// fallow-ignore-next-line complexity
async function runApiKeyLogin(inlineKey: string): Promise<void> {
const { trackAuthLoginStarted, trackAuthLoginCompleted, trackAuthLoginFailed } =
const { trackAuthLoginStarted, trackAuthLoginCompleted, trackAuthLoginFailed, identifyUser } =
await import("../../telemetry/index.js");
trackAuthLoginStarted("api_key");

Expand Down Expand Up @@ -218,13 +236,15 @@ async function runApiKeyLogin(inlineKey: string): Promise<void> {
const next: Credentials = { ...previous, api_key: key };
await writeStore(next);

const verifyOk = await verifyAndReport(key);
if (!verifyOk) {
const user = await verifyAndReport(key);
if (!user) {
trackAuthLoginFailed("api_key", "rejected");
await rollback(previous);
process.exit(1);
}
trackAuthLoginCompleted("api_key");
const id = identityKey(user);
if (id) identifyUser(id);
trackAuthLoginCompleted("api_key", id);
}

async function snapshotStore(): Promise<Credentials> {
Expand Down Expand Up @@ -258,8 +278,11 @@ async function rollback(previous: Credentials): Promise<void> {
}
}

// Returns the verified user on success (so the caller can attribute the
// completed sign-in to that identity), or null when the backend rejects the
// key. Other errors propagate.
// fallow-ignore-next-line complexity
async function verifyAndReport(key: string): Promise<boolean> {
async function verifyAndReport(key: string): Promise<UserInfo | null> {
const client = new AuthClient();
try {
const user = await client.getCurrentUser({ type: "api_key", key, source: "file_json" });
Expand All @@ -268,15 +291,15 @@ async function verifyAndReport(key: string): Promise<boolean> {
await persistUserInfo(user);
const identity = userDisplayName(toStoredUserInfo(user)) ?? "(unknown user)";
console.log(c.success(`✓ API key saved. Authenticated as ${identity}.`));
return true;
return user;
} catch (err) {
if (isAuthError(err) && err.code === "UNAUTHENTICATED") {
console.error(
`${c.warn("HeyGen rejected the API key.")}\n` +
` ${c.dim(err.message)}\n` +
`Run ${c.accent("hyperframes auth login --api-key")} again with a valid key.`,
);
return false;
return null;
}
throw err;
}
Expand Down
5 changes: 4 additions & 1 deletion packages/cli/src/telemetry/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,10 @@ export function showTelemetryNotice(): boolean {

console.log();
console.log(` ${c.dim("Hyperframes collects anonymous usage data to improve the tool.")}`);
console.log(` ${c.dim("No personal info, file paths, or content is collected.")}`);
console.log(` ${c.dim("File paths and composition content are never collected.")}`);
console.log(
` ${c.dim("If you sign in to HeyGen, your account (email, or username) is linked to your usage.")}`,
);
console.log();
console.log(` ${c.dim("Disable anytime:")} ${c.accent("hyperframes telemetry disable")}`);
console.log();
Expand Down
27 changes: 24 additions & 3 deletions packages/cli/src/telemetry/events.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ vi.mock("./client.js", () => ({
trackEvent: (...args: unknown[]) => trackEvent(...args),
}));

// identifyUser reads the install anonymousId; pin it so the $identify alias is
// deterministic and the test never touches disk.
vi.mock("./config.js", () => ({
readConfig: () => ({ anonymousId: "anon-test-123", telemetryEnabled: true }),
}));

const {
trackRenderComplete,
trackRenderError,
Expand All @@ -17,6 +23,7 @@ const {
trackAuthLoginStarted,
trackAuthLoginCompleted,
trackAuthLoginFailed,
identifyUser,
} = await import("./events.js");

describe("render telemetry events", () => {
Expand Down Expand Up @@ -285,12 +292,26 @@ describe("auth login telemetry events", () => {
);
});

it("forwards an explicit distinctId to trackEvent for future user-level attribution", () => {
trackAuthLoginCompleted("oauth", "heygen-user-123");
it("forwards an explicit distinctId to trackEvent for user-level attribution", () => {
trackAuthLoginCompleted("oauth", "alice@example.com");
expect(trackEvent).toHaveBeenCalledWith(
"auth_login_completed",
{ method: "oauth" },
"heygen-user-123",
"alice@example.com",
);
});

it("identifyUser emits a $identify alias linking the anon install to the identity", () => {
identifyUser("alice@example.com");
expect(trackEvent).toHaveBeenCalledWith(
"$identify",
{ $anon_distinct_id: "anon-test-123" },
"alice@example.com",
);
});

it("identifyUser is a no-op when there is no identity to attach", () => {
identifyUser("");
expect(trackEvent).not.toHaveBeenCalled();
});
});
13 changes: 13 additions & 0 deletions packages/cli/src/telemetry/events.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { redactTelemetryString, type OutputResolutionIssueKind } from "@hyperframes/core";
import { trackEvent } from "./client.js";
import { readConfig } from "./config.js";

export interface RenderObservabilityTelemetryPayload {
observabilityRenderJobId?: string;
Expand Down Expand Up @@ -381,6 +382,18 @@ export function trackAuthLoginFailed(
trackEvent("auth_login_failed", { method, reason }, distinctId);
}

// Associate this install with the signed-in HeyGen account after a completed
// sign-in. Emits a PostHog `$identify` alias whose `$anon_distinct_id` is the
// install's anonymousId, so events recorded before sign-in stitch to the same
// person instead of stranding as a separate anonymous profile. Routed through
// trackEvent so it shares the opt-out gate and flush path — a no-op when
// telemetry is disabled. `distinctId` is the account email (else username);
// see the privacy notice in showTelemetryNotice and docs/packages/cli.mdx.
export function identifyUser(distinctId: string): void {
if (!distinctId) return;
trackEvent("$identify", { $anon_distinct_id: readConfig().anonymousId }, distinctId);
}

// A render was rejected by the output-resolution/alpha/HDR pre-flight (P1-3)
// before any browser/ffmpeg work. Counts the "caught early" saves on dashboard
// 1783183, distinct from deep render failures. `kind` is the low-cardinality
Expand Down
1 change: 1 addition & 0 deletions packages/cli/src/telemetry/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,6 @@ export {
trackAuthLoginStarted,
trackAuthLoginCompleted,
trackAuthLoginFailed,
identifyUser,
} from "./events.js";
export { getSystemMeta, getShmSizeMb, getFreeDiskMb, bytesToMb } from "./system.js";
Loading