diff --git a/apps/penpal/ERD.md b/apps/penpal/ERD.md index ee1185986..44bd54360 100644 --- a/apps/penpal/ERD.md +++ b/apps/penpal/ERD.md @@ -85,7 +85,7 @@ see-also: - **E-PENPAL-SRC-MANUAL**: The `manual` source type is used for user-added sources (via `penpal open` CLI or the add-workspace/project modal). Directory sources create "tree" type entries; individual file sources create "files" type entries. `GroupFiles()` generates directory headings (`Dir`, `ShowDir` fields on `FileInfo`) for subdirectory boundaries. Configuration is persisted in `config.json` under `ProjectSources` (workspace projects) or inline with the `ProjectConfig` entry (standalone projects). ← [P-PENPAL-CLI-OPEN](PRODUCT.md#P-PENPAL-CLI-OPEN), [P-PENPAL-STANDALONE](PRODUCT.md#P-PENPAL-STANDALONE) -- **E-PENPAL-SRC-ALL-MD**: An "All Markdown" virtual source is always present in every project's sidebar. It shows all `.md` files in the project directory tree, organized by directory structure, regardless of whether they also appear in a typed source. No badge, no deduplication against other sources. The `Auto` field is `true` — the frontend uses this field to control whether the "Remove from Penpal" context menu item appears, and virtual sources cannot be removed. Implemented as a synthetic source group appended after the typed source groups in the project files response. +- **E-PENPAL-SRC-ALL-MD**: An "All Markdown" source is always present in every project. Registered as a `SourceType` with `GroupFiles` returning a single "All Markdown" group, `ShowDirHeadings: true`, and `SkipDirs` for `node_modules`, `.hg`, `.svn`. Always appended by `DetectSources()` after the detection loop — not conditionally detected. The `Auto` field is `true` — the frontend uses this to prevent removal. During scanning, `__all_markdown__` bypasses the per-source dedup (`seen` map) so it claims every `.md` file regardless of other sources. `AllFiles()` deduplicates by project+path, preferring typed-source entries over `__all_markdown__`, so the recent-files list and activity seeding see each file exactly once. The `handleAddSource` conflict check skips `__all_markdown__` so it doesn't block manual file additions. ← [P-PENPAL-SRC-ALL-MD](PRODUCT.md#P-PENPAL-SRC-ALL-MD) - **E-PENPAL-WORKTREE-DISCOVERY**: Worktrees are discovered by parsing `git worktree list --porcelain` output. Each worktree gets a name, path, branch, and `IsMain` flag. The `refs/heads/` prefix is stripped from branch names. @@ -101,7 +101,7 @@ see-also: - **E-PENPAL-CACHE**: An in-memory cache (`sync.RWMutex`-protected) holds the full project list and per-project file lists. `RefreshProject()` walks the filesystem; `RefreshAllProjects()` runs in parallel with no concurrency limit. `RescanWith()` replaces the project list while preserving git enrichment. ← [P-PENPAL-PROJECT-FILE-TREE](PRODUCT.md#P-PENPAL-PROJECT-FILE-TREE) -- **E-PENPAL-SCAN**: `scanProjectSources()` walks `RootPath` recursively for tree sources, skipping `.git`-file directories (nested worktrees), gitignored directories (via `git check-ignore`), source-type `SkipDirs`, and non-`.md` files. Gitignore checking is initialized once per scan via `newGitIgnoreChecker(projectPath)`, which detects whether the project is a git repo; non-git projects skip the gitignore check gracefully. On write or read failure (partial 4-field response), the checker disables itself (`isGitRepo=false`) to prevent permanent stream desync. The source's own `rootPath` is never checked against gitignore (the `path != rootPath` guard ensures registered sources always scan). Files returning `""` from `ClassifyFile()` are hidden. Files are de-duplicated by project-relative path (first source wins) and sorted by `ModTime` descending. +- **E-PENPAL-SCAN**: `scanProjectSources()` walks `RootPath` recursively for tree sources, skipping `.git`-file directories (nested worktrees), gitignored directories (via `git check-ignore`), source-type `SkipDirs`, and non-`.md` files. Gitignore checking is initialized once per scan via `newGitIgnoreChecker(projectPath)`, which detects whether the project is a git repo; non-git projects skip the gitignore check gracefully. On write or read failure (partial 4-field response), the checker disables itself (`isGitRepo=false`) to prevent permanent stream desync. The source's own `rootPath` is never checked against gitignore (the `path != rootPath` guard ensures registered sources always scan). Files returning `""` from `ClassifyFile()` are hidden. Files are de-duplicated by project-relative path (first source wins) and sorted by `ModTime` descending. `EnsureProjectScanned()` is the lazy-scan entry point — it uses write-lock gating (`projectScanned` set under `mu.Lock` before scanning) to prevent concurrent requests from triggering duplicate filesystem walks. `projectHasAnyMarkdown()` performs a cheap startup check that aligns with the full scan: it uses the same gitignore checking, skips `.git`, `node_modules`, `.hg`, `.svn`, and nested worktree directories, and stops at the first `.md` file found. ← [P-PENPAL-PROJECT-FILE-TREE](PRODUCT.md#P-PENPAL-PROJECT-FILE-TREE), [P-PENPAL-FILE-TYPES](PRODUCT.md#P-PENPAL-FILE-TYPES), [P-PENPAL-SRC-DEDUP](PRODUCT.md#P-PENPAL-SRC-DEDUP), [P-PENPAL-SRC-GITIGNORE](PRODUCT.md#P-PENPAL-SRC-GITIGNORE) - **E-PENPAL-TITLE-EXTRACT**: `EnrichTitles()` reads the first 20 lines of each file to extract H1 headings. Titles are cached and shown as the primary display name when present. @@ -351,7 +351,7 @@ see-also: ## Search -- **E-PENPAL-SEARCH**: `GET /api/search?q=` handled by `handleAPISearch()`. Iterates `s.cache.Projects()`, doing substring match on project names for the "Projects" section (`matchingProjects`). For file matches, walks each source's `RootPath` via `filepath.Walk`, skipping `SkipDirs` and non-`.md` files. Filenames checked via `strings.Contains(ToLower(...), query)` (flagged `nameMatch: true`); content checked via `bufio.Scanner` line-by-line (stops at first match). Files returning `""` from `ClassifyFile()` are excluded. Results per project sorted name-matches-first then alphabetical; capped at 100 total files with early `break` on project iteration. Response struct: `apiSearchResponse{Query, MatchingProjects, ProjectResults, TotalFiles}`. Frontend `SearchPage` component at route `/search` calls `api.search(query)` and renders project cards, grouped file rows with `FileTypeBadge`, and a "name" `` badge for name matches. +- **E-PENPAL-SEARCH**: `GET /api/search?q=` handled by `handleAPISearch()`. Iterates `s.cache.Projects()`, doing substring match on project names for the "Projects" section (`matchingProjects`). For file matches, walks each source's `RootPath` via `filepath.Walk`, skipping `.git` directories, nested worktree/submodule directories (`.git` file), `SkipDirs`, and non-`.md` files. Filenames checked via `strings.Contains(ToLower(...), query)` (flagged `nameMatch: true`); content checked via `bufio.Scanner` line-by-line (stops at first match). Files returning `""` from `ClassifyFile()` are excluded. Results per project sorted name-matches-first then alphabetical; capped at 100 total files with early `break` on project iteration. Response struct: `apiSearchResponse{Query, MatchingProjects, ProjectResults, TotalFiles}`. Frontend `SearchPage` component at route `/search` calls `api.search(query)` and renders project cards, grouped file rows with `FileTypeBadge`, and a "name" `` badge for name matches. ← [P-PENPAL-SEARCH](PRODUCT.md#P-PENPAL-SEARCH) --- @@ -368,7 +368,7 @@ see-also: ## Activity Tracking -- **E-PENPAL-ACTIVITY**: In-memory `activity.Tracker` stores one event per file (keyed by project + path). Event types: `viewed`, `modified`, `created`, `comment`, `published`. `Record()` always overwrites; `RecordAt()` (for seeding from mtime) does not overwrite. +- **E-PENPAL-ACTIVITY**: In-memory `activity.Tracker` stores one event per file (keyed by project + path). Event types: `viewed`, `modified`, `created`, `comment`, `published`. `Record()` always overwrites; `RecordAt()` (for seeding from mtime) does not overwrite. Activity is seeded per-project when `EnsureProjectScanned()` performs the first lazy scan — each file's `ModTime` is recorded via `RecordAt()`, excluding `__all_markdown__` duplicates. This populates `/api/recent` progressively as projects are opened. ← [P-PENPAL-GLOBAL-RECENT](PRODUCT.md#P-PENPAL-GLOBAL-RECENT), [P-PENPAL-PROJECT-RECENT](PRODUCT.md#P-PENPAL-PROJECT-RECENT) - **E-PENPAL-RECENT-PAGE**: `GET /api/recent` returns up to 50 recently active files across all projects, sorted by most recent activity. Frontend `RecentPage` renders two-line file rows with filename, workspace / project context, and relative timestamp. Clicking navigates the current tab to the file. diff --git a/apps/penpal/TESTING.md b/apps/penpal/TESTING.md index 916b4dc62..721b0fab0 100644 --- a/apps/penpal/TESTING.md +++ b/apps/penpal/TESTING.md @@ -64,7 +64,7 @@ see-also: | Source Types — anchors (P-PENPAL-SRC-ANCHORS, SRC-ANCHORS-GROUP, SRC-ANCHORS-NESTED) | discovery_test.go (TestClassifyAnchorsFile, TestGroupAnchorsPaths, TestGroupAnchorsPaths_MarkerOnlyModule, TestAnchorsFileOrder, TestAnchorsRequireSibling) | — | — | — | | Source Types — claude-plans (P-PENPAL-SRC-CLAUDE-PLANS) | — | — | — | — | | Source Types — manual (P-PENPAL-SRC-MANUAL) | — | — | grouping_test.go (TestBuildFileGroups_ManualSourceDirHeadings) | — | -| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go | — | — | — | +| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go (TestCheckAllProjectsHasFiles, TestProjectHasAnyMarkdown_SkipsGitignored, TestProjectHasAnyMarkdown_SkipsVCSDirs, TestAllFiles_DeduplicatesAllMarkdown, TestEnsureProjectScanned_NoDuplicateScans) | — | — | — | | Worktree Support (P-PENPAL-WORKTREE) | discovery/worktree_test.go, cache/worktree_test.go | Layout.test.tsx | worktree_test.go (API + MCP) | — | | Worktree Dropdown (P-PENPAL-PROJECT-WORKTREE-DROPDOWN) | — | Layout.test.tsx | — | — | | Git Integration (P-PENPAL-GIT-INFO) | — | — | — | — | @@ -89,7 +89,7 @@ see-also: | Search (P-PENPAL-SEARCH) | — | SearchPage.test.tsx | — | react-app.spec.ts | | Recent Files (P-PENPAL-RECENT) | activity_test.go | RecentPage.test.tsx | integration_test.go | — | | CLI Open (P-PENPAL-CLI-OPEN) | — | — | api_manage_test.go | cli-open.spec.ts | -| Source Management (P-PENPAL-ADD-SOURCE, REMOVE-SOURCE) | — | — | api_manage_test.go | — | +| Source Management (P-PENPAL-ADD-SOURCE, REMOVE-SOURCE) | — | — | api_manage_test.go (TestAPISources_AddFileNotBlockedByAllMarkdown) | — | | Real-Time Updates (P-PENPAL-REALTIME, FOCUS) | watcher_test.go | useSSE.test.ts | api_focus_test.go | — | | Config & Migration (E-PENPAL-CONFIG) | config_test.go, migrate_test.go | — | — | — | | Install Tools (P-PENPAL-INSTALL) | — | InstallStartup.test.tsx, InstallToolsModal.test.tsx | install_test.go | — | diff --git a/apps/penpal/frontend/src/components/HomeSidebar.tsx b/apps/penpal/frontend/src/components/HomeSidebar.tsx index f32c07c72..a9f4d0c32 100644 --- a/apps/penpal/frontend/src/components/HomeSidebar.tsx +++ b/apps/penpal/frontend/src/components/HomeSidebar.tsx @@ -144,7 +144,7 @@ export default function HomeSidebar({ return (
{ if (hasWorktrees) { onToggleWorktreeProject(p.qualifiedName); @@ -201,7 +201,7 @@ export default function HomeSidebar({ onShowContextMenu(e, [ { label: 'Close project', className: 'menu-muted', onClick: () => onCloseStandaloneProject(p) }, ])} diff --git a/apps/penpal/frontend/src/components/Layout.close-tab.test.tsx b/apps/penpal/frontend/src/components/Layout.close-tab.test.tsx index ccbb2d908..f66bc8fc4 100644 --- a/apps/penpal/frontend/src/components/Layout.close-tab.test.tsx +++ b/apps/penpal/frontend/src/components/Layout.close-tab.test.tsx @@ -51,8 +51,7 @@ vi.mock('../api', () => ({ qualifiedName: 'ws1/project-a', workspace: 'ws1', origin: 'workspace', - badges: [], - fileCount: 5, + hasFiles: true, lastModified: '2026-01-01T00:00:00Z', }, ]), diff --git a/apps/penpal/frontend/src/components/Layout.external-links.test.tsx b/apps/penpal/frontend/src/components/Layout.external-links.test.tsx index b1a11d025..94cfdfa2e 100644 --- a/apps/penpal/frontend/src/components/Layout.external-links.test.tsx +++ b/apps/penpal/frontend/src/components/Layout.external-links.test.tsx @@ -37,8 +37,7 @@ vi.mock('../api', () => ({ qualifiedName: 'ws1/project-a', workspace: 'ws1', origin: 'workspace', - badges: [], - fileCount: 5, + hasFiles: true, lastModified: '2026-01-01T00:00:00Z', }, ]), diff --git a/apps/penpal/frontend/src/components/Layout.test.tsx b/apps/penpal/frontend/src/components/Layout.test.tsx index 38653140a..7a02a063a 100644 --- a/apps/penpal/frontend/src/components/Layout.test.tsx +++ b/apps/penpal/frontend/src/components/Layout.test.tsx @@ -29,8 +29,8 @@ vi.mock('../api', () => ({ qualifiedName: 'ws1/project-a', workspace: 'ws1', origin: 'workspace', - badges: [], - fileCount: 5, + + hasFiles: true, lastModified: '2026-01-01T00:00:00Z', }, ]), @@ -541,8 +541,8 @@ describe('Layout', () => { qualifiedName: 'ws1/project-a', workspace: 'ws1', origin: 'workspace', - badges: [], - fileCount: 5, + + hasFiles: true, lastModified: '2026-01-01T00:00:00Z', worktrees: [ { name: 'main', path: '/tmp/main', branch: 'main', isMain: true }, @@ -581,8 +581,8 @@ describe('Layout', () => { qualifiedName: 'ws1/project-a', workspace: 'ws1', origin: 'workspace', - badges: [], - fileCount: 5, + + hasFiles: true, lastModified: '2026-01-01T00:00:00Z', worktrees: [ { name: 'main', path: '/tmp/main', branch: 'main', isMain: true }, @@ -642,8 +642,8 @@ describe('Layout', () => { workspace: '', projectPath: '/tmp/bravo', origin: 'standalone' as const, - badges: [], - fileCount: 3, + + hasFiles: true, lastModified: '2026-01-02T00:00:00Z', }, { @@ -652,8 +652,8 @@ describe('Layout', () => { workspace: '', projectPath: '/tmp/alpha', origin: 'standalone' as const, - badges: [], - fileCount: 5, + + hasFiles: true, lastModified: '2026-01-01T00:00:00Z', }, ]); @@ -696,8 +696,8 @@ describe('Layout', () => { workspace: '', projectPath: '/tmp/has-files', origin: 'standalone' as const, - badges: [], - fileCount: 5, + + hasFiles: true, lastModified: '2026-01-01T00:00:00Z', }, { @@ -706,8 +706,8 @@ describe('Layout', () => { workspace: '', projectPath: '/tmp/empty-project', origin: 'standalone' as const, - badges: [], - fileCount: 0, + + hasFiles: false, lastModified: '2026-01-01T00:00:00Z', }, ]); diff --git a/apps/penpal/frontend/src/components/Layout.tsx b/apps/penpal/frontend/src/components/Layout.tsx index a3ad6e31c..7e6b24ebd 100644 --- a/apps/penpal/frontend/src/components/Layout.tsx +++ b/apps/penpal/frontend/src/components/Layout.tsx @@ -327,13 +327,13 @@ export default function Layout() { ); // E-PENPAL-SORT: shared comparator — empty projects last, then alpha or API order. const projectSort = useCallback((a: APIProject, b: APIProject) => { - if ((a.fileCount > 0) !== (b.fileCount > 0)) return b.fileCount > 0 ? 1 : -1; + if (a.hasFiles !== b.hasFiles) return b.hasFiles ? 1 : -1; if (sortOrder === 'alpha') return a.name.localeCompare(b.name); return 0; }, [sortOrder]); // E-PENPAL-VIEW-OPTIONS: filter empty projects when showEmpty is false - const filterEmpty = useCallback((p: APIProject) => showEmpty || p.fileCount > 0, [showEmpty]); + const filterEmpty = useCallback((p: APIProject) => showEmpty || p.hasFiles, [showEmpty]); const standaloneProjects = useMemo(() => { return projects.filter((p) => p.origin === 'standalone').filter(filterEmpty).sort(projectSort); diff --git a/apps/penpal/frontend/src/pages/FilePage.test.tsx b/apps/penpal/frontend/src/pages/FilePage.test.tsx index d57fda6d8..734f08dae 100644 --- a/apps/penpal/frontend/src/pages/FilePage.test.tsx +++ b/apps/penpal/frontend/src/pages/FilePage.test.tsx @@ -37,7 +37,7 @@ vi.mock('../components/MermaidRenderer', () => ({ function LayoutWrapper() { const ctx: LayoutContext = { setHeadings: vi.fn(), - projects: [{ name: 'proj', qualifiedName: 'ws/proj', workspace: 'ws', origin: 'workspace' as const, badges: [], fileCount: 1, lastModified: '', projectPath: '/tmp/proj' }], + projects: [{ name: 'proj', qualifiedName: 'ws/proj', workspace: 'ws', origin: 'workspace' as const, hasFiles: true, lastModified: '', projectPath: '/tmp/proj' }], }; return ; } diff --git a/apps/penpal/frontend/src/types.ts b/apps/penpal/frontend/src/types.ts index 91386bf9e..ee5dc8742 100644 --- a/apps/penpal/frontend/src/types.ts +++ b/apps/penpal/frontend/src/types.ts @@ -1,13 +1,5 @@ // Types matching Go JSON API responses -export interface APIBadge { - text: string; - color: string; - bg: string; - activeBg?: string; - activeColor?: string; -} - export interface APIWorktree { name: string; path: string; @@ -22,10 +14,9 @@ export interface APIProject { workspacePath?: string; projectPath: string; origin: 'workspace' | 'standalone'; - badges: APIBadge[]; branch?: string; dirty?: boolean; - fileCount: number; + hasFiles: boolean; lastModified: string; agentConnected?: boolean; agentRunning?: boolean; @@ -147,7 +138,7 @@ export interface ReviewGroup { } export interface ProjectInfo { - fileCount: number; + hasFiles: boolean; dirty: boolean; unpushedCommits: number; } diff --git a/apps/penpal/internal/cache/cache.go b/apps/penpal/internal/cache/cache.go index 2122261c9..a784dbea2 100644 --- a/apps/penpal/internal/cache/cache.go +++ b/apps/penpal/internal/cache/cache.go @@ -3,7 +3,9 @@ package cache import ( "bufio" "bytes" + "errors" "io" + "io/fs" "os" "os/exec" "path/filepath" @@ -129,12 +131,15 @@ type Cache struct { projects []discovery.Project // projectFiles maps project name to its file list projectFiles map[string][]FileInfo + // projectScanned tracks which projects have had a full file scan + projectScanned map[string]bool } // New creates a new cache func New() *Cache { return &Cache{ - projectFiles: make(map[string][]FileInfo), + projectFiles: make(map[string][]FileInfo), + projectScanned: make(map[string]bool), } } @@ -262,6 +267,7 @@ func (c *Cache) SetProjectFiles(projectName string, files []FileInfo) { c.mu.Lock() defer c.mu.Unlock() c.projectFiles[projectName] = files + c.projectScanned[projectName] = true } // ProjectFiles returns the cached file list for a project @@ -276,14 +282,35 @@ func (c *Cache) ProjectFiles(projectName string) []FileInfo { return nil } -// AllFiles returns all files across all projects, sorted by modification time +// AllFiles returns all files across all projects, sorted by modification time. +// Files are deduplicated by project+path: when a file appears in both a typed +// source and __all_markdown__, only the typed-source entry is kept. +// E-PENPAL-SRC-ALL-MD: dedup prevents __all_markdown__ from doubling the list. func (c *Cache) AllFiles(limit int) []FileInfo { c.mu.RLock() defer c.mu.RUnlock() - var all []FileInfo + type fileKey struct { + project, path string + } + best := make(map[fileKey]FileInfo) for _, files := range c.projectFiles { - all = append(all, files...) + for _, f := range files { + k := fileKey{f.Project, f.FullPath} + if existing, ok := best[k]; ok { + // Prefer typed-source entry over __all_markdown__ + if existing.Source == "__all_markdown__" && f.Source != "__all_markdown__" { + best[k] = f + } + } else { + best[k] = f + } + } + } + + all := make([]FileInfo, 0, len(best)) + for _, f := range best { + all = append(all, f) } sort.Slice(all, func(i, j int) bool { @@ -315,6 +342,25 @@ func (c *Cache) FindFile(projectName, filePath string) *FileInfo { return nil } +// EnsureProjectScanned triggers a full file scan for a project if it hasn't +// been scanned yet. Returns true if a scan was actually performed (first call +// for this project). This is the lazy-scan entry point — called when a user +// first opens a project, rather than eagerly at startup. +// E-PENPAL-SCAN: lazy scan with write-lock gating to prevent duplicate walks. +func (c *Cache) EnsureProjectScanned(projectName string) bool { + c.mu.Lock() + if c.projectScanned[projectName] { + c.mu.Unlock() + return false + } + // Mark as in-progress under write lock to prevent concurrent scans. + c.projectScanned[projectName] = true + c.mu.Unlock() + + c.RefreshProject(projectName) + return true +} + // RefreshProject rescans a single project's files across all its sources. // projectName should be the qualified name (e.g., "Development/penpal"). // E-PENPAL-CACHE: walks filesystem and updates per-project file list and metadata. @@ -330,9 +376,10 @@ func (c *Cache) RefreshProject(projectName string) { // Update project metadata c.mu.Lock() defer c.mu.Unlock() + c.projectScanned[projectName] = true for i := range c.projects { if c.projects[i].QualifiedName() == projectName { - c.projects[i].FileCount = len(files) + c.projects[i].HasFiles = len(files) > 0 if len(files) > 0 { c.projects[i].LastModified = files[0].ModTime } @@ -356,6 +403,73 @@ func (c *Cache) RefreshAllProjects() { wg.Wait() } +// errFoundMarkdown is a sentinel used to short-circuit filepath.WalkDir +// once a .md file is found. +var errFoundMarkdown = errors.New("found markdown") + +// CheckAllProjectsHasFiles does a cheap per-project check to set HasFiles +// without doing a full file scan. For each project, it walks the project +// root and stops as soon as it finds any .md file. +// E-PENPAL-SCAN: lightweight startup check — no file list is built. +func (c *Cache) CheckAllProjectsHasFiles() { + projects := c.Projects() + var wg sync.WaitGroup + for _, p := range projects { + wg.Add(1) + go func(p discovery.Project) { + defer wg.Done() + found := projectHasAnyMarkdown(p.Path) + c.mu.Lock() + for i := range c.projects { + if c.projects[i].QualifiedName() == p.QualifiedName() { + c.projects[i].HasFiles = found + break + } + } + c.mu.Unlock() + }(p) + } + wg.Wait() +} + +// projectHasAnyMarkdown walks the directory tree and returns true as soon as +// it finds any .md file. Aligns skip behavior with scanProjectSources: skips +// .git, node_modules, .hg, .svn, nested worktrees/submodules, and gitignored dirs. +// E-PENPAL-SCAN: lightweight startup check consistent with full scan filtering. +func projectHasAnyMarkdown(projectPath string) bool { + gitChecker := newGitIgnoreChecker(projectPath) + defer gitChecker.Close() + + err := filepath.WalkDir(projectPath, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return nil + } + if d.IsDir() { + name := d.Name() + if name == ".git" || name == "node_modules" || name == ".hg" || name == ".svn" { + return filepath.SkipDir + } + // Skip nested git worktrees and submodules (.git file, not dir). + if path != projectPath { + gitEntry := filepath.Join(path, ".git") + if fi, err := os.Lstat(gitEntry); err == nil && !fi.IsDir() { + return filepath.SkipDir + } + } + // E-PENPAL-SRC-GITIGNORE: skip gitignored directories. + if path != projectPath && gitChecker.IsIgnored(path) { + return filepath.SkipDir + } + return nil + } + if strings.HasSuffix(d.Name(), ".md") { + return errFoundMarkdown + } + return nil + }) + return errors.Is(err, errFoundMarkdown) +} + // EnrichProject updates a project's git info without rescanning files. // name should be the qualified name (e.g., "Development/penpal"). func (c *Cache) EnrichProject(name string, git *discovery.GitInfo) { @@ -519,10 +633,10 @@ func ScanProjectSourcesForWorktree(project *discovery.Project, worktreePath stri } // scanProjectSources scans all sources of a project for markdown files. -// Files are de-duplicated by project-relative path: if multiple sources cover -// the same file, only the first source's entry is kept. This means auto-detected -// sources (which come first) take priority over manual ones. -// E-PENPAL-SCAN: walks filesystem, skips dirs, classifies files, deduplicates, sorts by ModTime. +// Each source does its own walk. Files are de-duplicated by project-relative +// path: if multiple sources cover the same file, only the first source's entry +// is kept — except __all_markdown__ which claims every file regardless. +// E-PENPAL-SCAN: per-source WalkDir, classifies files, deduplicates, sorts by ModTime. func scanProjectSources(project *discovery.Project) []FileInfo { var files []FileInfo seen := make(map[string]bool) // project-relative paths already claimed @@ -541,33 +655,30 @@ func scanProjectSources(project *discovery.Project) []FileInfo { stName = source.Name } st := discovery.GetSourceType(stName) + isAllMarkdown := source.Name == "__all_markdown__" - filepath.Walk(rootPath, func(path string, info os.FileInfo, err error) error { + filepath.WalkDir(rootPath, func(path string, d fs.DirEntry, err error) error { if err != nil { return nil } - if info.IsDir() { + if d.IsDir() { // Skip nested git worktrees and submodules: they contain a // .git file (not directory) pointing at the real gitdir. - // Without this, tree sources rooted at "." walk into - // .claude/worktrees// and surface duplicate files. if path != rootPath { gitEntry := filepath.Join(path, ".git") if fi, err := os.Lstat(gitEntry); err == nil && !fi.IsDir() { return filepath.SkipDir } } - // Never walk into .git (git doesn't report it as ignored). - if info.Name() == ".git" { + if d.Name() == ".git" { return filepath.SkipDir } - // Skip gitignored directories (build output, deps, etc.). // P-PENPAL-SRC-GITIGNORE: registered source roots are // always scanned even if gitignored. if path != rootPath && gitChecker.IsIgnored(path) { return filepath.SkipDir } - if st != nil && st.SkipDirs[info.Name()] { + if st != nil && st.SkipDirs[d.Name()] { return filepath.SkipDir } return nil @@ -585,7 +696,7 @@ func scanProjectSources(project *discovery.Project) []FileInfo { relToSource, _ := filepath.Rel(rootPath, path) relToProject, _ := filepath.Rel(project.Path, path) - if seen[relToProject] { + if !isAllMarkdown && seen[relToProject] { return nil // already claimed by an earlier source } @@ -603,9 +714,15 @@ func scanProjectSources(project *discovery.Project) []FileInfo { } } - title := extractTitle(path) + // Only stat .md files (for ModTime), not every entry. + info, err := d.Info() + if err != nil { + return nil + } - seen[relToProject] = true + if !isAllMarkdown { + seen[relToProject] = true + } files = append(files, FileInfo{ Project: project.QualifiedName(), Workspace: project.WorkspaceName, @@ -615,8 +732,8 @@ func scanProjectSources(project *discovery.Project) []FileInfo { SourceAuto: source.Auto, Path: relToSource, FullPath: relToProject, - Name: filepath.Base(path), - Title: title, + Name: d.Name(), + Title: extractTitle(path), ModTime: info.ModTime(), FileType: fileType, }) diff --git a/apps/penpal/internal/cache/cache_test.go b/apps/penpal/internal/cache/cache_test.go index aaf39e95d..19e4e6cf3 100644 --- a/apps/penpal/internal/cache/cache_test.go +++ b/apps/penpal/internal/cache/cache_test.go @@ -484,6 +484,143 @@ func TestGitIgnoreChecker_NonGitDir(t *testing.T) { } } +// E-PENPAL-SCAN: verifies CheckAllProjectsHasFiles sets HasFiles correctly. +func TestCheckAllProjectsHasFiles(t *testing.T) { + tmpDir := t.TempDir() + + // Project with markdown files + withMD := filepath.Join(tmpDir, "with-md") + os.MkdirAll(filepath.Join(withMD, "docs"), 0755) + os.WriteFile(filepath.Join(withMD, "docs", "readme.md"), []byte("# Hello"), 0644) + + // Project with no markdown files + withoutMD := filepath.Join(tmpDir, "without-md") + os.MkdirAll(withoutMD, 0755) + os.WriteFile(filepath.Join(withoutMD, "main.go"), []byte("package main"), 0644) + + c := New() + c.SetProjects([]discovery.Project{ + {Name: "has-files", Path: withMD}, + {Name: "no-files", Path: withoutMD}, + }) + + c.CheckAllProjectsHasFiles() + + projects := c.Projects() + for _, p := range projects { + switch p.Name { + case "has-files": + if !p.HasFiles { + t.Error("project with .md files should have HasFiles=true") + } + case "no-files": + if p.HasFiles { + t.Error("project without .md files should have HasFiles=false") + } + } + } +} + +// E-PENPAL-SCAN: verifies projectHasAnyMarkdown skips gitignored dirs. +func TestProjectHasAnyMarkdown_SkipsGitignored(t *testing.T) { + tmpDir := t.TempDir() + + runGit(t, tmpDir, "init") + runGit(t, tmpDir, "config", "user.email", "test@test.com") + runGit(t, tmpDir, "config", "user.name", "test") + + // All .md files are in a gitignored directory + os.WriteFile(filepath.Join(tmpDir, ".gitignore"), []byte("build/\n"), 0644) + os.MkdirAll(filepath.Join(tmpDir, "build"), 0755) + os.WriteFile(filepath.Join(tmpDir, "build", "output.md"), []byte("# Gen"), 0644) + + if projectHasAnyMarkdown(tmpDir) { + t.Error("expected false: only .md files are in gitignored dir") + } + + // Add a non-gitignored .md file and re-check + os.WriteFile(filepath.Join(tmpDir, "README.md"), []byte("# README"), 0644) + if !projectHasAnyMarkdown(tmpDir) { + t.Error("expected true: README.md is not gitignored") + } +} + +// E-PENPAL-SCAN: verifies projectHasAnyMarkdown skips .hg and .svn dirs. +func TestProjectHasAnyMarkdown_SkipsVCSDirs(t *testing.T) { + tmpDir := t.TempDir() + + // .md files only in .hg and .svn dirs + os.MkdirAll(filepath.Join(tmpDir, ".hg"), 0755) + os.WriteFile(filepath.Join(tmpDir, ".hg", "notes.md"), []byte("# HG"), 0644) + os.MkdirAll(filepath.Join(tmpDir, ".svn"), 0755) + os.WriteFile(filepath.Join(tmpDir, ".svn", "notes.md"), []byte("# SVN"), 0644) + + if projectHasAnyMarkdown(tmpDir) { + t.Error("expected false: only .md files are in .hg/.svn dirs") + } +} + +// E-PENPAL-SRC-ALL-MD: verifies AllFiles deduplicates __all_markdown__ entries. +func TestAllFiles_DeduplicatesAllMarkdown(t *testing.T) { + c := New() + now := time.Now() + + c.SetProjectFiles("proj", []FileInfo{ + {Project: "proj", Source: "thoughts", FullPath: "thoughts/plan.md", Name: "plan.md", ModTime: now}, + {Project: "proj", Source: "__all_markdown__", FullPath: "thoughts/plan.md", Name: "plan.md", ModTime: now}, + {Project: "proj", Source: "__all_markdown__", FullPath: "README.md", Name: "README.md", ModTime: now}, + }) + + files := c.AllFiles(0) + if len(files) != 2 { + t.Fatalf("expected 2 unique files, got %d", len(files)) + } + + // The thoughts/plan.md entry should prefer the typed source + for _, f := range files { + if f.FullPath == "thoughts/plan.md" && f.Source != "thoughts" { + t.Errorf("expected source 'thoughts' for thoughts/plan.md, got %q", f.Source) + } + // README.md only exists in __all_markdown__, so that's fine + if f.FullPath == "README.md" && f.Source != "__all_markdown__" { + t.Errorf("expected source '__all_markdown__' for README.md, got %q", f.Source) + } + } +} + +// E-PENPAL-SCAN: verifies EnsureProjectScanned prevents concurrent duplicate scans. +func TestEnsureProjectScanned_NoDuplicateScans(t *testing.T) { + tmpDir := t.TempDir() + os.WriteFile(filepath.Join(tmpDir, "notes.md"), []byte("# Notes"), 0644) + + c := New() + c.SetProjects([]discovery.Project{ + { + Name: "test", + Path: tmpDir, + Sources: []discovery.FileSource{ + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true}, + }, + }, + }) + + // First call should scan + if !c.EnsureProjectScanned("test") { + t.Error("first call should return true (scan performed)") + } + + // Second call should be a no-op + if c.EnsureProjectScanned("test") { + t.Error("second call should return false (already scanned)") + } + + // Files should be populated from the first scan + files := c.ProjectFiles("test") + if len(files) == 0 { + t.Error("expected files after scan") + } +} + func runGit(t *testing.T, dir string, args ...string) { t.Helper() cmd := exec.Command("git", args...) diff --git a/apps/penpal/internal/discovery/discovery.go b/apps/penpal/internal/discovery/discovery.go index 147378bba..bd2b05106 100644 --- a/apps/penpal/internal/discovery/discovery.go +++ b/apps/penpal/internal/discovery/discovery.go @@ -162,6 +162,22 @@ func init() { return "plan" }, }) + + // E-PENPAL-SRC-ALL-MD: catch-all source that finds all .md files in the + // project tree. Registered last so typed sources claim their files first. + // Always present — the scan simply finds nothing if there are no .md files. + RegisterSourceType(&SourceType{ + Name: "__all_markdown__", + ScanMode: "tree", + ShowDirHeadings: true, + SkipDirs: map[string]bool{ + "node_modules": true, ".hg": true, ".svn": true, + }, + GroupFiles: func(paths []string) []FileGroup { + sort.Strings(paths) + return []FileGroup{{Name: "All Markdown", Paths: paths}} + }, + }) } // classifyRP1Feature classifies a file under work/features/{id}/ by its filename. @@ -291,15 +307,6 @@ func groupAnchorsPaths(paths []string) []FileGroup { return result } -// Badge holds rendering metadata for a source type badge. -type Badge struct { - Text string - Color string - Bg string - ActiveBg string - ActiveColor string -} - // FileSource represents a set of files to display for a project. type FileSource struct { Name string // display name (e.g., "thoughts", "docs") @@ -318,7 +325,7 @@ type Project struct { WorkspacePath string // which workspace discovered this (empty for standalone) WorkspaceName string // display name of the workspace (empty for standalone) Git *GitInfo - FileCount int + HasFiles bool LastModified time.Time Worktrees []Worktree // discovered worktrees for this project } @@ -348,27 +355,6 @@ func (p *Project) HasSourceType(name string) bool { return false } -// Badges returns badge metadata for all auto-detected sources. -func (p *Project) Badges() []Badge { - var badges []Badge - for _, s := range p.Sources { - if !s.Auto { - continue - } - st := GetSourceType(s.Name) - if st != nil { - badges = append(badges, Badge{ - Text: st.DisplayName, - Color: st.BadgeColor, - Bg: st.BadgeBg, - ActiveBg: st.BadgeActiveBg, - ActiveColor: st.BadgeActiveColor, - }) - } - } - return badges -} - // QualifiedName returns the workspace-qualified project identifier // (e.g., "Development/penpal"). This is the unique key used for // cache lookups, comment storage, API calls, and SSE events. @@ -409,6 +395,17 @@ func DetectSources(projectPath string) []FileSource { } } } + + // E-PENPAL-SRC-ALL-MD: always append the catch-all source last so typed + // sources claim their files first during scanning. + sources = append(sources, FileSource{ + Name: "__all_markdown__", + Type: "tree", + SourceTypeName: "__all_markdown__", + RootPath: projectPath, + Auto: true, + }) + return sources } diff --git a/apps/penpal/internal/server/api_manage_test.go b/apps/penpal/internal/server/api_manage_test.go index 352ab0d28..1be9d2b0e 100644 --- a/apps/penpal/internal/server/api_manage_test.go +++ b/apps/penpal/internal/server/api_manage_test.go @@ -384,6 +384,51 @@ func TestAPIDeleteFile_CleansUpCommentSidecar(t *testing.T) { } } +// E-PENPAL-ADD-SOURCE: verifies __all_markdown__ source doesn't block manual file additions. +// The conflict check must skip __all_markdown__ since it covers the entire project tree. +func TestAPISources_AddFileNotBlockedByAllMarkdown(t *testing.T) { + s, _, _ := testServer(t) + + dir := t.TempDir() + os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes"), 0o644) + + // Register as standalone with __all_markdown__ source (always present in real projects) + s.cfg.Projects = append(s.cfg.Projects, config.ProjectConfig{Path: dir}) + s.refreshAfterConfigChange() + + projName := filepath.Base(dir) + + // Verify the project has __all_markdown__ source + project := s.cache.FindProject(projName) + if project == nil { + t.Fatal("project not found after refresh") + } + hasAllMD := false + for _, src := range project.Sources { + if src.Name == "__all_markdown__" { + hasAllMD = true + break + } + } + if !hasAllMD { + t.Fatal("project should have __all_markdown__ source") + } + + // Adding a file should succeed (not 409) even though __all_markdown__ covers it + body, _ := json.Marshal(map[string]string{ + "project": projName, + "path": "notes.md", + }) + req := httptest.NewRequest(http.MethodPost, "/api/sources", bytes.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + rec := httptest.NewRecorder() + s.ServeHTTP(rec, req) + + if rec.Code != http.StatusNoContent { + t.Fatalf("expected 204, got %d: %s — __all_markdown__ should not block file additions", rec.Code, rec.Body.String()) + } +} + // E-PENPAL-REMOVE-SOURCE: verifies adding a tree source then removing it via DELETE /api/sources. func TestAPISources_RemoveTreeSource(t *testing.T) { s, _, _ := testServer(t) diff --git a/apps/penpal/internal/server/api_projects_test.go b/apps/penpal/internal/server/api_projects_test.go index 2684133d3..f9236d064 100644 --- a/apps/penpal/internal/server/api_projects_test.go +++ b/apps/penpal/internal/server/api_projects_test.go @@ -11,6 +11,7 @@ import ( "time" "github.com/loganj/penpal/internal/cache" + "github.com/loganj/penpal/internal/discovery" ) // E-PENPAL-API-ROUTES: verifies GET /api/projects lists discovered projects. @@ -123,9 +124,11 @@ func TestAPIProjectInfo_ReturnsInfo(t *testing.T) { s, c, _ := testServer(t) dir := t.TempDir() - seedProject(c, "test-proj", dir, []cache.FileInfo{ + project := seedProject(c, "test-proj", dir, []cache.FileInfo{ {Project: "test-proj", Name: "plan.md", FullPath: "thoughts/plan.md", Source: "thoughts", ModTime: time.Now()}, }) + project.HasFiles = true + c.SetProjects([]discovery.Project{project}) req := httptest.NewRequest(http.MethodGet, "/api/project-info?name=test-proj", nil) rec := httptest.NewRecorder() @@ -139,8 +142,8 @@ func TestAPIProjectInfo_ReturnsInfo(t *testing.T) { if err := json.Unmarshal(rec.Body.Bytes(), &info); err != nil { t.Fatalf("parse JSON: %v", err) } - if info.FileCount != 1 { - t.Errorf("expected fileCount=1, got %d", info.FileCount) + if !info.HasFiles { + t.Errorf("expected hasFiles=true, got false") } } diff --git a/apps/penpal/internal/server/grouping_test.go b/apps/penpal/internal/server/grouping_test.go index 249b49f41..5046f89c1 100644 --- a/apps/penpal/internal/server/grouping_test.go +++ b/apps/penpal/internal/server/grouping_test.go @@ -10,25 +10,33 @@ import ( // E-PENPAL-API-ROUTES: verifies RP1 source produces grouped file sections. func TestBuildFileGroups_RP1Grouped(t *testing.T) { + now := time.Now() project := &discovery.Project{ Name: "test-project", Path: "/tmp/test", Sources: []discovery.FileSource{ {Name: "rp1", Type: "tree", SourceTypeName: "rp1", RootPath: "/tmp/test/.rp1", Auto: true}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, }, } files := []cache.FileInfo{ - {Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: time.Now()}, - {Source: "rp1", Path: "work/prds/my-prd.md", FullPath: ".rp1/work/prds/my-prd.md", Name: "my-prd.md", FileType: "prd", ModTime: time.Now()}, - {Source: "rp1", Path: "work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: time.Now()}, - {Source: "rp1", Path: "work/features/auth/design.md", FullPath: ".rp1/work/features/auth/design.md", Name: "design.md", FileType: "design", ModTime: time.Now()}, - {Source: "rp1", Path: "work/features/data-layer/tasks.md", FullPath: ".rp1/work/features/data-layer/tasks.md", Name: "tasks.md", FileType: "task", ModTime: time.Now()}, + {Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now}, + {Source: "rp1", Path: "work/prds/my-prd.md", FullPath: ".rp1/work/prds/my-prd.md", Name: "my-prd.md", FileType: "prd", ModTime: now}, + {Source: "rp1", Path: "work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: now}, + {Source: "rp1", Path: "work/features/auth/design.md", FullPath: ".rp1/work/features/auth/design.md", Name: "design.md", FileType: "design", ModTime: now}, + {Source: "rp1", Path: "work/features/data-layer/tasks.md", FullPath: ".rp1/work/features/data-layer/tasks.md", Name: "tasks.md", FileType: "task", ModTime: now}, + // __all_markdown__ claims all files + {Source: "__all_markdown__", Path: ".rp1/context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now}, + {Source: "__all_markdown__", Path: ".rp1/work/prds/my-prd.md", FullPath: ".rp1/work/prds/my-prd.md", Name: "my-prd.md", FileType: "prd", ModTime: now}, + {Source: "__all_markdown__", Path: ".rp1/work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: now}, + {Source: "__all_markdown__", Path: ".rp1/work/features/auth/design.md", FullPath: ".rp1/work/features/auth/design.md", Name: "design.md", FileType: "design", ModTime: now}, + {Source: "__all_markdown__", Path: ".rp1/work/features/data-layer/tasks.md", FullPath: ".rp1/work/features/data-layer/tasks.md", Name: "tasks.md", FileType: "task", ModTime: now}, } groups := buildFileGroups(project, files) - // Should have 4 typed groups + 1 "All Markdown" virtual group + // Should have 4 typed groups + 1 "All Markdown" group if len(groups) != 5 { t.Fatalf("expected 5 groups, got %d", len(groups)) } @@ -63,17 +71,21 @@ func TestBuildFileGroups_RP1Grouped(t *testing.T) { // E-PENPAL-API-ROUTES, E-PENPAL-SRC-THOUGHTS: verifies thoughts source produces a single flat group. func TestBuildFileGroups_ThoughtsFlat(t *testing.T) { + now := time.Now() project := &discovery.Project{ Name: "test-project", Path: "/tmp/test", Sources: []discovery.FileSource{ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, }, } files := []cache.FileInfo{ - {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: time.Now()}, - {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: time.Now()}, + {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: now}, + {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: now}, + {Source: "__all_markdown__", Path: "thoughts/plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: now}, + {Source: "__all_markdown__", Path: "thoughts/research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: now}, } groups := buildFileGroups(project, files) @@ -95,18 +107,22 @@ func TestBuildFileGroups_ThoughtsFlat(t *testing.T) { // E-PENPAL-API-ROUTES: verifies multiple sources produce separate groups. func TestBuildFileGroups_MultipleSources(t *testing.T) { + now := time.Now() project := &discovery.Project{ Name: "test-project", Path: "/tmp/test", Sources: []discovery.FileSource{ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true}, {Name: "rp1", Type: "tree", SourceTypeName: "rp1", RootPath: "/tmp/test/.rp1", Auto: true}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, }, } files := []cache.FileInfo{ - {Source: "thoughts", Path: "plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: time.Now()}, - {Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: time.Now()}, + {Source: "thoughts", Path: "plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: now}, + {Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now}, + {Source: "__all_markdown__", Path: "thoughts/plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: now}, + {Source: "__all_markdown__", Path: ".rp1/context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now}, } groups := buildFileGroups(project, files) @@ -126,23 +142,26 @@ func TestBuildFileGroups_MultipleSources(t *testing.T) { // E-PENPAL-API-ROUTES: verifies empty sources are omitted from groups. func TestBuildFileGroups_EmptySourceSkipped(t *testing.T) { + now := time.Now() project := &discovery.Project{ Name: "test-project", Path: "/tmp/test", Sources: []discovery.FileSource{ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true}, {Name: "rp1", Type: "tree", SourceTypeName: "rp1", RootPath: "/tmp/test/.rp1", Auto: true}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, }, } // Only thoughts has files, rp1 is empty files := []cache.FileInfo{ - {Source: "thoughts", Path: "plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: time.Now()}, + {Source: "thoughts", Path: "plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: now}, + {Source: "__all_markdown__", Path: "thoughts/plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: now}, } groups := buildFileGroups(project, files) - // 1 typed group (empty rp1 skipped) + All Markdown + // 1 thoughts group (empty rp1 skipped) + All Markdown if len(groups) != 2 { t.Fatalf("expected 2 groups (empty rp1 skipped), got %d", len(groups)) } @@ -153,19 +172,25 @@ func TestBuildFileGroups_EmptySourceSkipped(t *testing.T) { // E-PENPAL-ADD-SOURCE, E-PENPAL-SRC-MANUAL: verifies manual source produces directory headings. func TestBuildFileGroups_ManualSourceDirHeadings(t *testing.T) { + now := time.Now() project := &discovery.Project{ Name: "test-project", Path: "/tmp/test", Sources: []discovery.FileSource{ {Name: "docs", Type: "tree", SourceTypeName: "manual", RootPath: "/tmp/test/docs", Auto: false}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, }, } files := []cache.FileInfo{ - {Source: "docs", Path: "root.md", FullPath: "docs/root.md", Name: "root.md", FileType: "other", ModTime: time.Now()}, - {Source: "docs", Path: "guides/setup.md", FullPath: "docs/guides/setup.md", Name: "setup.md", FileType: "other", ModTime: time.Now()}, - {Source: "docs", Path: "guides/deploy.md", FullPath: "docs/guides/deploy.md", Name: "deploy.md", FileType: "other", ModTime: time.Now()}, - {Source: "docs", Path: "api/endpoints.md", FullPath: "docs/api/endpoints.md", Name: "endpoints.md", FileType: "other", ModTime: time.Now()}, + {Source: "docs", Path: "root.md", FullPath: "docs/root.md", Name: "root.md", FileType: "other", ModTime: now}, + {Source: "docs", Path: "guides/setup.md", FullPath: "docs/guides/setup.md", Name: "setup.md", FileType: "other", ModTime: now}, + {Source: "docs", Path: "guides/deploy.md", FullPath: "docs/guides/deploy.md", Name: "deploy.md", FileType: "other", ModTime: now}, + {Source: "docs", Path: "api/endpoints.md", FullPath: "docs/api/endpoints.md", Name: "endpoints.md", FileType: "other", ModTime: now}, + {Source: "__all_markdown__", Path: "docs/root.md", FullPath: "docs/root.md", Name: "root.md", FileType: "other", ModTime: now}, + {Source: "__all_markdown__", Path: "docs/guides/setup.md", FullPath: "docs/guides/setup.md", Name: "setup.md", FileType: "other", ModTime: now}, + {Source: "__all_markdown__", Path: "docs/guides/deploy.md", FullPath: "docs/guides/deploy.md", Name: "deploy.md", FileType: "other", ModTime: now}, + {Source: "__all_markdown__", Path: "docs/api/endpoints.md", FullPath: "docs/api/endpoints.md", Name: "endpoints.md", FileType: "other", ModTime: now}, } groups := buildFileGroups(project, files) @@ -197,17 +222,21 @@ func TestBuildFileGroups_ManualSourceDirHeadings(t *testing.T) { // E-PENPAL-API-ROUTES: verifies file titles flow through to group view. func TestBuildFileGroups_TitleFlowsThrough(t *testing.T) { + now := time.Now() project := &discovery.Project{ Name: "test-project", Path: "/tmp/test", Sources: []discovery.FileSource{ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, }, } files := []cache.FileInfo{ - {Source: "thoughts", Path: "plans/my-plan.md", FullPath: "thoughts/plans/my-plan.md", Name: "my-plan.md", Title: "Per-Tab Navigation", FileType: "plan", ModTime: time.Now()}, - {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", Title: "", FileType: "research", ModTime: time.Now()}, + {Source: "thoughts", Path: "plans/my-plan.md", FullPath: "thoughts/plans/my-plan.md", Name: "my-plan.md", Title: "Per-Tab Navigation", FileType: "plan", ModTime: now}, + {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", Title: "", FileType: "research", ModTime: now}, + {Source: "__all_markdown__", Path: "thoughts/plans/my-plan.md", FullPath: "thoughts/plans/my-plan.md", Name: "my-plan.md", Title: "Per-Tab Navigation", FileType: "plan", ModTime: now}, + {Source: "__all_markdown__", Path: "thoughts/research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", Title: "", FileType: "research", ModTime: now}, } groups := buildFileGroups(project, files) @@ -227,17 +256,21 @@ func TestBuildFileGroups_TitleFlowsThrough(t *testing.T) { // E-PENPAL-API-ROUTES, E-PENPAL-SRC-THOUGHTS: verifies thoughts source does not produce directory headings. func TestBuildFileGroups_ThoughtsNoDirHeadings(t *testing.T) { + now := time.Now() project := &discovery.Project{ Name: "test-project", Path: "/tmp/test", Sources: []discovery.FileSource{ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, }, } files := []cache.FileInfo{ - {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: time.Now()}, - {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: time.Now()}, + {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: now}, + {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: now}, + {Source: "__all_markdown__", Path: "thoughts/plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: now}, + {Source: "__all_markdown__", Path: "thoughts/research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: now}, } groups := buildFileGroups(project, files) @@ -260,12 +293,17 @@ func TestBuildFileGroups_AllMarkdownVirtual(t *testing.T) { Path: "/tmp/test", Sources: []discovery.FileSource{ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, }, } + // scanProjectSources produces entries for both the typed source and + // the __all_markdown__ catch-all (which claims every .md file). files := []cache.FileInfo{ {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: time.Now()}, {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: time.Now()}, + {Source: "__all_markdown__", Path: "thoughts/plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: time.Now()}, + {Source: "__all_markdown__", Path: "thoughts/research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: time.Now()}, } groups := buildFileGroups(project, files) diff --git a/apps/penpal/internal/server/integration_test.go b/apps/penpal/internal/server/integration_test.go index 15af4604b..1f3388847 100644 --- a/apps/penpal/internal/server/integration_test.go +++ b/apps/penpal/internal/server/integration_test.go @@ -19,6 +19,7 @@ func TestAPIProjectFiles_ReturnsGroups(t *testing.T) { project := seedProject(c, projectName, "/tmp/test", nil) project.Sources = []discovery.FileSource{ {Name: "rp1", Type: "tree", SourceTypeName: "rp1", RootPath: "/tmp/test/.rp1", Auto: true}, + {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true}, } // Re-set projects to include the sources we just added c.SetProjects([]discovery.Project{project}) @@ -27,6 +28,9 @@ func TestAPIProjectFiles_ReturnsGroups(t *testing.T) { files := []cache.FileInfo{ {Project: projectName, Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now}, {Project: projectName, Source: "rp1", Path: "work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: now}, + // __all_markdown__ source claims all files (duplicates typed sources) + {Project: projectName, Source: "__all_markdown__", Path: ".rp1/context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now}, + {Project: projectName, Source: "__all_markdown__", Path: ".rp1/work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: now}, } c.SetProjectFiles(projectName, files) diff --git a/apps/penpal/internal/server/manage.go b/apps/penpal/internal/server/manage.go index ff67e85d6..42dd7e09e 100644 --- a/apps/penpal/internal/server/manage.go +++ b/apps/penpal/internal/server/manage.go @@ -325,6 +325,11 @@ func (s *Server) handleAddSource(w http.ResponseWriter, r *http.Request) { // Check if this file is already covered by an existing source for _, src := range project.Sources { + // Skip the virtual __all_markdown__ source — it covers the whole + // project tree but should not block manual file additions. + if src.Name == "__all_markdown__" { + continue + } if src.Type == "thoughts" || src.Type == "tree" { if src.RootPath != "" && strings.HasPrefix(absPath, src.RootPath+"/") { http.Error(w, fmt.Sprintf("file is already included via source %q", src.Name), http.StatusConflict) diff --git a/apps/penpal/internal/server/search.go b/apps/penpal/internal/server/search.go index bbcb73e05..1da105f59 100644 --- a/apps/penpal/internal/server/search.go +++ b/apps/penpal/internal/server/search.go @@ -79,7 +79,18 @@ func (s *Server) handleAPISearch(w http.ResponseWriter, r *http.Request) { return nil } if info.IsDir() { - if st != nil && st.SkipDirs[info.Name()] { + name := info.Name() + // E-PENPAL-SEARCH: skip .git dirs and nested worktrees/submodules. + if name == ".git" { + return filepath.SkipDir + } + if path != source.RootPath { + gitEntry := filepath.Join(path, ".git") + if fi, gErr := os.Lstat(gitEntry); gErr == nil && !fi.IsDir() { + return filepath.SkipDir + } + } + if st != nil && st.SkipDirs[name] { return filepath.SkipDir } return nil diff --git a/apps/penpal/internal/server/server.go b/apps/penpal/internal/server/server.go index b52dcf668..7dbb8ca78 100644 --- a/apps/penpal/internal/server/server.go +++ b/apps/penpal/internal/server/server.go @@ -214,9 +214,8 @@ func (s *Server) handleReady(w http.ResponseWriter, _ *http.Request) { // populateProjects scans file lists and fills in git info in the background. // E-PENPAL-GIT-ENRICH: background git enrichment with SSE push. func (s *Server) populateProjects() { - s.cache.RefreshAllProjects() - s.seedRecentActivity() - log.Printf("Background file scan complete") + s.cache.CheckAllProjectsHasFiles() + log.Printf("Background hasFiles check complete") s.readyOnce.Do(func() { close(s.readyCh) }) // signal that essential data is ready s.watcher.Broadcast(watcher.Event{Type: watcher.EventProjectsChanged}) @@ -253,17 +252,6 @@ func (s *Server) populateProjects() { s.watcher.Broadcast(watcher.Event{Type: watcher.EventProjectsChanged}) } -// seedRecentActivity populates the activity tracker with ModTimes from the -// most recently modified files in the cache. This ensures /recent has data -// immediately on startup, even for files modified before the server started. -// E-PENPAL-ACTIVITY: seeds file modification events at startup. -func (s *Server) seedRecentActivity() { - files := s.cache.AllFiles(50) - for _, f := range files { - s.activity.RecordAt(activity.FileModified, f.Project, f.FullPath, f.ModTime) - } -} - // E-PENPAL-API-ROUTES: registers all REST endpoints, SPA, SSE, and MCP. func (s *Server) routes() { // Main mux (:8080) — API, SSE, MCP, and React SPA @@ -626,31 +614,6 @@ func buildFileGroups(project *discovery.Project, cachedFiles []cache.FileInfo) [ groups = append(groups, gv) } - // E-PENPAL-SRC-ALL-MD: virtual "All Markdown" group with every file, organized by directory. - // Always present, no dedup against typed sources. - allMd := FileGroupView{ - Name: "All Markdown", - Source: "__all_markdown__", - SourceType: "tree", - Auto: true, - } - for _, f := range cachedFiles { - allMd.Files = append(allMd.Files, ProjectFile{ - Name: f.Name, - Title: f.Title, - Path: f.FullPath, - Source: f.Source, - SourceType: f.SourceType, - ModTime: f.ModTime, - Age: formatAge(f.ModTime), - FileType: f.FileType, - }) - } - sort.Slice(allMd.Files, func(i, j int) bool { - return allMd.Files[i].Path < allMd.Files[j].Path - }) - groups = append(groups, allMd) - return groups } @@ -697,12 +660,6 @@ func (s *Server) handleEvents(w http.ResponseWriter, r *http.Request) { // API handlers for dynamic updates -type APIBadge struct { - Text string `json:"text"` - Color string `json:"color"` - Bg string `json:"bg"` -} - type APIWorktree struct { Name string `json:"name"` Path string `json:"path"` @@ -717,10 +674,9 @@ type APIProject struct { WorkspacePath string `json:"workspacePath,omitempty"` ProjectPath string `json:"projectPath"` Origin string `json:"origin"` - Badges []APIBadge `json:"badges"` Branch string `json:"branch,omitempty"` Dirty bool `json:"dirty,omitempty"` - FileCount int `json:"fileCount"` + HasFiles bool `json:"hasFiles"` LastModified string `json:"lastModified"` AgentConnected bool `json:"agentConnected,omitempty"` AgentRunning bool `json:"agentRunning,omitempty"` @@ -750,10 +706,6 @@ func (s *Server) handleListAPIProjects(w http.ResponseWriter, r *http.Request) { result := make([]APIProject, len(projects)) for i, p := range projects { qn := p.QualifiedName() - apiBadges := make([]APIBadge, 0) - for _, b := range p.Badges() { - apiBadges = append(apiBadges, APIBadge{Text: b.Text, Color: b.Color, Bg: b.Bg}) - } result[i] = APIProject{ Name: p.Name, QualifiedName: qn, @@ -761,8 +713,7 @@ func (s *Server) handleListAPIProjects(w http.ResponseWriter, r *http.Request) { WorkspacePath: p.WorkspacePath, ProjectPath: p.Path, Origin: p.Origin, - Badges: apiBadges, - FileCount: p.FileCount, + HasFiles: p.HasFiles, LastModified: p.LastModified.Format(time.RFC3339), Age: computeProjectAge(p), AgentConnected: s.agents != nil && s.agents.Status(qn) != nil && s.agents.Status(qn).Running, @@ -850,8 +801,16 @@ func (s *Server) handleAPIProjectFiles(w http.ResponseWriter, r *http.Request) { } cachedFiles = cache.ScanProjectSourcesForWorktree(project, wtPath) } else { - // Ensure file titles are populated before serving - s.cache.EnrichTitles(qualifiedName) + // Lazy scan: full file scan happens on first access, not at startup. + // E-PENPAL-ACTIVITY: seed recent activity on first scan so /api/recent + // populates progressively as projects are opened. + if s.cache.EnsureProjectScanned(qualifiedName) { + for _, f := range s.cache.ProjectFiles(qualifiedName) { + if f.Source != "__all_markdown__" { + s.activity.RecordAt(activity.FileModified, f.Project, f.FullPath, f.ModTime) + } + } + } cachedFiles = s.cache.ProjectFiles(qualifiedName) } fileGroups := buildFileGroups(project, cachedFiles) @@ -945,7 +904,7 @@ func (s *Server) handleCopyFile(w http.ResponseWriter, r *http.Request) { } type ProjectInfo struct { - FileCount int `json:"fileCount"` + HasFiles bool `json:"hasFiles"` Dirty bool `json:"dirty"` UnpushedCommits int `json:"unpushedCommits"` } @@ -964,7 +923,7 @@ func (s *Server) handleProjectInfo(w http.ResponseWriter, r *http.Request) { } info := ProjectInfo{ - FileCount: len(s.cache.ProjectFiles(qualifiedName)), + HasFiles: project.HasFiles, } // Fresh git status