diff --git a/apps/penpal/ERD.md b/apps/penpal/ERD.md
index ee1185986..44bd54360 100644
--- a/apps/penpal/ERD.md
+++ b/apps/penpal/ERD.md
@@ -85,7 +85,7 @@ see-also:
- **E-PENPAL-SRC-MANUAL**: The `manual` source type is used for user-added sources (via `penpal open` CLI or the add-workspace/project modal). Directory sources create "tree" type entries; individual file sources create "files" type entries. `GroupFiles()` generates directory headings (`Dir`, `ShowDir` fields on `FileInfo`) for subdirectory boundaries. Configuration is persisted in `config.json` under `ProjectSources` (workspace projects) or inline with the `ProjectConfig` entry (standalone projects).
← [P-PENPAL-CLI-OPEN](PRODUCT.md#P-PENPAL-CLI-OPEN), [P-PENPAL-STANDALONE](PRODUCT.md#P-PENPAL-STANDALONE)
-- **E-PENPAL-SRC-ALL-MD**: An "All Markdown" virtual source is always present in every project's sidebar. It shows all `.md` files in the project directory tree, organized by directory structure, regardless of whether they also appear in a typed source. No badge, no deduplication against other sources. The `Auto` field is `true` — the frontend uses this field to control whether the "Remove from Penpal" context menu item appears, and virtual sources cannot be removed. Implemented as a synthetic source group appended after the typed source groups in the project files response.
+- **E-PENPAL-SRC-ALL-MD**: An "All Markdown" source is always present in every project. Registered as a `SourceType` with `GroupFiles` returning a single "All Markdown" group, `ShowDirHeadings: true`, and `SkipDirs` for `node_modules`, `.hg`, `.svn`. Always appended by `DetectSources()` after the detection loop — not conditionally detected. The `Auto` field is `true` — the frontend uses this to prevent removal. During scanning, `__all_markdown__` bypasses the per-source dedup (`seen` map) so it claims every `.md` file regardless of other sources. `AllFiles()` deduplicates by project+path, preferring typed-source entries over `__all_markdown__`, so the recent-files list and activity seeding see each file exactly once. The `handleAddSource` conflict check skips `__all_markdown__` so it doesn't block manual file additions.
← [P-PENPAL-SRC-ALL-MD](PRODUCT.md#P-PENPAL-SRC-ALL-MD)
- **E-PENPAL-WORKTREE-DISCOVERY**: Worktrees are discovered by parsing `git worktree list --porcelain` output. Each worktree gets a name, path, branch, and `IsMain` flag. The `refs/heads/` prefix is stripped from branch names.
@@ -101,7 +101,7 @@ see-also:
- **E-PENPAL-CACHE**: An in-memory cache (`sync.RWMutex`-protected) holds the full project list and per-project file lists. `RefreshProject()` walks the filesystem; `RefreshAllProjects()` runs in parallel with no concurrency limit. `RescanWith()` replaces the project list while preserving git enrichment.
← [P-PENPAL-PROJECT-FILE-TREE](PRODUCT.md#P-PENPAL-PROJECT-FILE-TREE)
-- **E-PENPAL-SCAN**: `scanProjectSources()` walks `RootPath` recursively for tree sources, skipping `.git`-file directories (nested worktrees), gitignored directories (via `git check-ignore`), source-type `SkipDirs`, and non-`.md` files. Gitignore checking is initialized once per scan via `newGitIgnoreChecker(projectPath)`, which detects whether the project is a git repo; non-git projects skip the gitignore check gracefully. On write or read failure (partial 4-field response), the checker disables itself (`isGitRepo=false`) to prevent permanent stream desync. The source's own `rootPath` is never checked against gitignore (the `path != rootPath` guard ensures registered sources always scan). Files returning `""` from `ClassifyFile()` are hidden. Files are de-duplicated by project-relative path (first source wins) and sorted by `ModTime` descending.
+- **E-PENPAL-SCAN**: `scanProjectSources()` walks `RootPath` recursively for tree sources, skipping `.git`-file directories (nested worktrees), gitignored directories (via `git check-ignore`), source-type `SkipDirs`, and non-`.md` files. Gitignore checking is initialized once per scan via `newGitIgnoreChecker(projectPath)`, which detects whether the project is a git repo; non-git projects skip the gitignore check gracefully. On write or read failure (partial 4-field response), the checker disables itself (`isGitRepo=false`) to prevent permanent stream desync. The source's own `rootPath` is never checked against gitignore (the `path != rootPath` guard ensures registered sources always scan). Files returning `""` from `ClassifyFile()` are hidden. Files are de-duplicated by project-relative path (first source wins) and sorted by `ModTime` descending. `EnsureProjectScanned()` is the lazy-scan entry point — it uses write-lock gating (`projectScanned` set under `mu.Lock` before scanning) to prevent concurrent requests from triggering duplicate filesystem walks. `projectHasAnyMarkdown()` performs a cheap startup check that aligns with the full scan: it uses the same gitignore checking, skips `.git`, `node_modules`, `.hg`, `.svn`, and nested worktree directories, and stops at the first `.md` file found.
← [P-PENPAL-PROJECT-FILE-TREE](PRODUCT.md#P-PENPAL-PROJECT-FILE-TREE), [P-PENPAL-FILE-TYPES](PRODUCT.md#P-PENPAL-FILE-TYPES), [P-PENPAL-SRC-DEDUP](PRODUCT.md#P-PENPAL-SRC-DEDUP), [P-PENPAL-SRC-GITIGNORE](PRODUCT.md#P-PENPAL-SRC-GITIGNORE)
- **E-PENPAL-TITLE-EXTRACT**: `EnrichTitles()` reads the first 20 lines of each file to extract H1 headings. Titles are cached and shown as the primary display name when present.
@@ -351,7 +351,7 @@ see-also:
## Search
-- **E-PENPAL-SEARCH**: `GET /api/search?q=` handled by `handleAPISearch()`. Iterates `s.cache.Projects()`, doing substring match on project names for the "Projects" section (`matchingProjects`). For file matches, walks each source's `RootPath` via `filepath.Walk`, skipping `SkipDirs` and non-`.md` files. Filenames checked via `strings.Contains(ToLower(...), query)` (flagged `nameMatch: true`); content checked via `bufio.Scanner` line-by-line (stops at first match). Files returning `""` from `ClassifyFile()` are excluded. Results per project sorted name-matches-first then alphabetical; capped at 100 total files with early `break` on project iteration. Response struct: `apiSearchResponse{Query, MatchingProjects, ProjectResults, TotalFiles}`. Frontend `SearchPage` component at route `/search` calls `api.search(query)` and renders project cards, grouped file rows with `FileTypeBadge`, and a "name" `` badge for name matches.
+- **E-PENPAL-SEARCH**: `GET /api/search?q=` handled by `handleAPISearch()`. Iterates `s.cache.Projects()`, doing substring match on project names for the "Projects" section (`matchingProjects`). For file matches, walks each source's `RootPath` via `filepath.Walk`, skipping `.git` directories, nested worktree/submodule directories (`.git` file), `SkipDirs`, and non-`.md` files. Filenames checked via `strings.Contains(ToLower(...), query)` (flagged `nameMatch: true`); content checked via `bufio.Scanner` line-by-line (stops at first match). Files returning `""` from `ClassifyFile()` are excluded. Results per project sorted name-matches-first then alphabetical; capped at 100 total files with early `break` on project iteration. Response struct: `apiSearchResponse{Query, MatchingProjects, ProjectResults, TotalFiles}`. Frontend `SearchPage` component at route `/search` calls `api.search(query)` and renders project cards, grouped file rows with `FileTypeBadge`, and a "name" `` badge for name matches.
← [P-PENPAL-SEARCH](PRODUCT.md#P-PENPAL-SEARCH)
---
@@ -368,7 +368,7 @@ see-also:
## Activity Tracking
-- **E-PENPAL-ACTIVITY**: In-memory `activity.Tracker` stores one event per file (keyed by project + path). Event types: `viewed`, `modified`, `created`, `comment`, `published`. `Record()` always overwrites; `RecordAt()` (for seeding from mtime) does not overwrite.
+- **E-PENPAL-ACTIVITY**: In-memory `activity.Tracker` stores one event per file (keyed by project + path). Event types: `viewed`, `modified`, `created`, `comment`, `published`. `Record()` always overwrites; `RecordAt()` (for seeding from mtime) does not overwrite. Activity is seeded per-project when `EnsureProjectScanned()` performs the first lazy scan — each file's `ModTime` is recorded via `RecordAt()`, excluding `__all_markdown__` duplicates. This populates `/api/recent` progressively as projects are opened.
← [P-PENPAL-GLOBAL-RECENT](PRODUCT.md#P-PENPAL-GLOBAL-RECENT), [P-PENPAL-PROJECT-RECENT](PRODUCT.md#P-PENPAL-PROJECT-RECENT)
- **E-PENPAL-RECENT-PAGE**: `GET /api/recent` returns up to 50 recently active files across all projects, sorted by most recent activity. Frontend `RecentPage` renders two-line file rows with filename, workspace / project context, and relative timestamp. Clicking navigates the current tab to the file.
diff --git a/apps/penpal/TESTING.md b/apps/penpal/TESTING.md
index 916b4dc62..721b0fab0 100644
--- a/apps/penpal/TESTING.md
+++ b/apps/penpal/TESTING.md
@@ -64,7 +64,7 @@ see-also:
| Source Types — anchors (P-PENPAL-SRC-ANCHORS, SRC-ANCHORS-GROUP, SRC-ANCHORS-NESTED) | discovery_test.go (TestClassifyAnchorsFile, TestGroupAnchorsPaths, TestGroupAnchorsPaths_MarkerOnlyModule, TestAnchorsFileOrder, TestAnchorsRequireSibling) | — | — | — |
| Source Types — claude-plans (P-PENPAL-SRC-CLAUDE-PLANS) | — | — | — | — |
| Source Types — manual (P-PENPAL-SRC-MANUAL) | — | — | grouping_test.go (TestBuildFileGroups_ManualSourceDirHeadings) | — |
-| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go | — | — | — |
+| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go (TestCheckAllProjectsHasFiles, TestProjectHasAnyMarkdown_SkipsGitignored, TestProjectHasAnyMarkdown_SkipsVCSDirs, TestAllFiles_DeduplicatesAllMarkdown, TestEnsureProjectScanned_NoDuplicateScans) | — | — | — |
| Worktree Support (P-PENPAL-WORKTREE) | discovery/worktree_test.go, cache/worktree_test.go | Layout.test.tsx | worktree_test.go (API + MCP) | — |
| Worktree Dropdown (P-PENPAL-PROJECT-WORKTREE-DROPDOWN) | — | Layout.test.tsx | — | — |
| Git Integration (P-PENPAL-GIT-INFO) | — | — | — | — |
@@ -89,7 +89,7 @@ see-also:
| Search (P-PENPAL-SEARCH) | — | SearchPage.test.tsx | — | react-app.spec.ts |
| Recent Files (P-PENPAL-RECENT) | activity_test.go | RecentPage.test.tsx | integration_test.go | — |
| CLI Open (P-PENPAL-CLI-OPEN) | — | — | api_manage_test.go | cli-open.spec.ts |
-| Source Management (P-PENPAL-ADD-SOURCE, REMOVE-SOURCE) | — | — | api_manage_test.go | — |
+| Source Management (P-PENPAL-ADD-SOURCE, REMOVE-SOURCE) | — | — | api_manage_test.go (TestAPISources_AddFileNotBlockedByAllMarkdown) | — |
| Real-Time Updates (P-PENPAL-REALTIME, FOCUS) | watcher_test.go | useSSE.test.ts | api_focus_test.go | — |
| Config & Migration (E-PENPAL-CONFIG) | config_test.go, migrate_test.go | — | — | — |
| Install Tools (P-PENPAL-INSTALL) | — | InstallStartup.test.tsx, InstallToolsModal.test.tsx | install_test.go | — |
diff --git a/apps/penpal/frontend/src/components/HomeSidebar.tsx b/apps/penpal/frontend/src/components/HomeSidebar.tsx
index f32c07c72..a9f4d0c32 100644
--- a/apps/penpal/frontend/src/components/HomeSidebar.tsx
+++ b/apps/penpal/frontend/src/components/HomeSidebar.tsx
@@ -144,7 +144,7 @@ export default function HomeSidebar({
return (
{
if (hasWorktrees) {
onToggleWorktreeProject(p.qualifiedName);
@@ -201,7 +201,7 @@ export default function HomeSidebar({
onShowContextMenu(e, [
{ label: 'Close project', className: 'menu-muted', onClick: () => onCloseStandaloneProject(p) },
])}
diff --git a/apps/penpal/frontend/src/components/Layout.close-tab.test.tsx b/apps/penpal/frontend/src/components/Layout.close-tab.test.tsx
index ccbb2d908..f66bc8fc4 100644
--- a/apps/penpal/frontend/src/components/Layout.close-tab.test.tsx
+++ b/apps/penpal/frontend/src/components/Layout.close-tab.test.tsx
@@ -51,8 +51,7 @@ vi.mock('../api', () => ({
qualifiedName: 'ws1/project-a',
workspace: 'ws1',
origin: 'workspace',
- badges: [],
- fileCount: 5,
+ hasFiles: true,
lastModified: '2026-01-01T00:00:00Z',
},
]),
diff --git a/apps/penpal/frontend/src/components/Layout.external-links.test.tsx b/apps/penpal/frontend/src/components/Layout.external-links.test.tsx
index b1a11d025..94cfdfa2e 100644
--- a/apps/penpal/frontend/src/components/Layout.external-links.test.tsx
+++ b/apps/penpal/frontend/src/components/Layout.external-links.test.tsx
@@ -37,8 +37,7 @@ vi.mock('../api', () => ({
qualifiedName: 'ws1/project-a',
workspace: 'ws1',
origin: 'workspace',
- badges: [],
- fileCount: 5,
+ hasFiles: true,
lastModified: '2026-01-01T00:00:00Z',
},
]),
diff --git a/apps/penpal/frontend/src/components/Layout.test.tsx b/apps/penpal/frontend/src/components/Layout.test.tsx
index 38653140a..7a02a063a 100644
--- a/apps/penpal/frontend/src/components/Layout.test.tsx
+++ b/apps/penpal/frontend/src/components/Layout.test.tsx
@@ -29,8 +29,8 @@ vi.mock('../api', () => ({
qualifiedName: 'ws1/project-a',
workspace: 'ws1',
origin: 'workspace',
- badges: [],
- fileCount: 5,
+
+ hasFiles: true,
lastModified: '2026-01-01T00:00:00Z',
},
]),
@@ -541,8 +541,8 @@ describe('Layout', () => {
qualifiedName: 'ws1/project-a',
workspace: 'ws1',
origin: 'workspace',
- badges: [],
- fileCount: 5,
+
+ hasFiles: true,
lastModified: '2026-01-01T00:00:00Z',
worktrees: [
{ name: 'main', path: '/tmp/main', branch: 'main', isMain: true },
@@ -581,8 +581,8 @@ describe('Layout', () => {
qualifiedName: 'ws1/project-a',
workspace: 'ws1',
origin: 'workspace',
- badges: [],
- fileCount: 5,
+
+ hasFiles: true,
lastModified: '2026-01-01T00:00:00Z',
worktrees: [
{ name: 'main', path: '/tmp/main', branch: 'main', isMain: true },
@@ -642,8 +642,8 @@ describe('Layout', () => {
workspace: '',
projectPath: '/tmp/bravo',
origin: 'standalone' as const,
- badges: [],
- fileCount: 3,
+
+ hasFiles: true,
lastModified: '2026-01-02T00:00:00Z',
},
{
@@ -652,8 +652,8 @@ describe('Layout', () => {
workspace: '',
projectPath: '/tmp/alpha',
origin: 'standalone' as const,
- badges: [],
- fileCount: 5,
+
+ hasFiles: true,
lastModified: '2026-01-01T00:00:00Z',
},
]);
@@ -696,8 +696,8 @@ describe('Layout', () => {
workspace: '',
projectPath: '/tmp/has-files',
origin: 'standalone' as const,
- badges: [],
- fileCount: 5,
+
+ hasFiles: true,
lastModified: '2026-01-01T00:00:00Z',
},
{
@@ -706,8 +706,8 @@ describe('Layout', () => {
workspace: '',
projectPath: '/tmp/empty-project',
origin: 'standalone' as const,
- badges: [],
- fileCount: 0,
+
+ hasFiles: false,
lastModified: '2026-01-01T00:00:00Z',
},
]);
diff --git a/apps/penpal/frontend/src/components/Layout.tsx b/apps/penpal/frontend/src/components/Layout.tsx
index a3ad6e31c..7e6b24ebd 100644
--- a/apps/penpal/frontend/src/components/Layout.tsx
+++ b/apps/penpal/frontend/src/components/Layout.tsx
@@ -327,13 +327,13 @@ export default function Layout() {
);
// E-PENPAL-SORT: shared comparator — empty projects last, then alpha or API order.
const projectSort = useCallback((a: APIProject, b: APIProject) => {
- if ((a.fileCount > 0) !== (b.fileCount > 0)) return b.fileCount > 0 ? 1 : -1;
+ if (a.hasFiles !== b.hasFiles) return b.hasFiles ? 1 : -1;
if (sortOrder === 'alpha') return a.name.localeCompare(b.name);
return 0;
}, [sortOrder]);
// E-PENPAL-VIEW-OPTIONS: filter empty projects when showEmpty is false
- const filterEmpty = useCallback((p: APIProject) => showEmpty || p.fileCount > 0, [showEmpty]);
+ const filterEmpty = useCallback((p: APIProject) => showEmpty || p.hasFiles, [showEmpty]);
const standaloneProjects = useMemo(() => {
return projects.filter((p) => p.origin === 'standalone').filter(filterEmpty).sort(projectSort);
diff --git a/apps/penpal/frontend/src/pages/FilePage.test.tsx b/apps/penpal/frontend/src/pages/FilePage.test.tsx
index d57fda6d8..734f08dae 100644
--- a/apps/penpal/frontend/src/pages/FilePage.test.tsx
+++ b/apps/penpal/frontend/src/pages/FilePage.test.tsx
@@ -37,7 +37,7 @@ vi.mock('../components/MermaidRenderer', () => ({
function LayoutWrapper() {
const ctx: LayoutContext = {
setHeadings: vi.fn(),
- projects: [{ name: 'proj', qualifiedName: 'ws/proj', workspace: 'ws', origin: 'workspace' as const, badges: [], fileCount: 1, lastModified: '', projectPath: '/tmp/proj' }],
+ projects: [{ name: 'proj', qualifiedName: 'ws/proj', workspace: 'ws', origin: 'workspace' as const, hasFiles: true, lastModified: '', projectPath: '/tmp/proj' }],
};
return ;
}
diff --git a/apps/penpal/frontend/src/types.ts b/apps/penpal/frontend/src/types.ts
index 91386bf9e..ee5dc8742 100644
--- a/apps/penpal/frontend/src/types.ts
+++ b/apps/penpal/frontend/src/types.ts
@@ -1,13 +1,5 @@
// Types matching Go JSON API responses
-export interface APIBadge {
- text: string;
- color: string;
- bg: string;
- activeBg?: string;
- activeColor?: string;
-}
-
export interface APIWorktree {
name: string;
path: string;
@@ -22,10 +14,9 @@ export interface APIProject {
workspacePath?: string;
projectPath: string;
origin: 'workspace' | 'standalone';
- badges: APIBadge[];
branch?: string;
dirty?: boolean;
- fileCount: number;
+ hasFiles: boolean;
lastModified: string;
agentConnected?: boolean;
agentRunning?: boolean;
@@ -147,7 +138,7 @@ export interface ReviewGroup {
}
export interface ProjectInfo {
- fileCount: number;
+ hasFiles: boolean;
dirty: boolean;
unpushedCommits: number;
}
diff --git a/apps/penpal/internal/cache/cache.go b/apps/penpal/internal/cache/cache.go
index 2122261c9..a784dbea2 100644
--- a/apps/penpal/internal/cache/cache.go
+++ b/apps/penpal/internal/cache/cache.go
@@ -3,7 +3,9 @@ package cache
import (
"bufio"
"bytes"
+ "errors"
"io"
+ "io/fs"
"os"
"os/exec"
"path/filepath"
@@ -129,12 +131,15 @@ type Cache struct {
projects []discovery.Project
// projectFiles maps project name to its file list
projectFiles map[string][]FileInfo
+ // projectScanned tracks which projects have had a full file scan
+ projectScanned map[string]bool
}
// New creates a new cache
func New() *Cache {
return &Cache{
- projectFiles: make(map[string][]FileInfo),
+ projectFiles: make(map[string][]FileInfo),
+ projectScanned: make(map[string]bool),
}
}
@@ -262,6 +267,7 @@ func (c *Cache) SetProjectFiles(projectName string, files []FileInfo) {
c.mu.Lock()
defer c.mu.Unlock()
c.projectFiles[projectName] = files
+ c.projectScanned[projectName] = true
}
// ProjectFiles returns the cached file list for a project
@@ -276,14 +282,35 @@ func (c *Cache) ProjectFiles(projectName string) []FileInfo {
return nil
}
-// AllFiles returns all files across all projects, sorted by modification time
+// AllFiles returns all files across all projects, sorted by modification time.
+// Files are deduplicated by project+path: when a file appears in both a typed
+// source and __all_markdown__, only the typed-source entry is kept.
+// E-PENPAL-SRC-ALL-MD: dedup prevents __all_markdown__ from doubling the list.
func (c *Cache) AllFiles(limit int) []FileInfo {
c.mu.RLock()
defer c.mu.RUnlock()
- var all []FileInfo
+ type fileKey struct {
+ project, path string
+ }
+ best := make(map[fileKey]FileInfo)
for _, files := range c.projectFiles {
- all = append(all, files...)
+ for _, f := range files {
+ k := fileKey{f.Project, f.FullPath}
+ if existing, ok := best[k]; ok {
+ // Prefer typed-source entry over __all_markdown__
+ if existing.Source == "__all_markdown__" && f.Source != "__all_markdown__" {
+ best[k] = f
+ }
+ } else {
+ best[k] = f
+ }
+ }
+ }
+
+ all := make([]FileInfo, 0, len(best))
+ for _, f := range best {
+ all = append(all, f)
}
sort.Slice(all, func(i, j int) bool {
@@ -315,6 +342,25 @@ func (c *Cache) FindFile(projectName, filePath string) *FileInfo {
return nil
}
+// EnsureProjectScanned triggers a full file scan for a project if it hasn't
+// been scanned yet. Returns true if a scan was actually performed (first call
+// for this project). This is the lazy-scan entry point — called when a user
+// first opens a project, rather than eagerly at startup.
+// E-PENPAL-SCAN: lazy scan with write-lock gating to prevent duplicate walks.
+func (c *Cache) EnsureProjectScanned(projectName string) bool {
+ c.mu.Lock()
+ if c.projectScanned[projectName] {
+ c.mu.Unlock()
+ return false
+ }
+ // Mark as in-progress under write lock to prevent concurrent scans.
+ c.projectScanned[projectName] = true
+ c.mu.Unlock()
+
+ c.RefreshProject(projectName)
+ return true
+}
+
// RefreshProject rescans a single project's files across all its sources.
// projectName should be the qualified name (e.g., "Development/penpal").
// E-PENPAL-CACHE: walks filesystem and updates per-project file list and metadata.
@@ -330,9 +376,10 @@ func (c *Cache) RefreshProject(projectName string) {
// Update project metadata
c.mu.Lock()
defer c.mu.Unlock()
+ c.projectScanned[projectName] = true
for i := range c.projects {
if c.projects[i].QualifiedName() == projectName {
- c.projects[i].FileCount = len(files)
+ c.projects[i].HasFiles = len(files) > 0
if len(files) > 0 {
c.projects[i].LastModified = files[0].ModTime
}
@@ -356,6 +403,73 @@ func (c *Cache) RefreshAllProjects() {
wg.Wait()
}
+// errFoundMarkdown is a sentinel used to short-circuit filepath.WalkDir
+// once a .md file is found.
+var errFoundMarkdown = errors.New("found markdown")
+
+// CheckAllProjectsHasFiles does a cheap per-project check to set HasFiles
+// without doing a full file scan. For each project, it walks the project
+// root and stops as soon as it finds any .md file.
+// E-PENPAL-SCAN: lightweight startup check — no file list is built.
+func (c *Cache) CheckAllProjectsHasFiles() {
+ projects := c.Projects()
+ var wg sync.WaitGroup
+ for _, p := range projects {
+ wg.Add(1)
+ go func(p discovery.Project) {
+ defer wg.Done()
+ found := projectHasAnyMarkdown(p.Path)
+ c.mu.Lock()
+ for i := range c.projects {
+ if c.projects[i].QualifiedName() == p.QualifiedName() {
+ c.projects[i].HasFiles = found
+ break
+ }
+ }
+ c.mu.Unlock()
+ }(p)
+ }
+ wg.Wait()
+}
+
+// projectHasAnyMarkdown walks the directory tree and returns true as soon as
+// it finds any .md file. Aligns skip behavior with scanProjectSources: skips
+// .git, node_modules, .hg, .svn, nested worktrees/submodules, and gitignored dirs.
+// E-PENPAL-SCAN: lightweight startup check consistent with full scan filtering.
+func projectHasAnyMarkdown(projectPath string) bool {
+ gitChecker := newGitIgnoreChecker(projectPath)
+ defer gitChecker.Close()
+
+ err := filepath.WalkDir(projectPath, func(path string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return nil
+ }
+ if d.IsDir() {
+ name := d.Name()
+ if name == ".git" || name == "node_modules" || name == ".hg" || name == ".svn" {
+ return filepath.SkipDir
+ }
+ // Skip nested git worktrees and submodules (.git file, not dir).
+ if path != projectPath {
+ gitEntry := filepath.Join(path, ".git")
+ if fi, err := os.Lstat(gitEntry); err == nil && !fi.IsDir() {
+ return filepath.SkipDir
+ }
+ }
+ // E-PENPAL-SRC-GITIGNORE: skip gitignored directories.
+ if path != projectPath && gitChecker.IsIgnored(path) {
+ return filepath.SkipDir
+ }
+ return nil
+ }
+ if strings.HasSuffix(d.Name(), ".md") {
+ return errFoundMarkdown
+ }
+ return nil
+ })
+ return errors.Is(err, errFoundMarkdown)
+}
+
// EnrichProject updates a project's git info without rescanning files.
// name should be the qualified name (e.g., "Development/penpal").
func (c *Cache) EnrichProject(name string, git *discovery.GitInfo) {
@@ -519,10 +633,10 @@ func ScanProjectSourcesForWorktree(project *discovery.Project, worktreePath stri
}
// scanProjectSources scans all sources of a project for markdown files.
-// Files are de-duplicated by project-relative path: if multiple sources cover
-// the same file, only the first source's entry is kept. This means auto-detected
-// sources (which come first) take priority over manual ones.
-// E-PENPAL-SCAN: walks filesystem, skips dirs, classifies files, deduplicates, sorts by ModTime.
+// Each source does its own walk. Files are de-duplicated by project-relative
+// path: if multiple sources cover the same file, only the first source's entry
+// is kept — except __all_markdown__ which claims every file regardless.
+// E-PENPAL-SCAN: per-source WalkDir, classifies files, deduplicates, sorts by ModTime.
func scanProjectSources(project *discovery.Project) []FileInfo {
var files []FileInfo
seen := make(map[string]bool) // project-relative paths already claimed
@@ -541,33 +655,30 @@ func scanProjectSources(project *discovery.Project) []FileInfo {
stName = source.Name
}
st := discovery.GetSourceType(stName)
+ isAllMarkdown := source.Name == "__all_markdown__"
- filepath.Walk(rootPath, func(path string, info os.FileInfo, err error) error {
+ filepath.WalkDir(rootPath, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return nil
}
- if info.IsDir() {
+ if d.IsDir() {
// Skip nested git worktrees and submodules: they contain a
// .git file (not directory) pointing at the real gitdir.
- // Without this, tree sources rooted at "." walk into
- // .claude/worktrees// and surface duplicate files.
if path != rootPath {
gitEntry := filepath.Join(path, ".git")
if fi, err := os.Lstat(gitEntry); err == nil && !fi.IsDir() {
return filepath.SkipDir
}
}
- // Never walk into .git (git doesn't report it as ignored).
- if info.Name() == ".git" {
+ if d.Name() == ".git" {
return filepath.SkipDir
}
- // Skip gitignored directories (build output, deps, etc.).
// P-PENPAL-SRC-GITIGNORE: registered source roots are
// always scanned even if gitignored.
if path != rootPath && gitChecker.IsIgnored(path) {
return filepath.SkipDir
}
- if st != nil && st.SkipDirs[info.Name()] {
+ if st != nil && st.SkipDirs[d.Name()] {
return filepath.SkipDir
}
return nil
@@ -585,7 +696,7 @@ func scanProjectSources(project *discovery.Project) []FileInfo {
relToSource, _ := filepath.Rel(rootPath, path)
relToProject, _ := filepath.Rel(project.Path, path)
- if seen[relToProject] {
+ if !isAllMarkdown && seen[relToProject] {
return nil // already claimed by an earlier source
}
@@ -603,9 +714,15 @@ func scanProjectSources(project *discovery.Project) []FileInfo {
}
}
- title := extractTitle(path)
+ // Only stat .md files (for ModTime), not every entry.
+ info, err := d.Info()
+ if err != nil {
+ return nil
+ }
- seen[relToProject] = true
+ if !isAllMarkdown {
+ seen[relToProject] = true
+ }
files = append(files, FileInfo{
Project: project.QualifiedName(),
Workspace: project.WorkspaceName,
@@ -615,8 +732,8 @@ func scanProjectSources(project *discovery.Project) []FileInfo {
SourceAuto: source.Auto,
Path: relToSource,
FullPath: relToProject,
- Name: filepath.Base(path),
- Title: title,
+ Name: d.Name(),
+ Title: extractTitle(path),
ModTime: info.ModTime(),
FileType: fileType,
})
diff --git a/apps/penpal/internal/cache/cache_test.go b/apps/penpal/internal/cache/cache_test.go
index aaf39e95d..19e4e6cf3 100644
--- a/apps/penpal/internal/cache/cache_test.go
+++ b/apps/penpal/internal/cache/cache_test.go
@@ -484,6 +484,143 @@ func TestGitIgnoreChecker_NonGitDir(t *testing.T) {
}
}
+// E-PENPAL-SCAN: verifies CheckAllProjectsHasFiles sets HasFiles correctly.
+func TestCheckAllProjectsHasFiles(t *testing.T) {
+ tmpDir := t.TempDir()
+
+ // Project with markdown files
+ withMD := filepath.Join(tmpDir, "with-md")
+ os.MkdirAll(filepath.Join(withMD, "docs"), 0755)
+ os.WriteFile(filepath.Join(withMD, "docs", "readme.md"), []byte("# Hello"), 0644)
+
+ // Project with no markdown files
+ withoutMD := filepath.Join(tmpDir, "without-md")
+ os.MkdirAll(withoutMD, 0755)
+ os.WriteFile(filepath.Join(withoutMD, "main.go"), []byte("package main"), 0644)
+
+ c := New()
+ c.SetProjects([]discovery.Project{
+ {Name: "has-files", Path: withMD},
+ {Name: "no-files", Path: withoutMD},
+ })
+
+ c.CheckAllProjectsHasFiles()
+
+ projects := c.Projects()
+ for _, p := range projects {
+ switch p.Name {
+ case "has-files":
+ if !p.HasFiles {
+ t.Error("project with .md files should have HasFiles=true")
+ }
+ case "no-files":
+ if p.HasFiles {
+ t.Error("project without .md files should have HasFiles=false")
+ }
+ }
+ }
+}
+
+// E-PENPAL-SCAN: verifies projectHasAnyMarkdown skips gitignored dirs.
+func TestProjectHasAnyMarkdown_SkipsGitignored(t *testing.T) {
+ tmpDir := t.TempDir()
+
+ runGit(t, tmpDir, "init")
+ runGit(t, tmpDir, "config", "user.email", "test@test.com")
+ runGit(t, tmpDir, "config", "user.name", "test")
+
+ // All .md files are in a gitignored directory
+ os.WriteFile(filepath.Join(tmpDir, ".gitignore"), []byte("build/\n"), 0644)
+ os.MkdirAll(filepath.Join(tmpDir, "build"), 0755)
+ os.WriteFile(filepath.Join(tmpDir, "build", "output.md"), []byte("# Gen"), 0644)
+
+ if projectHasAnyMarkdown(tmpDir) {
+ t.Error("expected false: only .md files are in gitignored dir")
+ }
+
+ // Add a non-gitignored .md file and re-check
+ os.WriteFile(filepath.Join(tmpDir, "README.md"), []byte("# README"), 0644)
+ if !projectHasAnyMarkdown(tmpDir) {
+ t.Error("expected true: README.md is not gitignored")
+ }
+}
+
+// E-PENPAL-SCAN: verifies projectHasAnyMarkdown skips .hg and .svn dirs.
+func TestProjectHasAnyMarkdown_SkipsVCSDirs(t *testing.T) {
+ tmpDir := t.TempDir()
+
+ // .md files only in .hg and .svn dirs
+ os.MkdirAll(filepath.Join(tmpDir, ".hg"), 0755)
+ os.WriteFile(filepath.Join(tmpDir, ".hg", "notes.md"), []byte("# HG"), 0644)
+ os.MkdirAll(filepath.Join(tmpDir, ".svn"), 0755)
+ os.WriteFile(filepath.Join(tmpDir, ".svn", "notes.md"), []byte("# SVN"), 0644)
+
+ if projectHasAnyMarkdown(tmpDir) {
+ t.Error("expected false: only .md files are in .hg/.svn dirs")
+ }
+}
+
+// E-PENPAL-SRC-ALL-MD: verifies AllFiles deduplicates __all_markdown__ entries.
+func TestAllFiles_DeduplicatesAllMarkdown(t *testing.T) {
+ c := New()
+ now := time.Now()
+
+ c.SetProjectFiles("proj", []FileInfo{
+ {Project: "proj", Source: "thoughts", FullPath: "thoughts/plan.md", Name: "plan.md", ModTime: now},
+ {Project: "proj", Source: "__all_markdown__", FullPath: "thoughts/plan.md", Name: "plan.md", ModTime: now},
+ {Project: "proj", Source: "__all_markdown__", FullPath: "README.md", Name: "README.md", ModTime: now},
+ })
+
+ files := c.AllFiles(0)
+ if len(files) != 2 {
+ t.Fatalf("expected 2 unique files, got %d", len(files))
+ }
+
+ // The thoughts/plan.md entry should prefer the typed source
+ for _, f := range files {
+ if f.FullPath == "thoughts/plan.md" && f.Source != "thoughts" {
+ t.Errorf("expected source 'thoughts' for thoughts/plan.md, got %q", f.Source)
+ }
+ // README.md only exists in __all_markdown__, so that's fine
+ if f.FullPath == "README.md" && f.Source != "__all_markdown__" {
+ t.Errorf("expected source '__all_markdown__' for README.md, got %q", f.Source)
+ }
+ }
+}
+
+// E-PENPAL-SCAN: verifies EnsureProjectScanned prevents concurrent duplicate scans.
+func TestEnsureProjectScanned_NoDuplicateScans(t *testing.T) {
+ tmpDir := t.TempDir()
+ os.WriteFile(filepath.Join(tmpDir, "notes.md"), []byte("# Notes"), 0644)
+
+ c := New()
+ c.SetProjects([]discovery.Project{
+ {
+ Name: "test",
+ Path: tmpDir,
+ Sources: []discovery.FileSource{
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+ },
+ },
+ })
+
+ // First call should scan
+ if !c.EnsureProjectScanned("test") {
+ t.Error("first call should return true (scan performed)")
+ }
+
+ // Second call should be a no-op
+ if c.EnsureProjectScanned("test") {
+ t.Error("second call should return false (already scanned)")
+ }
+
+ // Files should be populated from the first scan
+ files := c.ProjectFiles("test")
+ if len(files) == 0 {
+ t.Error("expected files after scan")
+ }
+}
+
func runGit(t *testing.T, dir string, args ...string) {
t.Helper()
cmd := exec.Command("git", args...)
diff --git a/apps/penpal/internal/discovery/discovery.go b/apps/penpal/internal/discovery/discovery.go
index 147378bba..bd2b05106 100644
--- a/apps/penpal/internal/discovery/discovery.go
+++ b/apps/penpal/internal/discovery/discovery.go
@@ -162,6 +162,22 @@ func init() {
return "plan"
},
})
+
+ // E-PENPAL-SRC-ALL-MD: catch-all source that finds all .md files in the
+ // project tree. Registered last so typed sources claim their files first.
+ // Always present — the scan simply finds nothing if there are no .md files.
+ RegisterSourceType(&SourceType{
+ Name: "__all_markdown__",
+ ScanMode: "tree",
+ ShowDirHeadings: true,
+ SkipDirs: map[string]bool{
+ "node_modules": true, ".hg": true, ".svn": true,
+ },
+ GroupFiles: func(paths []string) []FileGroup {
+ sort.Strings(paths)
+ return []FileGroup{{Name: "All Markdown", Paths: paths}}
+ },
+ })
}
// classifyRP1Feature classifies a file under work/features/{id}/ by its filename.
@@ -291,15 +307,6 @@ func groupAnchorsPaths(paths []string) []FileGroup {
return result
}
-// Badge holds rendering metadata for a source type badge.
-type Badge struct {
- Text string
- Color string
- Bg string
- ActiveBg string
- ActiveColor string
-}
-
// FileSource represents a set of files to display for a project.
type FileSource struct {
Name string // display name (e.g., "thoughts", "docs")
@@ -318,7 +325,7 @@ type Project struct {
WorkspacePath string // which workspace discovered this (empty for standalone)
WorkspaceName string // display name of the workspace (empty for standalone)
Git *GitInfo
- FileCount int
+ HasFiles bool
LastModified time.Time
Worktrees []Worktree // discovered worktrees for this project
}
@@ -348,27 +355,6 @@ func (p *Project) HasSourceType(name string) bool {
return false
}
-// Badges returns badge metadata for all auto-detected sources.
-func (p *Project) Badges() []Badge {
- var badges []Badge
- for _, s := range p.Sources {
- if !s.Auto {
- continue
- }
- st := GetSourceType(s.Name)
- if st != nil {
- badges = append(badges, Badge{
- Text: st.DisplayName,
- Color: st.BadgeColor,
- Bg: st.BadgeBg,
- ActiveBg: st.BadgeActiveBg,
- ActiveColor: st.BadgeActiveColor,
- })
- }
- }
- return badges
-}
-
// QualifiedName returns the workspace-qualified project identifier
// (e.g., "Development/penpal"). This is the unique key used for
// cache lookups, comment storage, API calls, and SSE events.
@@ -409,6 +395,17 @@ func DetectSources(projectPath string) []FileSource {
}
}
}
+
+ // E-PENPAL-SRC-ALL-MD: always append the catch-all source last so typed
+ // sources claim their files first during scanning.
+ sources = append(sources, FileSource{
+ Name: "__all_markdown__",
+ Type: "tree",
+ SourceTypeName: "__all_markdown__",
+ RootPath: projectPath,
+ Auto: true,
+ })
+
return sources
}
diff --git a/apps/penpal/internal/server/api_manage_test.go b/apps/penpal/internal/server/api_manage_test.go
index 352ab0d28..1be9d2b0e 100644
--- a/apps/penpal/internal/server/api_manage_test.go
+++ b/apps/penpal/internal/server/api_manage_test.go
@@ -384,6 +384,51 @@ func TestAPIDeleteFile_CleansUpCommentSidecar(t *testing.T) {
}
}
+// E-PENPAL-ADD-SOURCE: verifies __all_markdown__ source doesn't block manual file additions.
+// The conflict check must skip __all_markdown__ since it covers the entire project tree.
+func TestAPISources_AddFileNotBlockedByAllMarkdown(t *testing.T) {
+ s, _, _ := testServer(t)
+
+ dir := t.TempDir()
+ os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes"), 0o644)
+
+ // Register as standalone with __all_markdown__ source (always present in real projects)
+ s.cfg.Projects = append(s.cfg.Projects, config.ProjectConfig{Path: dir})
+ s.refreshAfterConfigChange()
+
+ projName := filepath.Base(dir)
+
+ // Verify the project has __all_markdown__ source
+ project := s.cache.FindProject(projName)
+ if project == nil {
+ t.Fatal("project not found after refresh")
+ }
+ hasAllMD := false
+ for _, src := range project.Sources {
+ if src.Name == "__all_markdown__" {
+ hasAllMD = true
+ break
+ }
+ }
+ if !hasAllMD {
+ t.Fatal("project should have __all_markdown__ source")
+ }
+
+ // Adding a file should succeed (not 409) even though __all_markdown__ covers it
+ body, _ := json.Marshal(map[string]string{
+ "project": projName,
+ "path": "notes.md",
+ })
+ req := httptest.NewRequest(http.MethodPost, "/api/sources", bytes.NewReader(body))
+ req.Header.Set("Content-Type", "application/json")
+ rec := httptest.NewRecorder()
+ s.ServeHTTP(rec, req)
+
+ if rec.Code != http.StatusNoContent {
+ t.Fatalf("expected 204, got %d: %s — __all_markdown__ should not block file additions", rec.Code, rec.Body.String())
+ }
+}
+
// E-PENPAL-REMOVE-SOURCE: verifies adding a tree source then removing it via DELETE /api/sources.
func TestAPISources_RemoveTreeSource(t *testing.T) {
s, _, _ := testServer(t)
diff --git a/apps/penpal/internal/server/api_projects_test.go b/apps/penpal/internal/server/api_projects_test.go
index 2684133d3..f9236d064 100644
--- a/apps/penpal/internal/server/api_projects_test.go
+++ b/apps/penpal/internal/server/api_projects_test.go
@@ -11,6 +11,7 @@ import (
"time"
"github.com/loganj/penpal/internal/cache"
+ "github.com/loganj/penpal/internal/discovery"
)
// E-PENPAL-API-ROUTES: verifies GET /api/projects lists discovered projects.
@@ -123,9 +124,11 @@ func TestAPIProjectInfo_ReturnsInfo(t *testing.T) {
s, c, _ := testServer(t)
dir := t.TempDir()
- seedProject(c, "test-proj", dir, []cache.FileInfo{
+ project := seedProject(c, "test-proj", dir, []cache.FileInfo{
{Project: "test-proj", Name: "plan.md", FullPath: "thoughts/plan.md", Source: "thoughts", ModTime: time.Now()},
})
+ project.HasFiles = true
+ c.SetProjects([]discovery.Project{project})
req := httptest.NewRequest(http.MethodGet, "/api/project-info?name=test-proj", nil)
rec := httptest.NewRecorder()
@@ -139,8 +142,8 @@ func TestAPIProjectInfo_ReturnsInfo(t *testing.T) {
if err := json.Unmarshal(rec.Body.Bytes(), &info); err != nil {
t.Fatalf("parse JSON: %v", err)
}
- if info.FileCount != 1 {
- t.Errorf("expected fileCount=1, got %d", info.FileCount)
+ if !info.HasFiles {
+ t.Errorf("expected hasFiles=true, got false")
}
}
diff --git a/apps/penpal/internal/server/grouping_test.go b/apps/penpal/internal/server/grouping_test.go
index 249b49f41..5046f89c1 100644
--- a/apps/penpal/internal/server/grouping_test.go
+++ b/apps/penpal/internal/server/grouping_test.go
@@ -10,25 +10,33 @@ import (
// E-PENPAL-API-ROUTES: verifies RP1 source produces grouped file sections.
func TestBuildFileGroups_RP1Grouped(t *testing.T) {
+ now := time.Now()
project := &discovery.Project{
Name: "test-project",
Path: "/tmp/test",
Sources: []discovery.FileSource{
{Name: "rp1", Type: "tree", SourceTypeName: "rp1", RootPath: "/tmp/test/.rp1", Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
},
}
files := []cache.FileInfo{
- {Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: time.Now()},
- {Source: "rp1", Path: "work/prds/my-prd.md", FullPath: ".rp1/work/prds/my-prd.md", Name: "my-prd.md", FileType: "prd", ModTime: time.Now()},
- {Source: "rp1", Path: "work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: time.Now()},
- {Source: "rp1", Path: "work/features/auth/design.md", FullPath: ".rp1/work/features/auth/design.md", Name: "design.md", FileType: "design", ModTime: time.Now()},
- {Source: "rp1", Path: "work/features/data-layer/tasks.md", FullPath: ".rp1/work/features/data-layer/tasks.md", Name: "tasks.md", FileType: "task", ModTime: time.Now()},
+ {Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now},
+ {Source: "rp1", Path: "work/prds/my-prd.md", FullPath: ".rp1/work/prds/my-prd.md", Name: "my-prd.md", FileType: "prd", ModTime: now},
+ {Source: "rp1", Path: "work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: now},
+ {Source: "rp1", Path: "work/features/auth/design.md", FullPath: ".rp1/work/features/auth/design.md", Name: "design.md", FileType: "design", ModTime: now},
+ {Source: "rp1", Path: "work/features/data-layer/tasks.md", FullPath: ".rp1/work/features/data-layer/tasks.md", Name: "tasks.md", FileType: "task", ModTime: now},
+ // __all_markdown__ claims all files
+ {Source: "__all_markdown__", Path: ".rp1/context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now},
+ {Source: "__all_markdown__", Path: ".rp1/work/prds/my-prd.md", FullPath: ".rp1/work/prds/my-prd.md", Name: "my-prd.md", FileType: "prd", ModTime: now},
+ {Source: "__all_markdown__", Path: ".rp1/work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: now},
+ {Source: "__all_markdown__", Path: ".rp1/work/features/auth/design.md", FullPath: ".rp1/work/features/auth/design.md", Name: "design.md", FileType: "design", ModTime: now},
+ {Source: "__all_markdown__", Path: ".rp1/work/features/data-layer/tasks.md", FullPath: ".rp1/work/features/data-layer/tasks.md", Name: "tasks.md", FileType: "task", ModTime: now},
}
groups := buildFileGroups(project, files)
- // Should have 4 typed groups + 1 "All Markdown" virtual group
+ // Should have 4 typed groups + 1 "All Markdown" group
if len(groups) != 5 {
t.Fatalf("expected 5 groups, got %d", len(groups))
}
@@ -63,17 +71,21 @@ func TestBuildFileGroups_RP1Grouped(t *testing.T) {
// E-PENPAL-API-ROUTES, E-PENPAL-SRC-THOUGHTS: verifies thoughts source produces a single flat group.
func TestBuildFileGroups_ThoughtsFlat(t *testing.T) {
+ now := time.Now()
project := &discovery.Project{
Name: "test-project",
Path: "/tmp/test",
Sources: []discovery.FileSource{
{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
},
}
files := []cache.FileInfo{
- {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: time.Now()},
- {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: time.Now()},
+ {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: now},
+ {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: now},
+ {Source: "__all_markdown__", Path: "thoughts/plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: now},
+ {Source: "__all_markdown__", Path: "thoughts/research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: now},
}
groups := buildFileGroups(project, files)
@@ -95,18 +107,22 @@ func TestBuildFileGroups_ThoughtsFlat(t *testing.T) {
// E-PENPAL-API-ROUTES: verifies multiple sources produce separate groups.
func TestBuildFileGroups_MultipleSources(t *testing.T) {
+ now := time.Now()
project := &discovery.Project{
Name: "test-project",
Path: "/tmp/test",
Sources: []discovery.FileSource{
{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true},
{Name: "rp1", Type: "tree", SourceTypeName: "rp1", RootPath: "/tmp/test/.rp1", Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
},
}
files := []cache.FileInfo{
- {Source: "thoughts", Path: "plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: time.Now()},
- {Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: time.Now()},
+ {Source: "thoughts", Path: "plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: now},
+ {Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now},
+ {Source: "__all_markdown__", Path: "thoughts/plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: now},
+ {Source: "__all_markdown__", Path: ".rp1/context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now},
}
groups := buildFileGroups(project, files)
@@ -126,23 +142,26 @@ func TestBuildFileGroups_MultipleSources(t *testing.T) {
// E-PENPAL-API-ROUTES: verifies empty sources are omitted from groups.
func TestBuildFileGroups_EmptySourceSkipped(t *testing.T) {
+ now := time.Now()
project := &discovery.Project{
Name: "test-project",
Path: "/tmp/test",
Sources: []discovery.FileSource{
{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true},
{Name: "rp1", Type: "tree", SourceTypeName: "rp1", RootPath: "/tmp/test/.rp1", Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
},
}
// Only thoughts has files, rp1 is empty
files := []cache.FileInfo{
- {Source: "thoughts", Path: "plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: time.Now()},
+ {Source: "thoughts", Path: "plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: now},
+ {Source: "__all_markdown__", Path: "thoughts/plan.md", FullPath: "thoughts/plan.md", Name: "plan.md", FileType: "plan", ModTime: now},
}
groups := buildFileGroups(project, files)
- // 1 typed group (empty rp1 skipped) + All Markdown
+ // 1 thoughts group (empty rp1 skipped) + All Markdown
if len(groups) != 2 {
t.Fatalf("expected 2 groups (empty rp1 skipped), got %d", len(groups))
}
@@ -153,19 +172,25 @@ func TestBuildFileGroups_EmptySourceSkipped(t *testing.T) {
// E-PENPAL-ADD-SOURCE, E-PENPAL-SRC-MANUAL: verifies manual source produces directory headings.
func TestBuildFileGroups_ManualSourceDirHeadings(t *testing.T) {
+ now := time.Now()
project := &discovery.Project{
Name: "test-project",
Path: "/tmp/test",
Sources: []discovery.FileSource{
{Name: "docs", Type: "tree", SourceTypeName: "manual", RootPath: "/tmp/test/docs", Auto: false},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
},
}
files := []cache.FileInfo{
- {Source: "docs", Path: "root.md", FullPath: "docs/root.md", Name: "root.md", FileType: "other", ModTime: time.Now()},
- {Source: "docs", Path: "guides/setup.md", FullPath: "docs/guides/setup.md", Name: "setup.md", FileType: "other", ModTime: time.Now()},
- {Source: "docs", Path: "guides/deploy.md", FullPath: "docs/guides/deploy.md", Name: "deploy.md", FileType: "other", ModTime: time.Now()},
- {Source: "docs", Path: "api/endpoints.md", FullPath: "docs/api/endpoints.md", Name: "endpoints.md", FileType: "other", ModTime: time.Now()},
+ {Source: "docs", Path: "root.md", FullPath: "docs/root.md", Name: "root.md", FileType: "other", ModTime: now},
+ {Source: "docs", Path: "guides/setup.md", FullPath: "docs/guides/setup.md", Name: "setup.md", FileType: "other", ModTime: now},
+ {Source: "docs", Path: "guides/deploy.md", FullPath: "docs/guides/deploy.md", Name: "deploy.md", FileType: "other", ModTime: now},
+ {Source: "docs", Path: "api/endpoints.md", FullPath: "docs/api/endpoints.md", Name: "endpoints.md", FileType: "other", ModTime: now},
+ {Source: "__all_markdown__", Path: "docs/root.md", FullPath: "docs/root.md", Name: "root.md", FileType: "other", ModTime: now},
+ {Source: "__all_markdown__", Path: "docs/guides/setup.md", FullPath: "docs/guides/setup.md", Name: "setup.md", FileType: "other", ModTime: now},
+ {Source: "__all_markdown__", Path: "docs/guides/deploy.md", FullPath: "docs/guides/deploy.md", Name: "deploy.md", FileType: "other", ModTime: now},
+ {Source: "__all_markdown__", Path: "docs/api/endpoints.md", FullPath: "docs/api/endpoints.md", Name: "endpoints.md", FileType: "other", ModTime: now},
}
groups := buildFileGroups(project, files)
@@ -197,17 +222,21 @@ func TestBuildFileGroups_ManualSourceDirHeadings(t *testing.T) {
// E-PENPAL-API-ROUTES: verifies file titles flow through to group view.
func TestBuildFileGroups_TitleFlowsThrough(t *testing.T) {
+ now := time.Now()
project := &discovery.Project{
Name: "test-project",
Path: "/tmp/test",
Sources: []discovery.FileSource{
{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
},
}
files := []cache.FileInfo{
- {Source: "thoughts", Path: "plans/my-plan.md", FullPath: "thoughts/plans/my-plan.md", Name: "my-plan.md", Title: "Per-Tab Navigation", FileType: "plan", ModTime: time.Now()},
- {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", Title: "", FileType: "research", ModTime: time.Now()},
+ {Source: "thoughts", Path: "plans/my-plan.md", FullPath: "thoughts/plans/my-plan.md", Name: "my-plan.md", Title: "Per-Tab Navigation", FileType: "plan", ModTime: now},
+ {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", Title: "", FileType: "research", ModTime: now},
+ {Source: "__all_markdown__", Path: "thoughts/plans/my-plan.md", FullPath: "thoughts/plans/my-plan.md", Name: "my-plan.md", Title: "Per-Tab Navigation", FileType: "plan", ModTime: now},
+ {Source: "__all_markdown__", Path: "thoughts/research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", Title: "", FileType: "research", ModTime: now},
}
groups := buildFileGroups(project, files)
@@ -227,17 +256,21 @@ func TestBuildFileGroups_TitleFlowsThrough(t *testing.T) {
// E-PENPAL-API-ROUTES, E-PENPAL-SRC-THOUGHTS: verifies thoughts source does not produce directory headings.
func TestBuildFileGroups_ThoughtsNoDirHeadings(t *testing.T) {
+ now := time.Now()
project := &discovery.Project{
Name: "test-project",
Path: "/tmp/test",
Sources: []discovery.FileSource{
{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
},
}
files := []cache.FileInfo{
- {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: time.Now()},
- {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: time.Now()},
+ {Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: now},
+ {Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: now},
+ {Source: "__all_markdown__", Path: "thoughts/plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: now},
+ {Source: "__all_markdown__", Path: "thoughts/research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: now},
}
groups := buildFileGroups(project, files)
@@ -260,12 +293,17 @@ func TestBuildFileGroups_AllMarkdownVirtual(t *testing.T) {
Path: "/tmp/test",
Sources: []discovery.FileSource{
{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: "/tmp/test/thoughts", Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
},
}
+ // scanProjectSources produces entries for both the typed source and
+ // the __all_markdown__ catch-all (which claims every .md file).
files := []cache.FileInfo{
{Source: "thoughts", Path: "plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: time.Now()},
{Source: "thoughts", Path: "research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: time.Now()},
+ {Source: "__all_markdown__", Path: "thoughts/plans/foo.md", FullPath: "thoughts/plans/foo.md", Name: "foo.md", FileType: "plan", ModTime: time.Now()},
+ {Source: "__all_markdown__", Path: "thoughts/research/bar.md", FullPath: "thoughts/research/bar.md", Name: "bar.md", FileType: "research", ModTime: time.Now()},
}
groups := buildFileGroups(project, files)
diff --git a/apps/penpal/internal/server/integration_test.go b/apps/penpal/internal/server/integration_test.go
index 15af4604b..1f3388847 100644
--- a/apps/penpal/internal/server/integration_test.go
+++ b/apps/penpal/internal/server/integration_test.go
@@ -19,6 +19,7 @@ func TestAPIProjectFiles_ReturnsGroups(t *testing.T) {
project := seedProject(c, projectName, "/tmp/test", nil)
project.Sources = []discovery.FileSource{
{Name: "rp1", Type: "tree", SourceTypeName: "rp1", RootPath: "/tmp/test/.rp1", Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/test", Auto: true},
}
// Re-set projects to include the sources we just added
c.SetProjects([]discovery.Project{project})
@@ -27,6 +28,9 @@ func TestAPIProjectFiles_ReturnsGroups(t *testing.T) {
files := []cache.FileInfo{
{Project: projectName, Source: "rp1", Path: "context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now},
{Project: projectName, Source: "rp1", Path: "work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: now},
+ // __all_markdown__ source claims all files (duplicates typed sources)
+ {Project: projectName, Source: "__all_markdown__", Path: ".rp1/context/index.md", FullPath: ".rp1/context/index.md", Name: "index.md", FileType: "knowledge", ModTime: now},
+ {Project: projectName, Source: "__all_markdown__", Path: ".rp1/work/features/auth/requirements.md", FullPath: ".rp1/work/features/auth/requirements.md", Name: "requirements.md", FileType: "requirement", ModTime: now},
}
c.SetProjectFiles(projectName, files)
diff --git a/apps/penpal/internal/server/manage.go b/apps/penpal/internal/server/manage.go
index ff67e85d6..42dd7e09e 100644
--- a/apps/penpal/internal/server/manage.go
+++ b/apps/penpal/internal/server/manage.go
@@ -325,6 +325,11 @@ func (s *Server) handleAddSource(w http.ResponseWriter, r *http.Request) {
// Check if this file is already covered by an existing source
for _, src := range project.Sources {
+ // Skip the virtual __all_markdown__ source — it covers the whole
+ // project tree but should not block manual file additions.
+ if src.Name == "__all_markdown__" {
+ continue
+ }
if src.Type == "thoughts" || src.Type == "tree" {
if src.RootPath != "" && strings.HasPrefix(absPath, src.RootPath+"/") {
http.Error(w, fmt.Sprintf("file is already included via source %q", src.Name), http.StatusConflict)
diff --git a/apps/penpal/internal/server/search.go b/apps/penpal/internal/server/search.go
index bbcb73e05..1da105f59 100644
--- a/apps/penpal/internal/server/search.go
+++ b/apps/penpal/internal/server/search.go
@@ -79,7 +79,18 @@ func (s *Server) handleAPISearch(w http.ResponseWriter, r *http.Request) {
return nil
}
if info.IsDir() {
- if st != nil && st.SkipDirs[info.Name()] {
+ name := info.Name()
+ // E-PENPAL-SEARCH: skip .git dirs and nested worktrees/submodules.
+ if name == ".git" {
+ return filepath.SkipDir
+ }
+ if path != source.RootPath {
+ gitEntry := filepath.Join(path, ".git")
+ if fi, gErr := os.Lstat(gitEntry); gErr == nil && !fi.IsDir() {
+ return filepath.SkipDir
+ }
+ }
+ if st != nil && st.SkipDirs[name] {
return filepath.SkipDir
}
return nil
diff --git a/apps/penpal/internal/server/server.go b/apps/penpal/internal/server/server.go
index b52dcf668..7dbb8ca78 100644
--- a/apps/penpal/internal/server/server.go
+++ b/apps/penpal/internal/server/server.go
@@ -214,9 +214,8 @@ func (s *Server) handleReady(w http.ResponseWriter, _ *http.Request) {
// populateProjects scans file lists and fills in git info in the background.
// E-PENPAL-GIT-ENRICH: background git enrichment with SSE push.
func (s *Server) populateProjects() {
- s.cache.RefreshAllProjects()
- s.seedRecentActivity()
- log.Printf("Background file scan complete")
+ s.cache.CheckAllProjectsHasFiles()
+ log.Printf("Background hasFiles check complete")
s.readyOnce.Do(func() { close(s.readyCh) }) // signal that essential data is ready
s.watcher.Broadcast(watcher.Event{Type: watcher.EventProjectsChanged})
@@ -253,17 +252,6 @@ func (s *Server) populateProjects() {
s.watcher.Broadcast(watcher.Event{Type: watcher.EventProjectsChanged})
}
-// seedRecentActivity populates the activity tracker with ModTimes from the
-// most recently modified files in the cache. This ensures /recent has data
-// immediately on startup, even for files modified before the server started.
-// E-PENPAL-ACTIVITY: seeds file modification events at startup.
-func (s *Server) seedRecentActivity() {
- files := s.cache.AllFiles(50)
- for _, f := range files {
- s.activity.RecordAt(activity.FileModified, f.Project, f.FullPath, f.ModTime)
- }
-}
-
// E-PENPAL-API-ROUTES: registers all REST endpoints, SPA, SSE, and MCP.
func (s *Server) routes() {
// Main mux (:8080) — API, SSE, MCP, and React SPA
@@ -626,31 +614,6 @@ func buildFileGroups(project *discovery.Project, cachedFiles []cache.FileInfo) [
groups = append(groups, gv)
}
- // E-PENPAL-SRC-ALL-MD: virtual "All Markdown" group with every file, organized by directory.
- // Always present, no dedup against typed sources.
- allMd := FileGroupView{
- Name: "All Markdown",
- Source: "__all_markdown__",
- SourceType: "tree",
- Auto: true,
- }
- for _, f := range cachedFiles {
- allMd.Files = append(allMd.Files, ProjectFile{
- Name: f.Name,
- Title: f.Title,
- Path: f.FullPath,
- Source: f.Source,
- SourceType: f.SourceType,
- ModTime: f.ModTime,
- Age: formatAge(f.ModTime),
- FileType: f.FileType,
- })
- }
- sort.Slice(allMd.Files, func(i, j int) bool {
- return allMd.Files[i].Path < allMd.Files[j].Path
- })
- groups = append(groups, allMd)
-
return groups
}
@@ -697,12 +660,6 @@ func (s *Server) handleEvents(w http.ResponseWriter, r *http.Request) {
// API handlers for dynamic updates
-type APIBadge struct {
- Text string `json:"text"`
- Color string `json:"color"`
- Bg string `json:"bg"`
-}
-
type APIWorktree struct {
Name string `json:"name"`
Path string `json:"path"`
@@ -717,10 +674,9 @@ type APIProject struct {
WorkspacePath string `json:"workspacePath,omitempty"`
ProjectPath string `json:"projectPath"`
Origin string `json:"origin"`
- Badges []APIBadge `json:"badges"`
Branch string `json:"branch,omitempty"`
Dirty bool `json:"dirty,omitempty"`
- FileCount int `json:"fileCount"`
+ HasFiles bool `json:"hasFiles"`
LastModified string `json:"lastModified"`
AgentConnected bool `json:"agentConnected,omitempty"`
AgentRunning bool `json:"agentRunning,omitempty"`
@@ -750,10 +706,6 @@ func (s *Server) handleListAPIProjects(w http.ResponseWriter, r *http.Request) {
result := make([]APIProject, len(projects))
for i, p := range projects {
qn := p.QualifiedName()
- apiBadges := make([]APIBadge, 0)
- for _, b := range p.Badges() {
- apiBadges = append(apiBadges, APIBadge{Text: b.Text, Color: b.Color, Bg: b.Bg})
- }
result[i] = APIProject{
Name: p.Name,
QualifiedName: qn,
@@ -761,8 +713,7 @@ func (s *Server) handleListAPIProjects(w http.ResponseWriter, r *http.Request) {
WorkspacePath: p.WorkspacePath,
ProjectPath: p.Path,
Origin: p.Origin,
- Badges: apiBadges,
- FileCount: p.FileCount,
+ HasFiles: p.HasFiles,
LastModified: p.LastModified.Format(time.RFC3339),
Age: computeProjectAge(p),
AgentConnected: s.agents != nil && s.agents.Status(qn) != nil && s.agents.Status(qn).Running,
@@ -850,8 +801,16 @@ func (s *Server) handleAPIProjectFiles(w http.ResponseWriter, r *http.Request) {
}
cachedFiles = cache.ScanProjectSourcesForWorktree(project, wtPath)
} else {
- // Ensure file titles are populated before serving
- s.cache.EnrichTitles(qualifiedName)
+ // Lazy scan: full file scan happens on first access, not at startup.
+ // E-PENPAL-ACTIVITY: seed recent activity on first scan so /api/recent
+ // populates progressively as projects are opened.
+ if s.cache.EnsureProjectScanned(qualifiedName) {
+ for _, f := range s.cache.ProjectFiles(qualifiedName) {
+ if f.Source != "__all_markdown__" {
+ s.activity.RecordAt(activity.FileModified, f.Project, f.FullPath, f.ModTime)
+ }
+ }
+ }
cachedFiles = s.cache.ProjectFiles(qualifiedName)
}
fileGroups := buildFileGroups(project, cachedFiles)
@@ -945,7 +904,7 @@ func (s *Server) handleCopyFile(w http.ResponseWriter, r *http.Request) {
}
type ProjectInfo struct {
- FileCount int `json:"fileCount"`
+ HasFiles bool `json:"hasFiles"`
Dirty bool `json:"dirty"`
UnpushedCommits int `json:"unpushedCommits"`
}
@@ -964,7 +923,7 @@ func (s *Server) handleProjectInfo(w http.ResponseWriter, r *http.Request) {
}
info := ProjectInfo{
- FileCount: len(s.cache.ProjectFiles(qualifiedName)),
+ HasFiles: project.HasFiles,
}
// Fresh git status