Script and GitHub Action to maintain a Wiki frontend (#303)

* Created workflow to update wiki with documents from repository

* Allowed for manual running of action

* Included Adding of petshop.md

* Added petshop surface-level files

* Excluded template from wiki

* Changed files to match consistency of other markdown files

* Added pet shop files to path

* Excluded all example code folders and pruned empty folders

* Added steps to run python script

* Created Python script to form files for Wiki from repo

* Changed to double-quote string

* Combined methods with similar logic

* Turned comment into doc string

* Returned to single quotes

* Simplified enumeration and string strip

* Added type hints and docs

* Added docs temp folder to gitignore

* Added type hint

* Moved call to slugify to extract title

* Simplified markdown mapping

* Created dataclass structure

* Added parent to markdownfile

* Created second structure class, FilePath

* Made MarkdownFile subclass FilePath

* Changed dataclass to use properties and store dirpath

* Ran linter on files

* Made dirpath not private

* Used dataclass provided init

* Restructured with dataclasses and simplified logic

* Minor rename

* Removed redundant variables

* Fixed markdown extension

* Added type hints

* Passed link to variable to reduce code

* File refactors and helper methods

* Removed comments

* Deleted old file instead of renaming and then deleting in case of case-only change

* Added check for system's case sensitivity

* Added info as parameter and moved method out of loop

* Removed name maps since links cannot work unless relative or absolute

* Simplified with dictionary comprehension

* Inlined method

* Returned name map for markdown

* Returned back to tuple and name map

* Created initial _Sidebar.md, which is included in Wiki

* Added markdown newline spacers

* Added markdown newline spacers

* Removed pathing from push, as all files work together

* Added spaces lost to merge

* Simplified to always delete old_path before writing to new file if name changed at all

* Remove makedirs since only file renaming happens

* Markdown lint

* Made links to repo folders link to base repo

* Added extra base cases for no extension links

* Added extra base cases for no extension links

* Changed where Home base case is handled

* Changed Home to early return

* Added print statements to rewrite

* Added boolean prints

* Updated boolean prints

* Changed to compare to empty string instead of None

* Added phone number base case to links

* Added explicit group 0

* Added case for links within angle brackets

* Removed example-code case from direct link

* Changed to link to any extension found

* Simplified boolean logic

* Removed CODE_EXTS

* Changed base cases to a regex match array to simplify extensibility

* Added regex comment on groups

* Removed unnecessary parameter

* Removed if blocks

* Added a dash to titles with Phase additions

* Created TupleMapName for simpler map creation and accessing

* Added prints to code to see if partition is necessary

* Changed structure to do a string replacement instead of rebuilding

* Removed default dict values

* Returned to old link structure

* Returned from TupleNameMap structure

* Added more useful prints than previous

* Only print if due to sep being empty

* Only print if not caught by any case

* Added to else block to actually only print if passed

* Added Home with anchor as base case

* Added local anchor base case

* Joined base cases

* Removed case of No new link

* Returned to modified link changing

* Simplified local anchor base case

* Added wrong case prints

* Added an re.escape to re.sub

* Returned test TupleNameMap structure

* Added secondary structure to test

* Added actual new value to prints

* Moved print to init

* Added prints of whole structures

* Moved prints to get method

* Added None base case

* Printed results

* Fixed tuple building

* Returned to new structure

* Minor refactors

* Added more supported embed extensions

* Created file to contain repo-specific rewrite rules

* Moved EMBED_EXTS to rules file

* Moved logic to name the file from extracted data to rules

* Added module docstring

* Made wiki_page_title add the file extension

* Added EDIT_FILE_EXTS

* Changed extension removal to account for any given extensions

* Renamed elements to match generalization

* Added class docstrings

* Moved slugify to structure

* Made title not slugified until after helper function

* Added access to rewrite_rules to slugify if needed

* Extracted logic for getting link to second helper method to use match return

* Made extrac_title_and_body part of rules file

* Took imports from structure's shared code

* Added os and re imports per file

* Changed so Wiki structure is kept within its own folder

* Added -a flag to copy

* Added dot to path

* Moved python scripts to subfolder

* Made Sidebar links relative to root

* Renamed Sidebar links

* Fixed link not updated

* Used Em Dash for Getting Started rename

* Added links to the Auto-Grader and Help Queue

* Added some Phase 3 relevant links

* Changed to Hyphen character

* Added char const for special hypen

* Made title link to base page

* Changed to title being added

* Added newline spaces

* Removed temporary Phase 3 _Sidebar

Author: TheDavSmasher

.github/scripts/rewrite.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python3
+import sys
+import os
+import re
+from structure import ContentFilePath, FilePath, TupleNameMap, slugify
+from rewrite_rules import (LINK_BASE_CASES, EMBED_EXTS, EDIT_FILE_EXTS,
+    wiki_page_title, extract_title_and_body)
+
+
+def get_ext(path: str):
+    """Get the file extension, if present"""
+    return path.lower().rsplit('.', 1)[-1] if '.' in path else ''
+
+def find_files_with_exts(root: str, *exts: str):
+    """Yield FilePaths for files of the given extensions."""
+    for path_from_root, _, files in os.walk(root):
+        for f in files:
+            if get_ext(f) in exts:
+                yield FilePath(path_from_root, f)
+
+def main(root: str, code_base: str):
+    mapping: dict[str, ContentFilePath] = {}
+    for file_path in find_files_with_exts(root, *EDIT_FILE_EXTS):
+        title, body = extract_title_and_body(file_path.full_path)
+        filename = slugify(wiki_page_title(title, body, file_path))
+
+        mapping[file_path.full_path] = ContentFilePath(file_path.dirpath, filename, body)
+
+    edited_map = TupleNameMap(
+        (info.parent, os.path.basename(old_path), info.filename)
+        for (old_path, info) in mapping.items())
+
+    embed_map = TupleNameMap(
+        (file_path.parent, file_path.filename, file_path.relative_from(root))
+        for file_path in find_files_with_exts(root, *EMBED_EXTS))
+
+    # Groups: 1) Links inside <>, 2) All other links
+    link_re = re.compile(r'\[(?:[^\]]+)\]\((?:<([^>]+)>|((?:[^()\\]|\\[()])+))\)')
+
+    base_re = re.compile('|'.join(LINK_BASE_CASES))
+
+    clean_ext_pattern = r'\.(' + '|'.join(EDIT_FILE_EXTS) + r')$'
+
+    def extract_new_link(info: ContentFilePath, path_part: str) -> str:
+        dirname = os.path.basename(os.path.dirname(path_part))
+        basename = os.path.basename(path_part)
+
+        match get_ext(basename):
+            case embed if embed in EMBED_EXTS:
+                return embed_map.get(dirname, info.parent, basename)
+            case edit if edit in EDIT_FILE_EXTS:
+                new_base = edited_map.get(dirname, info.parent, basename)
+                return re.sub(clean_ext_pattern, '', new_base, flags=re.IGNORECASE)
+            case _:
+                abs_path = os.path.normpath(os.path.join(info.dirpath, path_part))
+                rel_to_root = os.path.relpath(abs_path, root)
+
+                if code_base.startswith(('http://', 'https://')):
+                    return code_base.rstrip('/') + '/' + rel_to_root
+                return os.path.normpath(os.path.join(code_base, rel_to_root))
+
+    def rewrite_line(line: str, info: ContentFilePath) -> str:
+        def repl(m: re.Match[str]) -> str:
+            target = m.group(1) or m.group(2)
+
+            if base_re.search(target):
+                return m.group(0)
+
+            path_part = target.partition('#')[0]
+            new_link = extract_new_link(info, path_part)
+
+            return re.sub(re.escape(path_part), new_link, m.group(0), 1)
+
+        return link_re.sub(repl, line)
+
+    for old_path, info in mapping.items():
+        new_body = [rewrite_line(ln, info) for ln in info.body]
+
+        if old_path != info.full_path:
+            os.remove(old_path)
+
+        with open(info.full_path, 'w', encoding='utf-8') as f:
+            f.writelines(new_body)
+
+if __name__ == '__main__':
+    if len(sys.argv) != 3:
+        print("Usage: rewrite.py <root-markdown-dir> <code-base-path-or-URL>")
+        sys.exit(1)
+    main(sys.argv[1], sys.argv[2])

.github/scripts/rewrite_rules.py

@@ -0,0 +1,57 @@
+"""
+This module provides a common place to change the specific information of the rewrite script
+to allow its reuse in other repositories as desired.
+"""
+import re
+from structure import FilePath, slugify     # pylint: disable=W0611
+
+GHWT_HYPHEN = '‐'
+"""Holds the U+2010 hyphen character, which correctly renders
+as a hyphen when used inside a GitHub Wiki page title"""
+
+LINK_BASE_CASES = [r':\/\/', r'^Home#?', r'^tel:.*', r'^mailto:.*', r'^#']
+"""List of regex strings that will leave a markdown link unchanged
+when transformed for wiki usage."""
+
+EMBED_EXTS = {'png', 'gif', 'jpg', 'jpeg', 'svg', 'webp', 'uml', 'mp4', 'mov', 'webm'}
+"""File extensions that are used inside Markdown files to embed."""
+
+EDIT_FILE_EXTS = {'md'}
+"""File extensions that will be modified by the script."""
+
+def wiki_page_title(title: str | None, body: list[str], file_path: FilePath) -> str:
+    # pylint: disable=W0613
+    """
+    Takes in the extracted file title if found, the file's remaining text, and the file path info.
+
+    Returns the new title for the editing file, which will be slugified and used to name the page
+    when used in the wiki, including the file extension.
+    
+    The logic to find the title is defined in extract_title_and_body.
+    """
+    if title:
+        phase_dir = re.search(r'(\d+)', file_path.parent)
+        if file_path.filename == 'getting-started.md' and phase_dir:
+            return f"{title} {GHWT_HYPHEN} Phase {phase_dir.group(1)}.md"
+        return title + '.md'
+    return file_path.filename
+
+def extract_title_and_body(path: str) -> tuple[str | None, list[str]]:
+    """
+    Takes in the full path to the file being accessed.
+
+    Returns (title, body_lines) as extracted from the file, and the rest of the file.
+    - title can be None if file's name must remain unchanged or doesn't match pattern.
+    """
+    with open(path, encoding='utf-8') as f:
+        lines = f.readlines()
+
+    line_number, first_line = next((i, ln) for (i, ln) in enumerate(lines) if ln.strip())
+    if not first_line.lstrip().startswith('# '):
+        return None, lines
+    title = first_line.strip()[2:]
+
+    j = line_number + 1
+    while j < len(lines) and lines[j].strip() == '':
+        j += 1
+    return title, lines[j:]

.github/scripts/structure.py

@@ -0,0 +1,61 @@
+from dataclasses import dataclass
+import os
+import re
+from typing import List, Iterator
+
+
+@dataclass
+class FilePath:
+    """Represents the path to a file from the source of the project."""
+    dirpath: str
+    """Path from root to parent directory of file"""
+    filename: str
+    """Name of the file"""
+
+    @property
+    def full_path(self) -> str:
+        """Full path from root to the file"""
+        return os.path.join(self.dirpath, self.filename)
+
+    @property
+    def parent(self) -> str:
+        """Immediate parent directory"""
+        return os.path.basename(self.dirpath)
+
+    def relative_from(self, root: str):
+        return os.path.relpath(self.full_path, root)
+
+
+@dataclass
+class ContentFilePath(FilePath):
+    """Represents a FilePath with a reference to the contents of such file."""
+    body: List[str]
+    """All content of the file as separate lines"""
+
+
+@dataclass(init=False)
+class TupleNameMap:
+    """Represents a conjoined map from an old filename to it's new name,
+    relative to room or just by name alone."""
+    _tuple_map: dict[tuple[str, str], str]
+    _name_map: dict[str, str]
+
+    def __init__(self, mapper: Iterator[tuple[str, str, str]]):
+        self._tuple_map = {}
+        self._name_map = {}
+        for parent, name, value in mapper:
+            self._tuple_map[(parent, name)] = value
+            self._name_map[name] = value
+
+    def get(self, dirpath: str, parent: str, name: str) -> str | None:
+        return (self._tuple_map.get((dirpath, name)) or
+                self._tuple_map.get((parent, name)) or
+                self._name_map.get(name))
+
+
+def slugify(name: str) -> str:
+    """Remove filesystem-unfriendly chars"""
+    name = name.strip()
+    name = re.sub(r'[\\/:"*?<>|]+', '', name)
+    name = re.sub(r'\s+', '-', name)
+    return name

.github/structure/_Sidebar.md

@@ -0,0 +1,35 @@
+[**Home**](Home)  
+[**Schedule**](Home#outcomes) <!--If structure for schedules is made, they could be moved to wiki-->  
+[**Syllabus**](/instruction/syllabus/syllabus.md)  
+[**Pet Shop**](/petshop/petshop.md)  
+
+[**Auto-Grader**](https://cs240.click)  
+[**Help Queue**](https://help.cs240.click)  
+
+## Chess
+
+[**Chess Application**](/chess/chess.md)  
+<!--Chess Assignments-->
+[**GitHub Repository**](/chess/chess-github-repository/chess-github-repository.md)  
+[**Phase 0**](/chess/0-chess-moves/chess-moves.md)  
+[**Phase 1**](/chess/1-chess-game/chess-game.md)  
+[**Phase 2**](/chess/2-server-design/server-design.md)  
+[**Phase 3**](/chess/3-web-api/web-api.md)  
+[**Phase 4**](/chess/4-database/database.md)  
+[**Phase 5**](/chess/5-pregame/pregame.md)  
+[**Phase 6**](/chess/6-gameplay/gameplay.md)  
+<!--I don't think we need to link to getting started directly through here?-->
+
+## Instruction
+
+[**Instructional topics**](/instruction/modules.md)  
+<!--Write out topics in either alphabetical or instruction order, or only parent links within modules.md-->
+
+<!--Files not listed:
+  - Chess: Code Quality Rubric
+  - Phase 0: Game of Chess
+  - Phase 3: TA Tips
+  - Phase 4: Debugging Tips
+  - Chess Phases: Getting Started (6)
+  - All 40 instruction topics
+-->

.github/workflows/update_on_push.yml

@@ -0,0 +1,43 @@
+name: Wiki Content Sync
+
+on:
+  push:
+    branches:
+      - main
+  repository_dispatch:
+    types: [docs]
+  workflow_dispatch: 
+  gollum:
+  
+env:
+  GIT_AUTHOR_NAME: GitHub Action
+  GIT_AUTHOR_EMAIL: action@github.com
+  GITHUB_REPO: ${{ github.server_url }}/${{ github.repository }}/blob/main
+
+jobs:
+  sync-content-to-wiki:
+    if: always() && format('refs/heads/{0}', github.event.repository.default_branch) == github.ref && github.event_name != 'gollum'
+    runs-on: [ ubuntu-latest ]
+    steps:
+      - name: Checkout Repo
+        uses: actions/checkout@v4
+      - name: Group Document files
+        run: |
+          rsync -av --exclude='.git*' --exclude='LICENSE' --include='petshop/*' --exclude='petshop/*/*' --exclude='*/example-code/' --exclude='schedule' --exclude='README.md' --exclude='instruction/template' ./ docs/
+          find docs -empty -type d -delete
+          cp README.md docs/Home.md
+          cp -ar .github/structure/. docs/
+      - name: Install dependencies for renaming
+        run: |
+          python3 -m pip install --upgrade pip
+      - name: Rewrite markdown titles, filenames, and links
+        run: |
+          python3 .github/scripts/rewrite.py docs $GITHUB_REPO
+      - name: Sync docs to wiki
+        uses: newrelic/wiki-sync-action@v1.0.1
+        with:
+          source: docs
+          destination: wiki
+          token: ${{ secrets.WIKI_SYNC_SECRET }}
+          gitAuthorName: ${{ env.GIT_AUTHOR_NAME }}
+          gitAuthorEmail: ${{ env.GIT_AUTHOR_EMAIL }}

.gitignore

@@ -1,2 +1,5 @@
 .DS_Store
 .idea/
+
+# Used to store files while they're modified for Wiki.
+docs