Skip to content

ci: parallelize sample compilation #1354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HecreReed wants to merge 1 commit into
base: main
Choose a base branch
from
Open

ci: parallelize sample compilation #1354

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,7 @@ jobs:
PTOBC_BIN: ${{ env.PTO_BUILD_DIR }}/tools/ptobc/ptobc
PTO_BUILD_DIR: ${{ env.PTO_BUILD_DIR }}
PYTHON_BIN: ${{ env.PTOAS_VENV }}/bin/python
PTOAS_SAMPLE_JOBS: "4"
run: |
set -euo pipefail
export PATH="${PTOAS_VENV}/bin:${PATH}"
Expand Down
38 changes: 36 additions & 2 deletions test/samples/runop.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ PTO_BUILD_DIR="${PTO_BUILD_DIR:-}"
PTOAS_ENABLE_INSERT_SYNC="${PTOAS_ENABLE_INSERT_SYNC:-1}"
PTOAS_FLAGS="${PTOAS_FLAGS:-}"
PTOAS_SKIP_CASES="${PTOAS_SKIP_CASES:-}"
PTOAS_SAMPLE_JOBS="${PTOAS_SAMPLE_JOBS:-1}"
PTOAS_SKIP_CASES_NORM="$(printf '%s\n' "${PTOAS_SKIP_CASES}" | tr ',[:space:]' '\n' | awk 'NF')"
MODEL_PTO_DIRS=""
for model_path in "${BASE_DIR}"/Qwen* "${BASE_DIR}"/Deepseek*; do
Expand Down Expand Up @@ -53,6 +54,7 @@ Env:
PTOAS_FLAGS # extra flags passed to ptoas (e.g. --enable-insert-sync)
PTOAS_ENABLE_INSERT_SYNC # 1 to append --enable-insert-sync to PTOAS_FLAGS (default: 1)
PTOAS_SKIP_CASES # comma/space-separated testcase basenames to skip while generating outputs
PTOAS_SAMPLE_JOBS # number of sample directories to process concurrently (default: 1)
PTO_PTO_DIRS # space-separated dirs to run .pto directly (default: Sync, every Qwen*/Deepseek* A3/A5 model dir, and the legacy direct-PTO dirs)

Flags:
Expand Down Expand Up @@ -1600,8 +1602,30 @@ write_board_case_manifest() {
echo "BOARD_CASE_MANIFEST=${manifest} ($(wc -l < "${manifest}") cases)"
}

wait_for_sample_batch() {
local summary_file="$1"
shift

local pid result_file
while [[ $# -ge 2 ]]; do
pid="$1"
result_file="$2"
shift 2
wait "${pid}" || true

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Propagate background worker failures

When a background process_one_dir is killed—for example by OOM under the new concurrency—or aborts because of set -u, this discards its nonzero status and aggregates only its partial log. If the worker exits before printing a FAIL record, the summary omits that directory and returns success; reproducing this with one killed worker and one successful worker produced FAIL=0 and exit code 0. Record a synthetic failure or otherwise propagate failed wait statuses into the final result.

Useful? React with 👍 / 👎.

cat "${result_file}" >>"${summary_file}"
done
}

run_all() {
local tmp out_dir summary_rc soc_arch="" dir_name dir_arch
local tmp out_dir result_dir result_file summary_rc soc_arch="" dir_name dir_arch
local dir_index=0
local -a batch=()

if [[ ! "${PTOAS_SAMPLE_JOBS}" =~ ^[1-9][0-9]*$ ]]; then
echo "PTOAS_SAMPLE_JOBS must be a positive integer, got: ${PTOAS_SAMPLE_JOBS}" >&2
return 2
fi

out_dir="${PTOAS_OUT_DIR}"
if [[ -z "${out_dir}" ]]; then
out_dir="$(mktemp -d -t ptoas.samples.XXXXXX)"
Expand All @@ -1612,6 +1636,7 @@ run_all() {
echo "PTOAS_OUT_DIR=${out_dir}"

tmp="$(mktemp -t ptoas.runop.XXXXXX)"
result_dir="$(mktemp -d -t ptoas.runop.results.XXXXXX)"
if [[ -n "${SOC_VERSION:-}" ]]; then
local soc_lc
soc_lc="$(printf '%s' "${SOC_VERSION}" | tr '[:upper:]' '[:lower:]')"
Expand All @@ -1628,8 +1653,17 @@ run_all() {
if [[ -n "${soc_arch}" && -n "${dir_arch}" && "${dir_arch}" != "${soc_arch}" ]]; then
continue
fi
process_one_dir "${dir_name}" "$out_dir" >>"$tmp"
result_file="${result_dir}/${dir_index}.log"
process_one_dir "${dir_name}" "$out_dir" >"${result_file}" &
batch+=("$!" "${result_file}")
dir_index=$((dir_index + 1))
if [[ ${#batch[@]} -ge $((PTOAS_SAMPLE_JOBS * 2)) ]]; then
wait_for_sample_batch "${tmp}" "${batch[@]}"
batch=()
fi
done
wait_for_sample_batch "${tmp}" "${batch[@]}"
rm -rf -- "${result_dir}"

echo "========== SUMMARY =========="
sort "$tmp" | awk -F'\t' '
Expand Down
Loading