Skip to content

fix graph_net/tools/generate_subgraph_dataset #603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Xreki merged 5 commits into from
Feb 2, 2026
+112 −33
Merged

fix graph_net/tools/generate_subgraph_dataset #603

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9376a35
fix graph_net/tools/generate_subgraph_dataset
Honglei-Qiu Jan 23, 2026
4d2226f
fix graph_net/tools/generate_subgraph_dataset
Honglei-Qiu Jan 23, 2026
b32363a
fix graph_net/tools/generate_subgraph_dataset
Honglei-Qiu Jan 23, 2026
98fd62a
fix generate_subgraph_dataset.sh
Honglei-Qiu Feb 2, 2026
2b15861
Merge branch 'develop' into feature/t4
Honglei-Qiu Feb 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 112 additions & 33 deletions graph_net/tools/generate_subgraph_dataset.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ set -x
MIN_SEQ_OPS=${1:-16}
MAX_SEQ_OPS=${2:-64}
GPU_ID=${3:-0}
LIMITS_HANDLED_MODELS=${4:-40}

OP_RANGE=$MIN_SEQ_OPS-$MAX_SEQ_OPS

Expand All @@ -14,6 +15,7 @@ RESUME="true"

DECOMPOSE_WORKSPACE=/tmp/subgraph_dataset_workspace
DEVICE_REWRITED_OUTPUT_DIR=$DECOMPOSE_WORKSPACE/device_rewrited
DIMENSION_GENERALIZER_OUTPUT_DIR=$DECOMPOSE_WORKSPACE/dimension_generalized_samples
OP_NAMES_OUTPUT_DIR=$DECOMPOSE_WORKSPACE/sample_op_names
SPLIT_POINTS_OUTPUT_DIR=$DECOMPOSE_WORKSPACE/split_points
RANGE_DECOMPOSE_OUTPUT_DIR=$DECOMPOSE_WORKSPACE/range_decompose
Expand All @@ -25,16 +27,39 @@ GROUPED_FUSIBLE_SUBGRAPH_RANGES_DIR=$DECOMPOSE_WORKSPACE/grouped_fusible_subgrap
FUSIBLE_SUBGRAPH_SAMPLES_DIR=$DECOMPOSE_WORKSPACE/fusible_subgraph_samples
RENAMED_FUSIBLE_SUBGRAPH_DIR=$DECOMPOSE_WORKSPACE/renamed_fusible_subgraphs
DEDUPLICATED_FUSIBLE_SUBGRAPH_DIR=$DECOMPOSE_WORKSPACE/deduplicated_fusible_subgraphs
DIMENSION_SUBGRAPH_SAMPLES_OUTPUT_DIR=$DECOMPOSE_WORKSPACE/workspace_dimension_subgraph_samples
UNITTESTS_OUTPUT_DIR=$DECOMPOSE_WORKSPACE/unittests

mkdir -p "$DECOMPOSE_WORKSPACE"

model_list="$GRAPH_NET_ROOT/graph_net/config/small100_torch_samples_list.txt"
model_list="$GRAPH_NET_ROOT/graph_net/config/small100_torch_samples_list.txt"
range_decomposed_subgraph_list=${DECOMPOSE_WORKSPACE}/range_decomposed_subgraph_sample_list.txt
deduplicated_subgraph_list=${DECOMPOSE_WORKSPACE}/deduplicated_subgraph_sample_list.txt
device_rewrited_subgraph_list=${DECOMPOSE_WORKSPACE}/device_rewrited_subgraph_sample_list.txt
fusible_subgraph_list=${DECOMPOSE_WORKSPACE}/fusible_subgraph_sample_list.txt
deduplicated_fusible_subgraphs_list=${DECOMPOSE_WORKSPACE}/deduplicated_fusible_subgraph_sample_list.txt
group_subgraph_sources_list=${DECOMPOSE_WORKSPACE}/group_subgraph_sources_sample_list.txt
dimension_generalizer_samples_list=dimension_generalizer_sample_list.txt
model_runnable_predicator=ModelRunnablePredicator

@Xreki Xreki Jan 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

该运行配置并没有给多个步骤复用，可直接写到对应的命令里面

@Honglei-Qiu Honglei-Qiu Jan 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

@Honglei-Qiu Honglei-Qiu Jan 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我觉得可以保留着，方便统一格式

@Honglei-Qiu Honglei-Qiu Jan 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我先使用了保留的方式，如果确实需要写到命令里，后续在修改


function generate_subgraph_list_by_index() {
local target_dir="$1"
local sample_list_name="$2"
local max_index="$3"
echo ">>> Generate subgraph_sample_list for samples under ${target_dir} with index 0-${max_index}."
echo ">>>"
for index in $(seq 0 $max_index); do
local sample_list="${target_dir}/${index}/${sample_list_name}"
echo ">>> Generating list for index ${index}"

find ${target_dir}/${index} -name "model.py" \
| xargs dirname \
| xargs realpath --relative-to=${target_dir}/${index} \
| tee $sample_list

echo "Generated: $sample_list"
done
}

function generate_subgraph_list() {
local target_dir="$1"
Expand All @@ -50,37 +75,57 @@ function generate_subgraph_list() {
}

function rewrite_device() {
echo ">>> [1] Rewrite devices for samples in ${model_list}."
echo ">>> [1] Rewrite devices for subgraph samples under ${GRAPH_NET_ROOT}."
echo ">>>"
python3 -m graph_net.model_path_handler \
--model-path-list $model_list \
--model-path-list ${model_list} \
--handler-config=$(base64 -w 0 <<EOF
{
"handler_path": "$GRAPH_NET_ROOT/graph_net/torch/sample_pass/device_rewrite_sample_pass.py",
"handler_class_name": "DeviceRewriteSamplePass",
"handler_config": {
"device": "cuda",
"resume": ${RESUME},
"model_path_prefix": "$GRAPH_NET_ROOT",
"model_path_prefix": "${GRAPH_NET_ROOT}",
"output_dir": "${DEVICE_REWRITED_OUTPUT_DIR}"
}
}
EOF
)
}

function generate_op_names() {
echo ">>> [2] Generate op_names.txt for samples in ${model_list}."
function dimension_generalizer(){
echo ">>> [2] dimension generalize for samoles under ${device_rewrited_subgraph_list}."
echo ">>>"
python3 -m graph_net.apply_sample_pass \
--model-path-list $device_rewrited_subgraph_list \
--sample-pass-file-path "$GRAPH_NET_ROOT/graph_net/dimension_generalizer.py" \
--sample-pass-class-name "ApplyDimGenPasses" \
--sample-pass-config $(base64 -w 0 <<EOF
{
"output_dir": "${DIMENSION_GENERALIZER_OUTPUT_DIR}",
"model_path_prefix": "$DEVICE_REWRITED_OUTPUT_DIR",
"dimension_generalizer_filepath": "$GRAPH_NET_ROOT/graph_net/torch/static_to_dynamic.py",
"dimension_generalizer_class_name": "StaticToDynamic",
"limits_handled_models": ${LIMITS_HANDLED_MODELS},
"last_model_log_file": "/tmp/a.py"
}
EOF
)
}

function generate_op_names() {
echo ">>> [3] Generate op_names.txt for samples in ${device_rewrited_subgraph_list}."
echo ">>>"
python3 -m graph_net.model_path_handler \
--model-path-list $model_list \
--model-path-list $device_rewrited_subgraph_list \
--handler-config=$(base64 -w 0 <<EOF
{
"handler_path": "$GRAPH_NET_ROOT/graph_net/torch/sample_pass/op_names_extractor.py",
"handler_class_name": "OpNamesExtractor",
"handler_config": {
"resume": ${RESUME},
"model_path_prefix": "${DEVICE_REWRITED_OUTPUT_DIR}",
"model_path_prefix": "$DEVICE_REWRITED_OUTPUT_DIR",
"output_dir": "${OP_NAMES_OUTPUT_DIR}"
}
}
Expand All @@ -89,17 +134,17 @@ EOF
}

function generate_split_point() {
echo ">>> [3] Generate split points for samples in ${model_list}."
echo ">>> [4] Generate split points for samples in ${device_rewrited_subgraph_list}."
echo ">>> MIN_SEQ_OPS: ${MIN_SEQ_OPS}, MAX_SEQ_OPS: ${MAX_SEQ_OPS}"
echo ">>>"
python3 -m graph_net.apply_sample_pass \
--model-path-list $model_list \
--model-path-list $device_rewrited_subgraph_list \
--sample-pass-file-path $GRAPH_NET_ROOT/graph_net/torch/sample_pass/typical_sequence_split_points.py \
--sample-pass-class-name TypicalSequenceSplitPointsGenerator \
--sample-pass-config=$(base64 -w 0 <<EOF
{
"model_path_prefix": "$GRAPH_NET_ROOT",
"output_dir": "$SPLIT_POINTS_OUTPUT_DIR",
"model_path_prefix": "$DEVICE_REWRITED_OUTPUT_DIR",
"output_dir": "$DECOMPOSE_WORKSPACE",
"op_names_path_prefix": "${OP_NAMES_OUTPUT_DIR}",
"device": "cuda",
"window_size": 64,
Expand All @@ -116,17 +161,17 @@ EOF
}

function range_decompose() {
echo ">>> [4] Decompose according to split_results.json for samples in ${model_list}."
echo ">>> [5] Decompose according to split_results.json for samples in ${device_rewrited_subgraph_list}."
echo ">>>"
python3 -m graph_net.model_path_handler \
--model-path-list "$model_list" \
--model-path-list "$device_rewrited_subgraph_list" \
--handler-config=$(base64 -w 0 <<EOF
{
"handler_path": "$GRAPH_NET_ROOT/graph_net/torch/sample_pass/subgraph_generator.py",
"handler_class_name": "SubgraphGenerator",
"handler_config": {
"resume": ${RESUME},
"model_path_prefix": "$GRAPH_NET_ROOT",
"model_path_prefix": "$DEVICE_REWRITED_OUTPUT_DIR",
"output_dir": "${RANGE_DECOMPOSE_OUTPUT_DIR}",
"subgraph_ranges_json_root": "$SPLIT_POINTS_OUTPUT_DIR",
"subgraph_ranges_json_file_name": "typical_subgraph_ranges.json",
Expand All @@ -139,7 +184,7 @@ EOF
}

function rename_decomposed_subgraph() {
echo ">>> [5] Rename subgraph samples under ${RANGE_DECOMPOSE_OUTPUT_DIR}."
echo ">>> [6] Rename subgraph samples under ${RANGE_DECOMPOSE_OUTPUT_DIR}."
echo ">>>"
python3 -m graph_net.model_path_handler \
--model-path-list ${range_decomposed_subgraph_list} \
Expand All @@ -164,16 +209,15 @@ EOF
}

function remove_duplicate_renamed_graphs() {
echo ">>> [6] Remove duplicated subgraph samples under ${GRAPH_VAR_RENAME_OUTPUT_DIR}."
echo ">>> [7] Remove duplicated subgraph samples under ${GRAPH_VAR_RENAME_OUTPUT_DIR}."
echo ">>>"
python3 -m graph_net.tools.deduplicated \
--samples-dir ${GRAPH_VAR_RENAME_OUTPUT_DIR} \
--target-dir ${DEDUPLICATED_OUTPUT_DIR}
}


function gen_fusible_subgraphs() {
echo ">>> [7] Generate fusible subgraphs for subgraph samples under ${DEDUPLICATED_OUTPUT_DIR}."
echo ">>> [8] Generate fusible subgraphs for subgraph samples under ${DEVICE_REWRITED_OUTPUT_DIR}."
echo ">>>"
python3 -m graph_net.model_path_handler \
--use-subprocess \
Expand Down Expand Up @@ -246,7 +290,7 @@ EOF


function rename_fusible_subgraph() {
echo ">>> [8] Rename subgraph samples under ${FUSIBLE_SUBGRAPH_SAMPLES_DIR}."
echo ">>> [9] Rename subgraph samples under ${FUSIBLE_SUBGRAPH_SAMPLES_DIR}."
echo ">>>"
python3 -m graph_net.model_path_handler \
--model-path-list ${fusible_subgraph_list} \
Expand All @@ -271,59 +315,94 @@ EOF
}

function remove_duplicate_fusible_graphs() {
echo ">>> [9] Remove duplicated subgraph samples under ${RENAMED_FUSIBLE_SUBGRAPH_DIR}."
echo ">>> [10] Remove duplicated subgraph samples under ${RENAMED_FUSIBLE_SUBGRAPH_DIR}."
echo ">>>"
python3 -m graph_net.tools.deduplicated \
--samples-dir ${RENAMED_FUSIBLE_SUBGRAPH_DIR} \
--target-dir ${DEDUPLICATED_FUSIBLE_SUBGRAPH_DIR}
}

function get_dimension_subgraph(){
echo ">>> [11] Generating dimension_subgraph samples under ${DIMENSION_GENERALIZER_OUTPUT_DIR}."
for index in {0..8}; do
echo ">>> Generating dimension_subgraph variant index: ${index}"
python3 -m graph_net.model_path_handler \
--model-path-list "${DIMENSION_GENERALIZER_OUTPUT_DIR}/${index}/${dimension_generalizer_samples_list}" \
--handler-config $(base64 -w 0 <<EOF
{
"handler_path": "$GRAPH_NET_ROOT/graph_net/torch/sample_pass/subgraph_generator.py",
"handler_class_name": "SubgraphGenerator",
"handler_config": {
"model_path_prefix": "${DIMENSION_GENERALIZER_OUTPUT_DIR}/${index}",
"output_dir": "${DIMENSION_SUBGRAPH_SAMPLES_OUTPUT_DIR}/${index}",
"subgraph_ranges_json_root": "$GROUPED_FUSIBLE_SUBGRAPH_RANGES_DIR",
"subgraph_ranges_json_file_name": "grouped_fusible_subgraph_ranges.json",
"device": "cuda",
"resume": ${RESUME}
}
}
EOF
)
done
}

function generate_unittests() {
echo ">>> [10] Generate unittests for subgraph samples under ${DEDUPLICATED_FUSIBLE_SUBGRAPH_DIR}."
echo ">>> [12] Generate unittests for subgraph samples under ${DIMENSION_SUBGRAPH_SAMPLES_OUTPUT_DIR}. "
echo ">>>"
python3 -m graph_net.model_path_handler \
--model-path-list ${deduplicated_fusible_subgraphs_list} \
--handler-config=$(base64 -w 0 <<EOF
for index in {0..8}; do
echo ">>> Generating unittests variant index: ${index}"
python3 -m graph_net.model_path_handler \
--model-path-list "${DIMENSION_SUBGRAPH_SAMPLES_OUTPUT_DIR}/${index}/dimension_subgraph_list.txt" \
--handler-config=$(base64 -w 0 <<EOF
{
"handler_path": "$GRAPH_NET_ROOT/graph_net/sample_pass/agent_unittest_generator.py",
"handler_path": "${GRAPH_NET_ROOT}/graph_net/sample_pass/agent_unittest_generator.py",
"handler_class_name": "AgentUnittestGeneratorPass",
"handler_config": {
"framework": "torch",
"model_path_prefix": "${DEDUPLICATED_FUSIBLE_SUBGRAPH_DIR}",
"output_dir": "$UNITTESTS_OUTPUT_DIR",
"model_path_prefix": "${DIMENSION_SUBGRAPH_SAMPLES_OUTPUT_DIR}/${index}",
"output_dir": "${UNITTESTS_OUTPUT_DIR}/${index}",
"device": "cuda",
"generate_main": false,
"try_run": true,
"resume": ${RESUME},
"data_input_predicator_filepath": "$GRAPH_NET_ROOT/graph_net/torch/constraint_util.py",
"data_input_predicator_filepath": "${GRAPH_NET_ROOT}/graph_net/torch/constraint_util.py",
"data_input_predicator_class_name": "RenamedDataInputPredicator"
}
}
EOF
)
done
}

main() {
timestamp=`date +%Y%m%d_%H%M`
suffix="${OP_RANGE}ops_${timestamp}"

rewrite_device 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_rewrite_device_${suffix}.txt
generate_subgraph_list ${DEVICE_REWRITED_OUTPUT_DIR} ${device_rewrited_subgraph_list}

dimension_generalizer 2>&1 | tee ${DIMENSION_GENERALIZER_OUTPUT_DIR}/log_dimension_generalizer_${suffix}.txt
generate_subgraph_list_by_index ${DIMENSION_GENERALIZER_OUTPUT_DIR} ${dimension_generalizer_samples_list} 8

generate_op_names 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_op_names_${suffix}.txt
generate_split_point 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_split_point_${suffix}.txt
range_decompose 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_range_decompose_${suffix}.txt

@Xreki Xreki Jan 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

前序新增了维度泛化步骤，维度泛化可能会修改model.py的内容，因为有些维度值是使用立即数的方式写死在了model.py里面，那range_decompose是不是应该以维度泛化后的样本作为输入？

@Honglei-Qiu Honglei-Qiu Jan 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

截屏2026-01-23 13 59 01 这就需要对齐一下了，如果“range_decompose以维度泛化后的样本作为输入”，那么就不符合这个流程图了，需要对齐一下

@Xreki Xreki Jan 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我以确认，这样执行理论上没有问题。

@Honglei-Qiu Honglei-Qiu Jan 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到


generate_subgraph_list ${RANGE_DECOMPOSE_OUTPUT_DIR} ${range_decomposed_subgraph_list}

rename_decomposed_subgraph 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_rename_decomposed_subgraph_${suffix}.txt
remove_duplicate_renamed_graphs 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_remove_duplicate_renamed_graphs_${suffix}.txt

generate_subgraph_list ${DEDUPLICATED_OUTPUT_DIR} ${deduplicated_subgraph_list}
gen_fusible_subgraphs 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_fusible_subgraphs_${suffix}.txt

gen_fusible_subgraphs 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_fusible_subgraphs_${suffix}.txt
generate_subgraph_list ${FUSIBLE_SUBGRAPH_SAMPLES_DIR} ${fusible_subgraph_list}

rename_fusible_subgraph 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_rename_fusible_subgraph_${suffix}.txt
remove_duplicate_fusible_graphs 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_remove_duplicate_fusible_graphs_${suffix}.txt

generate_subgraph_list ${DEDUPLICATED_FUSIBLE_SUBGRAPH_DIR} ${deduplicated_fusible_subgraphs_list}

get_dimension_subgraph 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_get_dimension_subgraph_${suffix}.txt
generate_subgraph_list_by_index ${DIMENSION_SUBGRAPH_SAMPLES_OUTPUT_DIR} dimension_subgraph_list.txt 8

generate_unittests 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_unittests_${suffix}.txt
}

Expand Down