Skip to content

训推不一致 #17979

Open
Open
训推不一致#17979
Assignees
cuicheng01
@xuexigpt

Description

@xuexigpt
xuexigpt
opened on Apr 27, 2026

🔎 Search before asking

  • I have searched the PaddleOCR Docs and found no similar bug report.
  • I have searched the PaddleOCR Issues and found no similar bug report.
  • I have searched the PaddleOCR Discussions and found no similar bug report.

🐛 Bug (问题描述)

你好 我用预训练模型训练过程中acc能到97%，然后验证集的最好效果在第134个epoch的时候，验证集准确率高达83%，但是我用官方导出脚本把模型导出后测试原训练集和验证集，但是准确率只有63%和57%，请老师们可以帮忙看看
训练脚本如下
python
-m paddle.distributed.launch
--gpus '0,1,2,3'
tools/train.py
-c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml
-o Global.use_gpu=true
-o Global.pretrained_model=model/PP-OCRv5_server_rec_pretrained.pdparams
-o Global.batch_size_per_card=8
-o Optimizer.lr.learning_rate=0.001
-o Global.save_model_dir={output_dir}
-o Train.dataset.data_dir={dataset_dir}
-o Train.dataset.label_file_list={train_labels}
-o Eval.dataset.data_dir={dataset_dir}
-o Eval.dataset.label_file_list={val_labels}
-o Global.character_dict_path=ppocr/utils/dict/vin_dict.txt
-o Global.distributed=true
模型评估如下
python3 tools/eval.py -c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml -o Global.pretrained_model=output/PP-OCRv5_server_rec_pretrain/iter_epoch_134.pdparams
模型导出如下
python3 tools/export_model.py -c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml -o Global.pretrained_model=output/PP-OCRv5_server_rec_pretrain/iter_epoch
_134.pdparams Global.save_inference_dir="./pretrain_134/"

模型训练日志
[2026/04/25 19:30:15] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/iter_epoch_1499
[2026/04/25 19:31:09] ppocr INFO: epoch: [1500/1500], global_step: 9000, lr: 0.000000, acc: 0.972610, norm_edit_dis: 0.998454, CTCLoss: 0.102627, NRTRL
oss: 0.692408, loss: 0.794872, avg_reader_cost: 5.74675 s, avg_batch_cost: 5.88237 s, avg_samples: 55.4, ips: 9.41797 samples/s, eta: 0:00:00, max_mem_
reserved: 12121 MB, max_mem_allocated: 10669 MB

eval model:: 0%| | 0/5 [00:00<?, ?it/s]
LAUNCH INFO 2026-04-25 19:31:17,345 Pod completed
LAUNCH INFO 2026-04-25 19:31:17,345 Exit code 0
eval model:: 20%|██ | 1/5 [00:01<00:05, 1.44s/it]
eval model:: 60%|██████ | 3/5 [00:01<00:00, 2.33it/s]
eval model:: 100%|██████████| 5/5 [00:01<00:00, 4.12it/s]
eval model:: 100%|██████████| 5/5 [00:03<00:00, 1.64it/s]
[2026/04/25 19:31:12] ppocr INFO: cur metric, acc: 0.7942238123786316, norm_edit_dis: 0.9817974542521836, fps: 1713.9939189637748
[2026/04/25 19:31:12] ppocr INFO: best metric, acc: 0.8321299488785208, is_float16: False, norm_edit_dis: 0.9803093024916347, fps: 1774.7213329860742,
best_epoch: 134
[2026/04/25 19:31:14] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/latest
[2026/04/25 19:31:14] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/iter_epoch_1500
[2026/04/25 19:31:14] ppocr INFO: best metric, acc: 0.8321299488785208, is_float16: False, norm_edit_dis: 0.9803093024916347, fps: 1774.7213329860742,
best_epoch: 134
I0425 19:31:15.709209 372790 process_group_nccl.cc:163] ProcessGroupNCCL destruct
模型评估日志
[2026/04/27 15:11:58] ppocr INFO: train with paddle 3.3.1 and device Place(gpu:0)
[2026/04/27 15:11:58] ppocr INFO: Initialize indexes of datasets:['/data01/temp/PaddleOCR/vin_dataset/val_labels.txt']
W0427 15:11:58.953311 1224943 gpu_resources.cc:116] Please NOTE: device: 0, GPU Compute Capability: 9.0, Driver API Version: 12.9, Runtime API Version:
12.9
[2026/04/27 15:11:59] ppocr INFO: load pretrain successful from output/PP-OCRv5_server_rec_pretrain/iter_epoch_134
[2026/04/27 15:11:59] ppocr INFO: metric in ckpt ***************
[2026/04/27 15:11:59] ppocr INFO: is_float16:False
eval model:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:02<00:00, 2.03it/s]
[2026/04/27 15:12:01] ppocr INFO: metric eval ***************
[2026/04/27 15:12:01] ppocr INFO: acc:0.8176895159261821
[2026/04/27 15:12:01] ppocr INFO: norm_edit_dis:0.9789127606205862
[2026/04/27 15:12:01] ppocr INFO: fps:832.6830134980511
我自己的测试脚本
from paddleocr import TextRecognition
ocr = TextRecognition(
model_dir=r'/data01/temp/PaddleOCR/pretrain_134',
)

filepath_list = ["/data01/temp/PaddleOCR/vin_dataset/train","/data01/temp/PaddleOCR/vin_dataset/val"]
for filepath in filepath_list:
result = []
dataset = filepath.split("/")[-1]
for file in os.listdir(filepath):
file_abs_path = os.path.join(filepath,file)
result = ocr.predict(file_abs_path)
pre_result = result[0].get("rec_text")
print(f"dataset:{dataset},filename:{file},pre_result:{pre_result}")

最后实测效果和训练过程中和评估脚本中的结果相去胜远，不知道什么原因

🏃‍♂️ Environment (运行环境)

你好 我用预训练模型训练过程中acc能到97%，然后验证集的最好效果在第134个epoch的时候，验证集准确率高达83%，但是我用官方导出脚本把模型导出后测试原训练集和验证集，但是准确率只有63%和57%，请老师们可以帮忙看看
训练脚本如下
python
-m paddle.distributed.launch
--gpus '0,1,2,3'
tools/train.py
-c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml
-o Global.use_gpu=true
-o Global.pretrained_model=model/PP-OCRv5_server_rec_pretrained.pdparams
-o Global.batch_size_per_card=8
-o Optimizer.lr.learning_rate=0.001
-o Global.save_model_dir={output_dir}
-o Train.dataset.data_dir={dataset_dir}
-o Train.dataset.label_file_list={train_labels}
-o Eval.dataset.data_dir={dataset_dir}
-o Eval.dataset.label_file_list={val_labels}
-o Global.character_dict_path=ppocr/utils/dict/vin_dict.txt
-o Global.distributed=true
模型评估如下
python3 tools/eval.py -c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml -o Global.pretrained_model=output/PP-OCRv5_server_rec_pretrain/iter_epoch_134.pdparams
模型导出如下
python3 tools/export_model.py -c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml -o Global.pretrained_model=output/PP-OCRv5_server_rec_pretrain/iter_epoch
_134.pdparams Global.save_inference_dir="./pretrain_134/"

模型训练日志
[2026/04/25 19:30:15] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/iter_epoch_1499
[2026/04/25 19:31:09] ppocr INFO: epoch: [1500/1500], global_step: 9000, lr: 0.000000, acc: 0.972610, norm_edit_dis: 0.998454, CTCLoss: 0.102627, NRTRL
oss: 0.692408, loss: 0.794872, avg_reader_cost: 5.74675 s, avg_batch_cost: 5.88237 s, avg_samples: 55.4, ips: 9.41797 samples/s, eta: 0:00:00, max_mem_
reserved: 12121 MB, max_mem_allocated: 10669 MB

eval model:: 0%| | 0/5 [00:00<?, ?it/s]
LAUNCH INFO 2026-04-25 19:31:17,345 Pod completed
LAUNCH INFO 2026-04-25 19:31:17,345 Exit code 0
eval model:: 20%|██ | 1/5 [00:01<00:05, 1.44s/it]
eval model:: 60%|██████ | 3/5 [00:01<00:00, 2.33it/s]
eval model:: 100%|██████████| 5/5 [00:01<00:00, 4.12it/s]
eval model:: 100%|██████████| 5/5 [00:03<00:00, 1.64it/s]
[2026/04/25 19:31:12] ppocr INFO: cur metric, acc: 0.7942238123786316, norm_edit_dis: 0.9817974542521836, fps: 1713.9939189637748
[2026/04/25 19:31:12] ppocr INFO: best metric, acc: 0.8321299488785208, is_float16: False, norm_edit_dis: 0.9803093024916347, fps: 1774.7213329860742,
best_epoch: 134
[2026/04/25 19:31:14] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/latest
[2026/04/25 19:31:14] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/iter_epoch_1500
[2026/04/25 19:31:14] ppocr INFO: best metric, acc: 0.8321299488785208, is_float16: False, norm_edit_dis: 0.9803093024916347, fps: 1774.7213329860742,
best_epoch: 134
I0425 19:31:15.709209 372790 process_group_nccl.cc:163] ProcessGroupNCCL destruct
模型评估日志
[2026/04/27 15:11:58] ppocr INFO: train with paddle 3.3.1 and device Place(gpu:0)
[2026/04/27 15:11:58] ppocr INFO: Initialize indexes of datasets:['/data01/temp/PaddleOCR/vin_dataset/val_labels.txt']
W0427 15:11:58.953311 1224943 gpu_resources.cc:116] Please NOTE: device: 0, GPU Compute Capability: 9.0, Driver API Version: 12.9, Runtime API Version:
12.9
[2026/04/27 15:11:59] ppocr INFO: load pretrain successful from output/PP-OCRv5_server_rec_pretrain/iter_epoch_134
[2026/04/27 15:11:59] ppocr INFO: metric in ckpt ***************
[2026/04/27 15:11:59] ppocr INFO: is_float16:False
eval model:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:02<00:00, 2.03it/s]
[2026/04/27 15:12:01] ppocr INFO: metric eval ***************
[2026/04/27 15:12:01] ppocr INFO: acc:0.8176895159261821
[2026/04/27 15:12:01] ppocr INFO: norm_edit_dis:0.9789127606205862
[2026/04/27 15:12:01] ppocr INFO: fps:832.6830134980511
我自己的测试脚本
from paddleocr import TextRecognition
ocr = TextRecognition(
model_dir=r'/data01/temp/PaddleOCR/pretrain_134',
)

filepath_list = ["/data01/temp/PaddleOCR/vin_dataset/train","/data01/temp/PaddleOCR/vin_dataset/val"]
for filepath in filepath_list:
result = []
dataset = filepath.split("/")[-1]
for file in os.listdir(filepath):
file_abs_path = os.path.join(filepath,file)
result = ocr.predict(file_abs_path)
pre_result = result[0].get("rec_text")
print(f"dataset:{dataset},filename:{file},pre_result:{pre_result}")

最后实测效果和训练过程中和评估脚本中的结果相去胜远，不知道什么原因

🌰 Minimal Reproducible Example (最小可复现问题的Demo)

你好 我用预训练模型训练过程中acc能到97%，然后验证集的最好效果在第134个epoch的时候，验证集准确率高达83%，但是我用官方导出脚本把模型导出后测试原训练集和验证集，但是准确率只有63%和57%，请老师们可以帮忙看看
训练脚本如下
python
-m paddle.distributed.launch
--gpus '0,1,2,3'
tools/train.py
-c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml
-o Global.use_gpu=true
-o Global.pretrained_model=model/PP-OCRv5_server_rec_pretrained.pdparams
-o Global.batch_size_per_card=8
-o Optimizer.lr.learning_rate=0.001
-o Global.save_model_dir={output_dir}
-o Train.dataset.data_dir={dataset_dir}
-o Train.dataset.label_file_list={train_labels}
-o Eval.dataset.data_dir={dataset_dir}
-o Eval.dataset.label_file_list={val_labels}
-o Global.character_dict_path=ppocr/utils/dict/vin_dict.txt
-o Global.distributed=true
模型评估如下
python3 tools/eval.py -c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml -o Global.pretrained_model=output/PP-OCRv5_server_rec_pretrain/iter_epoch_134.pdparams
模型导出如下
python3 tools/export_model.py -c configs/rec/PP-OCRv5/PP-OCRv5_server_rec.yml -o Global.pretrained_model=output/PP-OCRv5_server_rec_pretrain/iter_epoch
_134.pdparams Global.save_inference_dir="./pretrain_134/"

模型训练日志
[2026/04/25 19:30:15] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/iter_epoch_1499
[2026/04/25 19:31:09] ppocr INFO: epoch: [1500/1500], global_step: 9000, lr: 0.000000, acc: 0.972610, norm_edit_dis: 0.998454, CTCLoss: 0.102627, NRTRL
oss: 0.692408, loss: 0.794872, avg_reader_cost: 5.74675 s, avg_batch_cost: 5.88237 s, avg_samples: 55.4, ips: 9.41797 samples/s, eta: 0:00:00, max_mem_
reserved: 12121 MB, max_mem_allocated: 10669 MB

eval model:: 0%| | 0/5 [00:00<?, ?it/s]
LAUNCH INFO 2026-04-25 19:31:17,345 Pod completed
LAUNCH INFO 2026-04-25 19:31:17,345 Exit code 0
eval model:: 20%|██ | 1/5 [00:01<00:05, 1.44s/it]
eval model:: 60%|██████ | 3/5 [00:01<00:00, 2.33it/s]
eval model:: 100%|██████████| 5/5 [00:01<00:00, 4.12it/s]
eval model:: 100%|██████████| 5/5 [00:03<00:00, 1.64it/s]
[2026/04/25 19:31:12] ppocr INFO: cur metric, acc: 0.7942238123786316, norm_edit_dis: 0.9817974542521836, fps: 1713.9939189637748
[2026/04/25 19:31:12] ppocr INFO: best metric, acc: 0.8321299488785208, is_float16: False, norm_edit_dis: 0.9803093024916347, fps: 1774.7213329860742,
best_epoch: 134
[2026/04/25 19:31:14] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/latest
[2026/04/25 19:31:14] ppocr INFO: save model in ./output/PP-OCRv5_server_rec_pretrain/iter_epoch_1500
[2026/04/25 19:31:14] ppocr INFO: best metric, acc: 0.8321299488785208, is_float16: False, norm_edit_dis: 0.9803093024916347, fps: 1774.7213329860742,
best_epoch: 134
I0425 19:31:15.709209 372790 process_group_nccl.cc:163] ProcessGroupNCCL destruct
模型评估日志
[2026/04/27 15:11:58] ppocr INFO: train with paddle 3.3.1 and device Place(gpu:0)
[2026/04/27 15:11:58] ppocr INFO: Initialize indexes of datasets:['/data01/temp/PaddleOCR/vin_dataset/val_labels.txt']
W0427 15:11:58.953311 1224943 gpu_resources.cc:116] Please NOTE: device: 0, GPU Compute Capability: 9.0, Driver API Version: 12.9, Runtime API Version:
12.9
[2026/04/27 15:11:59] ppocr INFO: load pretrain successful from output/PP-OCRv5_server_rec_pretrain/iter_epoch_134
[2026/04/27 15:11:59] ppocr INFO: metric in ckpt ***************
[2026/04/27 15:11:59] ppocr INFO: is_float16:False
eval model:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:02<00:00, 2.03it/s]
[2026/04/27 15:12:01] ppocr INFO: metric eval ***************
[2026/04/27 15:12:01] ppocr INFO: acc:0.8176895159261821
[2026/04/27 15:12:01] ppocr INFO: norm_edit_dis:0.9789127606205862
[2026/04/27 15:12:01] ppocr INFO: fps:832.6830134980511
我自己的测试脚本
from paddleocr import TextRecognition
ocr = TextRecognition(
model_dir=r'/data01/temp/PaddleOCR/pretrain_134',
)

filepath_list = ["/data01/temp/PaddleOCR/vin_dataset/train","/data01/temp/PaddleOCR/vin_dataset/val"]
for filepath in filepath_list:
result = []
dataset = filepath.split("/")[-1]
for file in os.listdir(filepath):
file_abs_path = os.path.join(filepath,file)
result = ocr.predict(file_abs_path)
pre_result = result[0].get("rec_text")
print(f"dataset:{dataset},filename:{file},pre_result:{pre_result}")

最后实测效果和训练过程中和评估脚本中的结果相去胜远，不知道什么原因

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions