code_dapo agentic task fails

[canghongjian]

I encountered an issue of being too long and uncut while running your examples/qwen3_agentic_gem/gem_math_dapo.yamlconfiguration: index 0 is out of bounds for dimension 1 with size 0:

�[36m(EnvironmentWorker(val_env-0) pid=2125841)�[0m   File "/code/roll/distributed/scheduler/generate_scheduler.py", line 913, in generate_one_request�[32m [repeated 6x across cluster]�[0m

  | 16:52:17.393 | �[36m(EnvironmentWorker(val_env-0) pid=2125841)�[0m     output: DataProto = postprocess_generate(�[32m [repeated 6x across cluster]�[0m
  | 16:52:17.393 | �[36m(EnvironmentWorker(val_env-0) pid=2125841)�[0m   File "/code/roll/utils/functionals.py", line 815, in postprocess_generate�[32m [repeated 6x across cluster]�[0m
  | 16:52:17.393 | �[36m(EnvironmentWorker(val_env-0) pid=2125841)�[0m     response_mask = get_pad_mask(response_id=response, pad_token=pad_token_id, dtype=attention_mask.dtype)�[32m [repeated 6x across cluster]�[0m
  | 16:52:17.393 | �[36m(EnvironmentWorker(val_env-0) pid=2125841)�[0m   File "/code/roll/utils/functionals.py", line 318, in get_pad_mask�[32m [repeated 6x across cluster]�[0m
  | 16:52:17.393 | �[36m(EnvironmentWorker(val_env-0) pid=2125841)�[0m     not (pad_mask[:, 0] == 0).logical_and(pad_mask.sum(-1) != 0).any()�[32m [repeated 6x across cluster]�[0m
  | 16:52:17.393 | �[36m(EnvironmentWorker(val_env-0) pid=2125841)�[0m **IndexError: index 0 is out of bounds for dimension 1 with size 0**�[32m [repeated 6x across cluster]�[0m

[canghongjian]

Full configuration:

defaults:
  - ../config/traj_envs_gem_math@_here_

hydra:
  run:
    dir: .
  output_subdir: null

pretrain: /publicdata/huggingface.co/Qwen/Qwen3-8B
reward_pretrain: /publicdata/huggingface.co/Qwen/Qwen3-8B

exp_name: roll_${oc.env:BIFROST_JOB_NAME}_qwen3-8B-agentic-code-dapo
seed: 42
local_save_path: /log/roll_output
logging_dir: ${local_save_path}/logs
output_dir: ${oc.env:CPFS_DIR}/roll_output/${exp_name}
system_envs:
  USE_MODELSCOPE: '1'

checkpoint_config:
  type: file_system
  output_dir: ${output_dir}/roll_ckpt

track_with: swanlab  
tracker_kwargs:  
  login_kwargs:  
    api_key: ${oc.env:SWANLAB_API_KEY}  # 您的SwanLab API密钥  
  project: roll-experiments  # 项目名称  
  logdir: ${output_dir}/swanlog  # 日志存储目录  
  experiment_name: ${exp_name}  # 实验名称  
  tags:  # 实验标签  
    - qwen3_8b  
    - code_dapo  
    - reinforce

num_gpus_per_node: 8

max_steps: 1024
save_steps: 10000
logging_steps: 1
eval_steps: 10
resume_from_checkpoint: false

rollout_batch_size: 128
val_batch_size: 128
sequence_length: 16384

advantage_clip: 20
ppo_epochs: 1
adv_estimator: "reinforce"

#pg_clip: 0.1
#dual_clip_loss: True
init_kl_coef: 0.0
whiten_advantages: true
entropy_loss_coef: 0
max_grad_norm: 1.0

actor_train:
  model_args:
    attn_implementation: fa2
    disable_gradient_checkpointing: false
    dtype: bf16
    model_type: ~
  training_args:
    learning_rate: 1.0e-6
    weight_decay: 0
    per_device_train_batch_size: 2
    gradient_accumulation_steps: 8
    warmup_steps: 0
    warmup_ratio: 0
    lr_scheduler_type: constant
  strategy_args:
#    strategy_name: deepspeed_train
#    strategy_config: ${deepspeed_zero3}
    strategy_name: megatron_train
    strategy_config:
      tensor_model_parallel_size: 1
      pipeline_model_parallel_size: 1
      expert_model_parallel_size: 1
      use_distributed_optimizer: true
      recompute_granularity: full
  device_mapping: list(range(0,8))
  infer_batch_size: 2

actor_infer:
  model_args:
    disable_gradient_checkpointing: true
    dtype: bf16
  generating_args:
    max_new_tokens: ${max_tokens_per_step} # single-turn response length
    top_p: 0.99
    top_k: 100
    num_beams: 1
    temperature: 0.99
    num_return_sequences: 1
    stop_strings: ["</code>"]
  strategy_args:
    strategy_name: sglang
    strategy_config:
      skip_tokenizer_init: False
      mem_fraction_static: 0.7
      load_format: dummy
  num_gpus_per_worker: 2
  device_mapping: list(range(0,8))

reference:
  model_args:
    attn_implementation: fa2
    disable_gradient_checkpointing: true
    dtype: bf16
    model_type: ~
  strategy_args:
    strategy_name: hf_infer
    strategy_config: ~
  device_mapping: list(range(0,8))
  infer_batch_size: 2

reward_normalization:
  grouping: batch # 可以tags(env_type)/traj_group_id(group)/batch(rollout_batch)... group_by计算reward/adv
  method: identity # asym_clip / identity / mean_std

train_env_manager:
  max_env_num_per_worker: 16
  num_env_groups: 128
  # under the same group, the env config and env seed are ensured to be equal
  group_size: 1
  tags: [dapo_17k_with_python_code]
  num_groups_partition: [128] # If not set, all env names divide nums equally. Under the same group, the env config and env seed (prompt) are equal in each generation

val_env_manager:
  max_env_num_per_worker: 32
  num_env_groups: 128
  group_size: 1 # should be set to 1 because val temperature is set to 0 and same prompt leads to same output
  tags: [dapo_17k_with_python_code]
  num_groups_partition: [128] # TODO: If not set, all env names divide nums equally. Under the same group, the env config and env seed (prompt) are equal in each generation


# Here, you can override variables defined in the imported envs. max_tokens_per_step: 128 in custom_env.SimpleSokoban, here replaced by 64
max_tokens_per_step: 4096

math_agent_system_template: Please reason step by step, and put your final answer within '\\boxed{}', e.g. \\boxed{{A}}..
math_agent_template: "{observation}\nEnsure that your response includes the format of '\\boxed{{answer}}', e.g. \\boxed{{A}}."
custom_envs:
  dapo_17k:
    ${gem_math.dapo_17k}
  dapo_17k_with_python_code:
    ${gem_math.dapo_17k_with_python_code}

[canghongjian]

Seems to be an issue with the inference engine. Specifically, in agentic training (at least in qa_search and code_dapo), only vllm can be used instead of sglang, otherwise it will report the above boundary error, i.e. the inference engine will do an empty rollout, which is mainly caused by the stop words setting.
Needs a closer look.