Question about expected runtime of the algorithm

[yannikkellerde]

I looked through your paper and while I did find a lot on sample efficiency and iterations, I did not find anything about the amount of hours you trained your models to get the results reported.

I've been trying to reproduce your results and started a run with your provided command

accelerate launch \ --num_processes 4 \ --num_machines 1\ --machine_rank 0 \ countdown/es_fine-tuning_countdown.py \ --data_sample 200 \ --model_name Qwen/Qwen2.5-1.5B-Instruct \ --gpu_threads 2

on four H200 GPUs with the fairly small 1.5B model. Each iteration seems to take about 10 minutes Iteration 7/1000, Time: 558.99s, Mean: 0.10, Min: 0.08, Max: 0.13, which would mean I would have to train for ~166 hours to reach the 1000 iterations from your paper. Is this expected and also how long it took for you? Or should I check if something is off with my setup?

[VsonicV]

Hi @yannikkellerde , thanks for your feedback. We ran our experiments on several machines with different GPUs. The exact computation time is affected by many factors, e.g., the type of GPUs, the inference engine, the status of the machine (whether there are other works running at the same time). I tested it on one machine using four H100 today, and it took ~420s for each iteration to complete (with --gpu_threads 1 and --gpu_threads 2). We did notice that the iteration time reduced with larger "num_processes" (almost linearly), but not with larger "gpu_threads". We are currently using the raw huggingface inference function, which is indeed slower than other SOTA inference engines, e.g., vLLM or SGLang. We are working on two aspects that can significantly improve the evaluation speed: 1) integrating inference engines, 2) making multi-threading work more efficiently inside each GPU.

BTW, the total number of iterations we used was 500 for the countdown task (as stated in the first paragraph of Section A.1). Would you mind trying one more run using --gpu_threads 1 and let us know the computation time for each iteration. We are curious about the effect of threading inside each GPU. Thank you!

[yannikkellerde]

Hi, thanks for your swift response! Tried with 1 gpu_thread and that indeed does not seem to make a difference as it takes the same amount per iteration as 2 gpu_threads. Also, I can happily report that I indeed do not run out of VRAM even for the 7B model and it takes 935.17s per iteration which is unexpectedly not so much slower than running with the 1.5B model.

Looking forward to any progress you make regarding the inference engine integration.

[ZhangEnmao]

The same as A800，"num_processes" can reduce time linearly, it makes sense; "gpu_threads" is useless

[VsonicV]

@ZhangEnmao Thanks for your update! We created a discussion thread regarding how we can make multi-threading work better inside each GPU here. Feel free to share your thoughts there. It is possible that one inference is already taking all the GPU cores, so multi-threading does not give us any benefits here.

We are working on another version that uses more advanced inference engines to significantly speed up the inference time. It will be released very soon.

[kroggen]

With an RTX 4090 each iteration took ~1 hour (Qwen2.5-3B-Instruct)

Process 0 updating model weights
Iteration 1/1000, Time: 3748.24s, Mean: 0.17, Min: 0.09, Max: 0.24
GPU Memory: 5894.09MB allocated, 16009.40MB peak

If 4 H200 took 10 min, then 1 alone would take 40 min? Not that big of a difference.

It may be cheaper to use many 4090. Or even 3090, AMD, etc. Now that we know that having other instances of the model on the same GPU does not help.

[VsonicV]

@kroggen good point. The number of GPUs indeed matters more. So in your case, if we have 30 4090, then we only need ~2min for one iteration.

[kroggen]

Running on 8 RTX3090 is taking 406 seconds per iteration with Flash Attention 2

Iteration 2/1000, Time: 406.10s, Mean: 0.17, Min: 0.11, Max: 0.24
GPU Memory: 6033.09MB allocated, 14733.53MB peak

A pod like this costs USD 1.38 per hour
While 4 H200 costs USD 9.40 per hour

Remember to update the POPULATION_SIZE to a multiple of GPUs used, to have better utilization (32 on this case is better than 30)

[VsonicV]

@kroggen thanks for sharing this result! If possible, could you please also test the running time without Flash Attention 2? We want to see how big its effect is. We had one version used Flash Attention 2, but we also noticed that the inferences became non-deterministic because of it. To rule out the effect of noisy inferences, we did not include Flash Attention 2 in our main experiments. Would be interested to investigate this tradeoff.

[kroggen]

Sure

Using exactly the same pod (8 RTX3090) it took ~838 seconds per iteration without Flash Attention:

Iteration 1/1000, Time: 836.20s, Mean: 0.17, Min: 0.11, Max: 0.23
GPU Memory: 6033.09MB allocated, 16148.85MB peak
...
Iteration 2/1000, Time: 839.76s, Mean: 0.17, Min: 0.10, Max: 0.24
GPU Memory: 6033.09MB allocated, 16192.56MB peak

More than double the time

[VsonicV]

Thanks for the information! This comparison is helpful! We will need to run some full evolutions to see whether the fine-tuning performance is affected by this non-deterministic inferences introduced by Flash Attention.

[VsonicV]

@yannikkellerde @kroggen We just added an accelerated version with 10X+ speed-up. The running time should be significantly reduced now.