Regarding llama-bench and llama-parallel commands

[vineelabhinav]

Hello @ggerganov @ngxson ,
I have queries regarding the two commands:

1. llama-bench : How to do test which involves prompt processing(pp) + text generation at same time(tg) ? As of now this command supports only -p option and -n option which gives separate evaluation values for pp and tg but not combined. I see there is -pg option but its not working (says option not found, I dint understand correct format to give for it)
2. llama-parallel: How parallelism is done on the batch dimension ? Assume I have input of shape [batch_size, M, N] and also there is for loop running over each dimension. Does lllama-parallel parallelizes the batch_size dimension's for loop using openmp parallel for pragma ? If its not case how it does parallelism? Can you mention file where this parallelism code is written?

[vineelabhinav]

@ggerganov @ngxson any information regarding this?

[ggerganov]

You can use:

llama-bench -m model.gguf -pg 512,32

You can also use llama-batched-bench

llama-batched-bench --help

[fairydreaming]

@vineelabhinav Check out this discussion regarding the batching, it should shed some light on the topic: #4130

[vineelabhinav]

@ggerganov Thanks for the reply.
I have doubt on performance improvement of for loop in llama.cpp. We have a for loop inside which it does some elementwise multiplication in llama.cpp . There is no Neon/SVE intrinsics used for this code. I have written intrinsics code for the same for loop but there is no performance improvement (expected atleast some performance gain as SVE has SIMD where I parallelize operations for 8 elements). What could be the reason for this? Do llama.cpp does any further optimizations which overlooks the speedup from SVE?
How exactly does llama.cpp optimizes the operations inside it and how SVE intrinsics code has effect on it?

[ggerganov]

Does your code execute? If it does and you don't see an improvement then likely it is not faster than what the original version does.

[vineelabhinav]

@ggerganov
Yes, the code executes. I expected atleast a little speedup over the normal implementation since with SVE, all operations are done with data kept inside registers(reduce memory latency) with SIMD parallelism over 8 elements at a time where the original code does not do. Anything that I'm missing out or llama.cpp optimizations are ahead of what SVE provides? How exactly does llama.cpp does optimizations?

[ggerganov]

It's hard to answer without any specific information about your changes. Which loop did you implement?

[vineelabhinav]

@ggerganov
I implemented this loop : https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cpu/ggml-cpu.c#L12722 . This loop iterates for 16 times (nc=16). I have removed this loop and used SVE to load chunk of 8 elements at a time into registers which does multiplication of 8 elements at a time in parallel (load two chunks of 8 elements into separate registers, which is total 16 elements) without the use of for loop. Even expf() function is implemented in SVE.

[vineel96]

@ggerganov
Have you had a chance to look into this. Let me know your thoughts when you get time to see?

[ggerganov]

Not sure, maybe the loop is not a bottleneck or the compiler has done a good job to optimize it. You can try to remove the loop all together to see what would be theoretical speed-up if you had an infinitely fast implementation - probably this will give you some insights.

[vineelabhinav]

Hello @ggerganov ,
When I use below command, I get below output. How to interpret the output? What's each test for?
Command: ./build/bin/llama-bench -m ../../gguf_files/falcon-mamba-7B-F32.gguf -b 512 -t 64 -pg 1,100 .
Output:

model	size	params	backend	threads	n_batch	test	t/s
mamba ?B all F32	27.09 GiB	7.27 B	CPU	64	512	pp512	16.73 ± 0.00
mamba ?B all F32	27.09 GiB	7.27 B	CPU	64	512	tg128	3.46 ± 0.00
mamba ?B all F32	27.09 GiB	7.27 B	CPU	64	512	pp1+tg100	3.46 ± 0.00

1. ) In Above output table, when test is pp512 what is value of tg for this test? and when test is tg128, what is the value of pp for this test? and why last two rows have same t/s values?
2. ) If --batch_size option is used, llama-bench uses llama-parallel or llama-cli ?

[vineelabhinav]

@ggerganov , Can you please provide clarification on this ?

[ggerganov]

• pp4096
  Process 4096 tokens in batces of n_batch tokens (i.e. typical prompt processing task). No generation involved
• tg128
  Generate 128 tokens starting with an empty context. 1 token per batch
• pp4096+tg128
  Generate 128 tokens after first prompt processing 4096 tokens. The result is (4096 + 128)/total_time

[vineelabhinav]

@ggerganov ,
If --batch_size option is used, llama-bench internally uses llama-parallel or llama-cli ? How batches are processed?

I want to reproduce the below command with llama-bench command:
command: ./build/bin/llama-parallel -m ../../gguf_files/falcon-mamba-7B-F32.gguf --ctx-size 2048 --ignore-eos -n 100 -np 4 -ns 4 -f input.txt
task: generate 100 tokens with batch_size=4 and input prompt length=1
input.txt: input file with 4 lines where each line contains 1 length token prompt (4 lines as we use batch_size=4)

[vineelabhinav]

@ggerganov, Can you provide insights for above question?