Some questions

[abdelkareemkobo]

Thanks for creating and sharing this tool

I am native Arabic and u found this tool has a great potential.
I have quick questions.

1. How can you handle the process of translate multiple samples at the same time knowing that every sample will have different size one maybe 500 tokens and the other is 1000 tokens... For Arabic i didn't find a good models from translation expect the cohere models and i am taking about models in sizes from 4b to 12b that i can try.
   I found theses models are very good even with technical ones but they can't handle multiple sentences and you must play a lot with the generation and specify the maximum number of tokens or it will continue generating.
2. For splitting, did you tried chonkie? The Neural chuker i think will help you a lot.

[strickvl]

Thanks for opening an issue!

1. How can you handle the process of translate multiple samples at the same time knowing that every sample will have different size one maybe 500 tokens and the other is 1000 tokens... For Arabic i didn't find a good models from translation expect the cohere models and i am taking about models in sizes from 4b to 12b that i can try.

At the moment this tool deals with things however most makes sense for you. Usually with the latest VLMs, page by page is enough for good quality. Unfortunately the open-source models are not really strong enough to handle the OCR part and translation in a single shot.

I found theses models are very good even with technical ones but they can't handle multiple sentences and you must play a lot with the generation and specify the maximum number of tokens or it will continue generating.

That just isn't a problem with the gemini series models, for example, or many / most of the really large models.

2.>

2. For splitting, did you tried chonkie? The Neural chuker i think will help you a lot.

Splitting is done fairly simply here, i.e. mostly by page but if you have a long text document you can split by token or character count. The sliding window helps address any issues that come with that.

[abdelkareemkobo]

I am not talking about pdfs here but using It for translating hugging face dataset for the part about how to make handle multiple samples which sometimes can be the same length like pages.