CoT skipping fails with models like Qwen3 0.6B

[p-e-w]

Even if the response is forced to start with a closed CoT block, the model performs CoT after it.

[Vinay-Umrethe]

Even if the response is forced to start with a closed CoT block, the model performs CoT after it.

How are you "forcing" the model to do reasoning <think>...</think> I think it's just a parameter toggle transformers do, maybe by switching internal system prompt / chat-template.

Current heretic marks Checking for common response prefix... Found OR Not only if all responses contain them, which some models might have trouble producing. What if, even if found in few responses the <think> tags; we'll mark them as found?

Some models have different variants like Qwen3-VL-2B-Instruct & Qwen3-VL-2B-Thinking while some models have both interchangeable with config like: Qwen3.5-27B

chat_response = client.chat.completions.create(
    model="Qwen/Qwen3.5-27B",
    messages=messages,
    max_tokens=32768,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},

[Vinay-Umrethe]

Checking for common response prefix...
* Prefix found: '<think>\nOkay,'
* Closed Chain-of-Thought block: '<think></think>'
* Rechecking with prefix...
* Extended prefix found: '<think></think>\nOkay,'

Is this kind of detection fine for optuna quality? any bad effects on optimizer?

Model used: Qwen/Qwen-3-0.6B

[p-e-w]

This isn't about the optimizer, it's about KLD computation. Without skipping the CoT, the probability distribution will be dominated by the CoT initializer, making comparisons meaningless.

The recognition works fine. The problem is that some models are so strongly trained to think that if you give them an empty CoT block they will do the thinking in the regular response portion.