The output of Qwen3 0.6B model is not correct

[diuzi]

System Info

A100

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

For the Qwen3 0.6B model, the vectors obtained using TEI and other frameworks are inconsistent, and the similarity scores from TEI are clearly unreasonable.

model=Qwen/Qwen3-Embedding-0.6B
volume=~/model
name=qwen3_emb

docker run \
  -d \
  --gpus all \
  -p 8080:80 \
  -v $volume:/data \
  --name $name \
  --pull always ghcr.io/huggingface/text-embeddings-inference:1.7.2 \
  --model-id $model \
  --auto-truncate \
  --max-batch-tokens 25600 \
  --max-client-batch-size 512

import numpy as np
from tqdm import tqdm
from langchain_community.embeddings import HuggingFaceHubEmbeddings
from sentence_transformers import SentenceTransformer


def st(batch_size: int = 48):
    print('======================================= ST =======================================')
    model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
    print(model.encode([test])[0, :10])

    emb_lst = []
    texts = queries + documents
    for i in tqdm(range(0, len(texts), batch_size), total=len(texts) // batch_size):
        emb = model.encode(texts[i:i + batch_size])
        emb_lst.append(emb)
    emb = np.concatenate(emb_lst, axis=0)
    emb = emb / np.linalg.norm(emb, axis=1, keepdims=True)
    sim = emb[:2] @ emb[2:].T
    print(sim)


def tei():
    print('======================================= TEI =======================================')
    qwen3_model = HuggingFaceHubEmbeddings(
        model='http://localhost:8080/',
    )
    print(qwen3_model.embed_query(test)[:10])

    emb = qwen3_model.embed_documents(
        queries + documents
    )
    emb = np.array(emb)
    emb = emb / np.linalg.norm(emb, axis=1, keepdims=True)
    sim = emb[:2] @ emb[2:].T
    print(sim)


if __name__ == '__main__':
    test = 'hello'
    queries = [
        "What is the capital of China?",
        "Explain gravity",
    ]
    documents = [
        "The capital of China is Beijing.",
        "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
    ]

    st()
    tei()

• output

  ======================================= ST =======================================
  [-0.01811265 -0.02243944 -0.0128016  -0.04207409  0.00253615 -0.05358043
   -0.0454641   0.04452865 -0.06027853  0.00281567]
   
   [[0.8015334  0.3347563 ]
   [0.17402115 0.7029334 ]]
   
   ======================================= TEI =======================================
   [-0.02259685, -0.3247632, -0.0070690215, 0.07453357, 0.03596279, -0.11605549, 0.0022583979, 0.1535968, -0.0062840516, -0.076112084]
  [[0.37315232 0.13970019]
   [0.26710618 0.23133032]]
  

Expected behavior

.

[MohannadEhabBarakat]

Do you get the correct output shapes? I get wrong shapes for TEI.

[diuzi]

Do you get the correct output shapes? I get wrong shapes for TEI.

Correct, default 1024.

[pocman]

Yes, this is a known issue and it was fixed by #648
1.7.3 should behave as expected.