Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepare_inputs_for_generation and position_embeddings #683

Open
ambroser53 opened this issue Dec 17, 2024 · 0 comments
Open

prepare_inputs_for_generation and position_embeddings #683

ambroser53 opened this issue Dec 17, 2024 · 0 comments

Comments

@ambroser53
Copy link

Not really an issue with AutoAWQ as it is with transformers but the prepare_inputs_for_generation functions are not being updated to include position_embeddings which models are needing for decoder layer inference (i.e. to be put in module_kwargs). The main reason this hasn't been flagged yet is that transformers keeps pushing it being mandatory back (modeling_llama.py in tranformers version v4.47.1 -> position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None, # will become mandatory in v4.45) but eventually it'll become an issue and it does break things when using some specific models that were brought in more recently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant