[tests] fix torchao tests - #14258

[tests] fix torchao tests#14258

sayakpaul wants to merge 2 commits into

mainhuggingface/diffusers:mainfrom

torchao-fixeshuggingface/diffusers:torchao-fixesCopy head branch name to clipboard