soundwire: fix programing unattached BUSCLOCK_SCALE issue #8185

Job	Run time
GCC build X86_64 with W=1 allmodconfig	1m 53s
Sparse check	3m 16s
GCC build arm64	20m 51s
GCC static -fanalyzer	4m 27s
GCC build i386	17m 2s
Clang build x86_64	4m 41s
GCC build X86_64	20m 8s
GCC build X86_64 with W=1	3m 13s
GCC build arm64 allmodconfig	1m 46s
	1h 17m 17s

Provide feedback