Hi
You can't change the files shells32.dll, but Windows will and can pick any .ico that got needed format needed for it without problem. (16x16, ...) See the format spec more here; https://learn.microsoft.com/en-us/windows/win32/uxguide/vis-icons
Windows will not let you use a .ico that didnt got the right needed size.
The message you got seem to indicate that you tried to use a file that got no icon in the file or an unsupported size.