In the Replace dialog, did you click the More button and check the box for "Use wildcards"? If you want to use a search expression like [R^#^#] then you should NOT check that box. Note that with non-wildcard expressions like this, you'll need to do three separate searches for references with one-, two-, or three-digit numbers.
The Replace With expression should be ^& and then click the Format button, choose Font in the menu, and select Italic and the red color.
What Word refers to as wildcards (Microsoft's modified version of regex search) does not recognize the caret codes such as ^#. The caret codes are only for non-wildcard searches.
If you want to use wildcards, the search expression should be
(\[R[0-9]{1,3}\])
which will match the letter R followed by any one to three digits. Be careful to use the square brackets, curly brackets, parentheses, and backslashes in the proper places. For explanations, see https://www.gmayor.com/replace_using_wildcards.htm
The parentheses around the expression allow you to use \1 as the "found text" in the Replace With box instead of ^&. The setting of font formatting is the same as above.