btw %problem% is decimal number
Then just use it as is. If it's a hex value, then it has to start with "0x".
reg add hkcu\software /v TestValue /d 0xffff /t REG_DWORD /f
reg query hkcu\software /v TestValue
reg add hkcu\software /v TestValue /d 16 /t REG_DWORD /f
reg query hkcu\software /v TestValue
set problem=ffff
reg add hkcu\software /v TestValue /d 0x%problem% /t REG_DWORD /f
reg query hkcu\software /v TestValue
set problem=16
reg add hkcu\software /v TestValue /d %problem% /t REG_DWORD /f
reg query hkcu\software /v TestValue