If the function succeeds, the return value is a handle to the specified server.
If the function fails, it returns a handle that is not valid. You can test the validity of the handle by using it in another function call.
Note that unlike some other Windows API functions where a failure will return a NULL handle or a handle containing INVALID_HANDLE_VALUE the documentation for WTSOpenServerA does not give a value to identify an invalid handle if the function fails. So the only way to know if you have received a valid handle is to use it in another function call and be sure to test whether that call succeeds or fails.
You should treat the server handle as an opaque value. The numeric representation of a server handle will appear differently between a 32-bit process and a 64-bit process. Multiple executions of a process that calls WTSOpenServerA successfully will not always obtain the same server handle.