customized error format will be Json format.
Of course the response is JSON. JSON is standard in Web API these days.
Your original post asked for a "customized error message" but you did not provide an example of the JSON message. Is your expectation that the community will come up with a custom design? Then explain how the design works? That's a large ask for a support forum.
Moreover, REST error responses follow a standard and Web API provides these response out of the box. Why can't you use the responses that come with Web API? I get the feeling that you did not go through the linked tutorial which covers these basics.
At this point, it is not clear exactly what you want.
The following snippet is a ASP.NET Web API bad request response. BadRequest() has three overloads and one of the overloads is a string. You get to add what ever string you like. Is this what you're looking for???
public IHttpActionResult Get()
{
return BadRequest("Houston, we have a problem!");
}