Photo Video camera in Unity
Enabling the capability for camera access
The "WebCam" capability must be declared for an app to use the camera.
- In the Unity Editor, go to the player settings by navigating to the "Edit > Project Settings > Player" page
- Select the "Windows Store" tab
- In the "Publishing Settings > Capabilities" section, check the WebCam and Microphone capabilities
Only a single operation can occur with the camera at a time. You can check which mode the camera is currently in with UnityEngine.XR.WSA.WebCam.Mode in Unity 2018 and earlier or UnityEngine.Windows.WebCam.Mode in Unity 2019 and later. Available modes are photo, video, or none.
Photo capture
Namespace (before Unity 2019): UnityEngine.XR.WSA.WebCam
Namespace (Unity 2019 and later): UnityEngine.Windows.WebCam
Type: PhotoCapture
The PhotoCapture type allows you to take still photographs with the Photo Video Camera. The general pattern for using PhotoCapture to take a photo is as follows:
- Create a PhotoCapture object
- Create a CameraParameters object with the settings you want
- Start Photo Mode via StartPhotoModeAsync
- Take the photo you want
- (optional) Interact with that picture
- Stop Photo Mode and clean up resources
Common set-up for PhotoCapture
For all three uses, start with the same first three steps above
Start by creating a PhotoCapture object
private void Start()
{
PhotoCapture.CreateAsync(false, OnPhotoCaptureCreated);
}
Next, store your object, set your parameters, and start Photo Mode
private PhotoCapture photoCaptureObject = null;
void OnPhotoCaptureCreated(PhotoCapture captureObject)
{
photoCaptureObject = captureObject;
Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
CameraParameters c = new CameraParameters();
c.hologramOpacity = 0.0f;
c.cameraResolutionWidth = cameraResolution.width;
c.cameraResolutionHeight = cameraResolution.height;
c.pixelFormat = CapturePixelFormat.BGRA32;
captureObject.StartPhotoModeAsync(c, false, OnPhotoModeStarted);
}
In the end, you'll also use the same clean-up code presented here
void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
{
photoCaptureObject.Dispose();
photoCaptureObject = null;
}
After these steps, you can choose which type of photo to capture.
Capture a photo to a file
The simplest operation is to capture a photo directly to a file. The photo can be saved as a JPG or a PNG.
If you successfully started photo mode, take a photo and store it on disk
private void OnPhotoModeStarted(PhotoCapture.PhotoCaptureResult result)
{
if (result.success)
{
string filename = string.Format(@"CapturedImage{0}_n.jpg", Time.time);
string filePath = System.IO.Path.Combine(Application.persistentDataPath, filename);
photoCaptureObject.TakePhotoAsync(filePath, PhotoCaptureFileOutputFormat.JPG, OnCapturedPhotoToDisk);
}
else
{
Debug.LogError("Unable to start photo mode!");
}
}
After capturing the photo to disk, exit photo mode and then clean up your objects
void OnCapturedPhotoToDisk(PhotoCapture.PhotoCaptureResult result)
{
if (result.success)
{
Debug.Log("Saved Photo to disk!");
photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}
else
{
Debug.Log("Failed to save Photo to disk");
}
}
Capture a photo to a Texture2D with location
When capturing data to a Texture2D, the process is similar to capturing to disk.
Follow the setup process above.
In OnPhotoModeStarted, capture a frame to memory.
private void OnPhotoModeStarted(PhotoCapture.PhotoCaptureResult result)
{
if (result.success)
{
photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
}
else
{
Debug.LogError("Unable to start photo mode!");
}
}
You'll then apply your result to a texture and use the common clean-up code above.
void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
if (result.success)
{
// Create our Texture2D for use and set the correct resolution
Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
Texture2D targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);
// Copy the raw image data into our target texture
photoCaptureFrame.UploadImageDataToTexture(targetTexture);
// Do as we wish with the texture such as apply it to a material, etc.
}
// Clean up
photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}
Locatable camera
To place this texture in the scene and display it using the locatable camera matrices, add the following code to OnCapturedPhotoToMemory in the result.success check:
if (photoCaptureFrame.hasLocationData)
{
photoCaptureFrame.TryGetCameraToWorldMatrix(out Matrix4x4 cameraToWorldMatrix);
Vector3 position = cameraToWorldMatrix.GetColumn(3) - cameraToWorldMatrix.GetColumn(2);
Quaternion rotation = Quaternion.LookRotation(-cameraToWorldMatrix.GetColumn(2), cameraToWorldMatrix.GetColumn(1));
photoCaptureFrame.TryGetProjectionMatrix(Camera.main.nearClipPlane, Camera.main.farClipPlane, out Matrix4x4 projectionMatrix);
}
Unity has provided sample code for applying the projection matrix to a specific shader on their forums.
Capture a photo and interact with the raw bytes
To interact with the raw bytes of an in memory frame, follow the same setup steps as above and OnPhotoModeStarted as in capturing a photo to a Texture2D. The difference is in OnCapturedPhotoToMemory where you can get the raw bytes and interact with them.
In this example, you'll create a List to be further processed or applied to a texture via SetPixels()
void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
if (result.success)
{
List<byte> imageBufferList = new List<byte>();
// Copy the raw IMFMediaBuffer data into our empty byte list.
photoCaptureFrame.CopyRawImageDataIntoBuffer(imageBufferList);
// In this example, we captured the image using the BGRA32 format.
// So our stride will be 4 since we have a byte for each rgba channel.
// The raw image data will also be flipped so we access our pixel data
// in the reverse order.
int stride = 4;
float denominator = 1.0f / 255.0f;
List<Color> colorArray = new List<Color>();
for (int i = imageBufferList.Count - 1; i >= 0; i -= stride)
{
float a = (int)(imageBufferList[i - 0]) * denominator;
float r = (int)(imageBufferList[i - 1]) * denominator;
float g = (int)(imageBufferList[i - 2]) * denominator;
float b = (int)(imageBufferList[i - 3]) * denominator;
colorArray.Add(new Color(r, g, b, a));
}
// Now we could do something with the array such as texture.SetPixels() or run image processing on the list
}
photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}
Video capture
Namespace (before Unity 2019): UnityEngine.XR.WSA.WebCam
Namespace (Unity 2019 and later): UnityEngine.Windows.WebCam
Type: VideoCapture
VideoCapture functions similarly to PhotoCapture. The only two differences are that you must specify a Frames Per Second (FPS) value and you can only save directly to disk as a .mp4 file. The steps to use VideoCapture are as follows:
- Create a VideoCapture object
- Create a CameraParameters object with the settings you want
- Start Video Mode via StartVideoModeAsync
- Start recording video
- Stop recording video
- Stop Video Mode and clean up resources
Start by creating our VideoCapture object VideoCapture m_VideoCapture = null;
void Start ()
{
VideoCapture.CreateAsync(false, OnVideoCaptureCreated);
}
Next, set up the parameters you'll want for the recording and start.
void OnVideoCaptureCreated(VideoCapture videoCapture)
{
if (videoCapture != null)
{
m_VideoCapture = videoCapture;
Resolution cameraResolution = VideoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
float cameraFramerate = VideoCapture.GetSupportedFrameRatesForResolution(cameraResolution).OrderByDescending((fps) => fps).First();
CameraParameters cameraParameters = new CameraParameters();
cameraParameters.hologramOpacity = 0.0f;
cameraParameters.frameRate = cameraFramerate;
cameraParameters.cameraResolutionWidth = cameraResolution.width;
cameraParameters.cameraResolutionHeight = cameraResolution.height;
cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
m_VideoCapture.StartVideoModeAsync(cameraParameters,
VideoCapture.AudioState.None,
OnStartedVideoCaptureMode);
}
else
{
Debug.LogError("Failed to create VideoCapture Instance!");
}
}
Once started, begin the recording
void OnStartedVideoCaptureMode(VideoCapture.VideoCaptureResult result)
{
if (result.success)
{
string filename = string.Format("MyVideo_{0}.mp4", Time.time);
string filepath = System.IO.Path.Combine(Application.persistentDataPath, filename);
m_VideoCapture.StartRecordingAsync(filepath, OnStartedRecordingVideo);
}
}
After recording has started, you could update your UI or behaviors to enable stopping. Here you just log.
void OnStartedRecordingVideo(VideoCapture.VideoCaptureResult result)
{
Debug.Log("Started Recording Video!");
// We will stop the video from recording via other input such as a timer or a tap, etc.
}
At a later point, you'll want to stop the recording using a timer or user input, for instance.
// The user has indicated to stop recording
void StopRecordingVideo()
{
m_VideoCapture.StopRecordingAsync(OnStoppedRecordingVideo);
}
Once the recording has stopped, stop video mode and clean up your resources.
void OnStoppedRecordingVideo(VideoCapture.VideoCaptureResult result)
{
Debug.Log("Stopped Recording Video!");
m_VideoCapture.StopVideoModeAsync(OnStoppedVideoCaptureMode);
}
void OnStoppedVideoCaptureMode(VideoCapture.VideoCaptureResult result)
{
m_VideoCapture.Dispose();
m_VideoCapture = null;
}
Troubleshooting
- No resolutions are available
- Ensure the WebCam capability is specified in your project.
Next Development Checkpoint
If you're following the Unity development checkpoint journey we've laid out, you're in the midst of exploring the Mixed Reality platform capabilities and APIs. From here, you can continue to the next topic:
Or jump directly to deploying your app on a device or emulator:
You can always go back to the Unity development checkpoints at any time.