Frame

interface Frame extends HybridObject

Represents a single Frame the Camera "sees".

Discussion

Typically, a CameraOutput like CameraFrameOutput produces instances of this type.

Discussion

The Frame is either planar or non-planar (isPlanar).

• A planar Frame's (often YUV) pixel data can be accessed on the CPU via Frame.getPlanes(), where each FramePlane represents one plane of the pixel data. In YUV, this is often 1 Y Plane in full resolution, and 1 UV Plane in half resolution. In this case, Frame.getPixelBuffer()'s behaviour is undefined - sometimes it contains the whole pixel data in a contiguous block of memory, sometimes it just contains the data in the control block.
• A non-planar Frame's (often RGB) pixel data can be accessed on the CPU via Frame.getPixelBuffer() as one contiguous block of memory. In this case, Frame.getPlanes() will return an empty array ([]).

It is recommended to not rely on Frame.getPixelBuffer() for planar Frames.

Discussion

Frames have to be disposed (see dispose()) to free up the memory, otherwise the producing pipeline might stall.

Properties

bytesPerRow

readonly bytesPerRow: number

Get the number of bytes per row of the underlying pixel buffer.

This may return 0 if the Depth is planar, in which case you should get the number of bytes per row of individual planes using getPlanes() / FramePlane.bytesPerRow.

See

• width
• getPixelBuffer()

cameraIntrinsicMatrix?

readonly optional cameraIntrinsicMatrix?: number[]

Gets the Camera intrinsic matrix for this Frame.

The returned array is a 3x3 matrix with column-major ordering. Its origin is the top-left of the Frame.

K = [ fx   0  cx ]
    [  0  fy  cy ]
    [  0   0   1 ]

• fx, fy: focal length in pixels
• cx, cy: principal point in pixels

Example

const matrix = frame.cameraIntrinsicMatrix
const fx = matrix[0]
const fy = matrix[4]

hasNativeBuffer

readonly hasNativeBuffer: boolean

Get whether this Frame has a Native Buffer attached to it.

Discussion

A Native Buffer can be sampled on the GPU as an external texture, or used in Media Encoder APIs.

Discussion

Usually a Frame has a Native Buffer if the platform supports it. Only legacy platforms don't support Native Buffers.

See

• getNativeBuffer()
• NativeBuffer

hasPixelBuffer

readonly hasPixelBuffer: boolean

Get whether this Frame has a CPU-accessible Pixel Buffer attached to it.

Discussion

Usually a Frame has an application-accessible Pixel Buffer if its pixelFormat is application-accessible - aka every PixelFormat except for 'private'. On iOS, every Frame has an application-accessible Pixel Buffer.

See

getPixelBuffer()

height

readonly height: number

Gets the total height of the Frame.

If this is a planar Frame (see isPlanar), the individual planes (see getPlanes()) will likely have different heights than the total height. For example, a YUV Frame's UV Plane is half the size of its Y Plane.

isMirrored

readonly isMirrored: boolean

Indicates whether this Frame's pixel data must be interpreted as mirrored along the vertical axis.

This value is always relative to the target output's mirror mode. (see CameraOutputConfiguration.mirrorMode)

Frames are not automatically mirroring their pixels because physically mirroring buffers is expensive. The Camera streams frames in the hardware's native mirroring mode and adjusts presentation later using metadata (such as EXIF), transforms, or here; a flag.

• If the output is mirrored but the underlying pixel buffer is NOT, isMirrored will be true. You must treat the pixels as flipped (for example, read them right-to-left).
• If both the output and the pixel buffer are mirrored (for example when FrameOutputOptions.enablePhysicalBufferRotation is enabled), isMirrored will be false because the buffer already matches the output orientation.
• If neither the output nor the pixel buffer are mirrored, isMirrored will be false.

Discussion

If you process the Frame, you must interpret the pixel data in this Frame as mirrored if isMirrored is true. In rendering libraries you would typically scale the Frame by -1 on the X axis if it is mirrored to cancel out the mirroring at render time.

Most ML libraries (for example Google MLKit) accept a mirroring flag for input images — pass isMirrored directly in those cases.

isPlanar

readonly isPlanar: boolean

Returns whether this Frame is planar or non-planar.

• If a Frame is planar (e.g. YUV), you should only access its pixel buffer via getPlanes() instead of getPixelBuffer().
• If a Frame is non-planar (e.g. RGB), you can access its pixel buffer via getPixelBuffer() instead of getPlanes().

isValid

readonly isValid: boolean

Gets whether this Frame is still valid, or not. If the Frame is invalid, you cannot access its data anymore. A Frame is automatically invalidated via dispose().

orientation

readonly orientation: CameraOrientation

The rotation of this Frame relative to the CameraOutput's target output orientation. (see CameraOutput.outputOrientation)

Frames are not automatically rotated to 'up' because physically rotating buffers is expensive. The Camera streams frames in the hardware's native orientation and adjusts presentation later using metadata (such as EXIF), transforms, or here; a flag.

Examples:

• 'up' — The Frame is already correctly oriented.
• 'right' — The pixel data is rotated +90° relative to the desired orientation.
• 'left' — The pixel data is rotated -90° relative to the desired orientation.
• 'down' — The pixel data is rotated 180° upside down relative to the desired orientation.

Discussion

If you process the Frame, you must interpret the pixel data in this Frame to be rotated by this orientation. In rendering libraries you would typically counter-rotate the Frame by this orientation to get it "up-right".

Most ML libraries (for example Google MLKit) accept an orientation flag for input images - pass orientation directly in those cases.

pixelFormat

readonly pixelFormat: PixelFormat

Gets the PixelFormat of this Frame's pixel data. Common formats are 'yuv-420-8-bit-video' for native YUV Frames, 'rgb-bgra-32-bit' for processed RGB Frames, or 'private' for zero-copy GPU-only Frames.

Discussion

If the Frame is a GPU-only buffer, its pixelFormat is 'private', wich allows zero-copy importing it into GPU pipelines directly, however its pixel data is likely not accessible on the CPU. You can use getNativeBuffer() to get a handle to the native GPU-based buffer, which can be used in GPU pipelines like Skia, or custom implementations using OpenGL/Vulkan/Metal shaders with external texture importers.

timestamp

readonly timestamp: number

Gets the presentation timestamp this Frame was timed at.

width

readonly width: number

Gets the total width of the Frame.

If this is a planar Frame (see isPlanar), the individual planes (see getPlanes()) will likely have different widths than the total width. For example, a YUV Frame's UV Plane is half the size of its Y Plane.

Methods

convertCameraPointToFramePoint(...)

convertCameraPointToFramePoint(cameraPoint: Point): Point

Converts the given cameraPoint in camera sensor coordinates into a Point in Frame coordinates, relative to this Frame.

Example

const cameraPoint = { x: 0.5, y: 0.5 }
const framePoint = frame.convertCameraPointToFramePoint(cameraPoint)
console.log(framePoint) // { x: 960, y: 360 }

convertFramePointToCameraPoint(...)

convertFramePointToCameraPoint(framePoint: Point): Point

Converts the given framePoint in Frame coordinates relative to this Frame into a Point in camera sensor coordinates.

Example

const framePoint = { x: frame.width / 2, y: frame.height / 2 }
const cameraPoint = frame.convertFramePointToCameraPoint(framePoint)
console.log(cameraPoint) // { x: 0.5, y: 0.5 }

getNativeBuffer()

getNativeBuffer(): NativeBuffer

Get a NativeBuffer that points to this Frame.

This is a shared contract between libraries to pass Native Buffers around without natively typed bindings.

The Native Buffer can be sampled by the GPU as an external texture, or passed to Media Encoder APIs.

The NativeBuffer must be released again by its consumer via release(), otherwise the Camera pipeline might stall.

Discussion

Libraries like Skia or WebGPU implement Native Buffer APIs.

Examples

if (frame.hasNativeBuffer) {
  const nativeBuffer = frame.getNativeBuffer()
  console.log(`Native Buffer pointer: ${nativeBuffer.pointer}`)
  nativeBuffer.release()
}

Import a Frame into Skia via NativeBuffer

if (frame.hasNativeBuffer) {
  const nativeBuffer = frame.getNativeBuffer()
  const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
  // Render `image` via Skia APIs
  image.dispose()
  nativeBuffer.release()
}

Import a Frame into WebGPU via NativeBuffer

const device = ... // WebGPU device
if (frame.hasNativeBuffer) {
  const nativeBuffer = frame.getNativeBuffer()
  const videoFrame = RNWebGPU.createVideoFrameFromNativeBuffer(nativeBuffer.pointer)
  const externalTexture = device.importExternalTexture({
    source: videoFrame,
    label: 'camera-frame'
  })
  // After submitting commands that sample `externalTexture`:
  externalTexture.destroy()
  videoFrame.release()
  nativeBuffer.release()
}

getPixelBuffer()

getPixelBuffer(): ArrayBuffer

Gets the Frame's entire pixel data as a full contiguous ArrayBuffer, if it contains one.

Discussion

• If the frame is planar (see Frame.isPlanar, e.g. YUV), this might or might not contain valid data.
• If the frame is not planar (e.g. RGB), this will contain the entire pixel data.

Discussion

This does not perform a copy, but since the Frame's data is stored on the GPU, it might lazily perform a GPU -> CPU download.

Discussion

Once the Frame gets invalidated (isValid == false), this ArrayBuffer is no longer safe to access.

Example

if (frame.hasPixelBuffer) {
  const pixelBuffer = frame.getPixelBuffer()
  console.log(`Frame has ${pixelBuffer.byteLength} bytes.`)
}

getPlanes()

getPlanes(): FramePlane[]

Returns each plane of a planar Frame (see isPlanar). If this Frame is non-planar, this method returns an empty array ([]).