Skip to content

Video Preprocessing

Correctly formatting the video input is the most critical step for using the Direct API. Unlike high-level video platforms, the VitalLens API does not accept arbitrary video files (like MP4s) directly. It expects a pre-calculated, highly compressed tensor.

The Goal

The API expects a Base64-encoded string representing a raw tensor of shape (frames, 40, 40, 3). To get there from a standard video file, you must perform three specific operations in order:

  1. Crop the Region of Interest (ROI).
  2. Scale the ROI to exactly 40x40 pixels.
  3. Encode the pixel data as raw RGB bytes.

Step 1: Cropping

First, you must define a single Region of Interest (ROI) that contains the subject.

The Ideal Crop

The crop should center on the face but must extend downwards to include the upper chest and shoulders.

  • Face: Provides the strongest rPPG signal (blood volume changes) for Heart Rate and HRV.
  • Upper Chest: Provides subtle motion cues required for Respiratory Rate.

One Crop per Request

Apply the same crop to every frame of a request, as in the example below. Do not re-detect the face and move the crop from frame to frame. The model accounts for the subject's movement within the crop.

If the subject moves out of the crop, start a new request with a new crop.

Example Values

For our sample video, the recommended crop is:

  • Width: 250px
  • Height: 400px
  • X-Coordinate: 335
  • Y-Coordinate: 60

Recommended cropping

Common Errors

  • No Cropping: Sending the full wide-angle frame means the face is only a few pixels wide after resizing. The signal will be weak and accuracy low.
  • Face-Only Crop: Cropping tightly around the chin excludes the upper torso. This degrades Respiratory Rate accuracy, as the model cannot "see" the breathing motion.
  • Moving Crop: Using a new face detection box for every frame. Even if the subject sits still, the box jitters and degrades accuracy, especially for HRV and Respiratory Rate.

Step 2: Scaling

Once you have your crop, you must resize it to exactly 40x40 pixels.

How to Scale

Use area averaging (a box filter) to resize the image: Each output pixel is the average of all source pixels it covers. This compresses the visual information of the entire face/chest area into the small 40x40 grid.

  • OpenCV: cv2.resize(crop, (40, 40), interpolation=cv2.INTER_AREA)
  • Pillow: img.resize((40, 40), Image.BOX)
  • ffmpeg: scale=40:40:flags=area

Common Errors

  • Tiling: A frequent mistake is taking a 40x40 pixel "patch" from the center of the face and discarding the rest. This removes 90% of the signal. The API needs the entire ROI compressed, not a cutout.
  • Padding: Do not "letterbox" or pad the image with black bars to make it square. This introduces artificial edges that confuse the model.
  • Bad Algorithms: Avoid nearest neighbor, and plain bilinear or bicubic interpolation. When shrinking a large crop, they only use a few source pixels per output pixel, which discards most of the signal and adds noise.

Step 3: Encoding

Finally, you must convert the pixel data into a format the API can parse.

The Format

The API requires Raw RGB24 bytes.

  1. Color Space: RGB (3 channels).
  2. Pixel Format: uint8 (values 0-255).
  3. Serialization: Concatenate the bytes of every frame and encode the result as a standard Base64 string.

Common Errors

Common Errors

  • BGR Color Space: Libraries like OpenCV (cv2.imread) default to BGR. If you send BGR data, the model will see blue skin and orange lips, resulting in garbage vital signs.
  • Float Normalization: Do not divide pixel values by 255. The API expects integers (e.g., 255), not floats (e.g., 1.0).
  • 4-Channel Images: Do not send RGBA or BGRA data. The alpha channel changes the byte alignment and breaks the reshaping logic (causing 422 errors).

Frame Sampling

If you send fps, the API assumes that the frames are evenly spaced in time (frame i was captured at i / fps seconds).

  • Send consecutive frames from the camera or video, and pass their actual frame rate.
  • To send fewer frames, keep every n-th frame (e.g. every 2nd frame of a 30 fps camera) and divide fps by n.

If your frames are not evenly spaced, for example from a variable frame rate recording, send timestamps instead of fps: The capture time of each frame in seconds (see the API reference).

Common Errors

  • Timer-Based Selection: Picking frames with a timer (e.g. one every 100 ms) produces uneven spacing, because the timer and the camera do not run in sync.
  • Dropped or Duplicated Frames: Gaps or repeats break the even spacing. Check for them in live capture.

Verification (Sanity Check)

Because the processing happens blindly on the server, we strongly recommend verifying your payload locally before sending it.

You can run this command to extract the video from your JSON payload and save the first frame as an image. This lets you see what the API sees.

Command

# Requires: jq, base64, ffmpeg
jq -r .video payload.json | base64 -d | \
ffmpeg -f rawvideo -pixel_format rgb24 -video_size 40x40 \
-i - -frames:v 1 -vf "scale=400:400:flags=neighbor" -y sanity_check.png

Result

If your preprocessing is correct, sanity_check.png should look like the image below (pixelated, but recognizable).

Sanity check result for our sample request

What to Look For

  • ✅ Correct: A single, pixelated face and upper chest filling the square. Colors look natural.
  • ❌ Blue/Orange Skin: You sent BGR data. Convert to RGB.
  • ❌ Grid of Faces: You tiled the image instead of scaling it.
  • ❌ Black Bars: You padded the image instead of scaling or cropping correctly.
  • ❌ Slanted/Skewed: You likely sent the wrong resolution or extra bytes (like an Alpha channel).