Feature matching vs Optical flow for camera movement and zoom detection

In our work it is beneficial to be able to deduce camera movement and zoom level from the video without having access to the telemetry data such as GPS or IMU / accelerometer data.

There are two major approaches to solving this problem which we will cover here:

(1) Feature Matching

(2) Optical Flow

Optical Flow work example

Feature Detection, Camera transformations and Feature Matching approach

To output camera angle and zoom level, you’ll typically need to employ methods that involve frame comparison, feature matching, and estimation of camera parameters based on these features.

1. Prerequisites

OpenCV: Install OpenCV if not already installed (pip install opencv-python).
NumPy: Generally comes with OpenCV, but ensure it’s installed (pip install numpy).

2. Feature Detection and Matching

To estimate camera motion and zoom levels, first detect key features across consecutive frames and then match these features to estimate the relative transformations.

pythonCopy codeimport cv2
import numpy as np

def get_features(frame):
    # Convert frame to grayscale
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    # Use ORB to detect features
    orb = cv2.ORB_create()
    keypoints, descriptors = orb.detectAndCompute(gray, None)
    return keypoints, descriptors

def match_features(desc1, desc2):
    # Create BFMatcher and match descriptors
    bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
    matches = bf.match(desc1, desc2)
    matches = sorted(matches, key=lambda x: x.distance)
    return matches

3. Estimate Camera Transformations

Using the matches and keypoints, estimate the transformation matrix using RANSAC, which helps in determining camera movement and zooming robustly against noise and outliers.

pythonCopy codedef estimate_motion(matches, kp1, kp2):
    # Extract location of good matches
    points1 = np.zeros((len(matches), 2), dtype=np.float32)
    points2 = np.zeros((len(matches), 2), dtype=np.float32)

    for i, match in enumerate(matches):
        points1[i, :] = kp1[match.queryIdx].pt
        points2[i, :] = kp2[match.trainIdx].pt

    # Find homography
    matrix, mask = cv2.findHomography(points1, points2, cv2.RANSAC, 5.0)
    return matrix

def get_camera_parameters(matrix):
    # Decompose homography matrix to get scale, translation and rotation
    _, _, trans, _, _, _, scale = cv2.decomposeHomographyMat(matrix)
    return trans, scale

4. Process Video Stream

Integrate everything to process a video stream:

pythonCopy codecap = cv2.VideoCapture('path_to_uav_footage.mp4')
ret, previous_frame = cap.read()

prev_keypoints, prev_descriptors = get_features(previous_frame)

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    
    keypoints, descriptors = get_features(frame)
    matches = match_features(prev_descriptors, descriptors)
    matrix = estimate_motion(matches, prev_keypoints, keypoints)
    trans, scale = get_camera_parameters(matrix)
    
    # Assuming scale and trans are being outputted directly
    print(f"Translation: {trans}, Scale: {scale}")
    
    # Update previous frame
    previous_frame = frame
    prev_keypoints = keypoints
    prev_descriptors = descriptors

cap.release()

5. Handling Zoom and Angle

Zoom Level: Can be inferred from the scaling part of the transformation matrix. A scale greater than 1 indicates zooming in, and less than 1 indicates zooming out.
Camera Angle: This is more complex to infer directly and might require additional information like initial camera orientation or 3D scene structure, which might be estimated using gyroscopic data if available, or more advanced scene reconstruction techniques.

6. Note

This example assumes that the camera’s internal parameters (like focal length, principal point) are constant or normalized, which might not hold true with significant zooming. More sophisticated methods might be required depending on the precision needed and the nature of the UAV footage.

This method gives a basic framework, but depending on the specifics of your UAV and its camera system, additional calibration or adjustments might be necessary. For professional or highly accurate applications, consider integrating IMU (Inertial Measurement Unit) data, which UAVs often have, to get precise measurements of camera orientation changes.

Optical Flow approach

Optical flow is a concept used in computer vision to estimate the motion of objects between two consecutive frames of a video based on the apparent velocities of the objects in the scene. It assumes that the flow is derived from the movement of brightness patterns in the image, and typically, that these patterns do not change abruptly.

An example of Optical Flow implementation can be found is this Github project: https://github.com/antiboredom/camera-motion-detector. We provide the relevant code excerpt below.

 for frame in frames:
        frame_num += 1
        progress_bar.update(1)

        frame = frame[rect_y : rect_y + rect_h, rect_x : rect_x + rect_w]
        next_frame = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)

        if gpu:
            gpu_frame.upload(next_frame)
            gpu_prev.upload(prvs)

            # calculate optical flow
            flow = cv.cuda_FarnebackOpticalFlow.calc(
                gpu_flow,
                gpu_prev,
                gpu_frame,
                None,
            )

            flow_x = cv.cuda_GpuMat(flow.size(), cv.CV_32FC1)
            flow_y = cv.cuda_GpuMat(flow.size(), cv.CV_32FC1)
            cv.cuda.split(flow, [flow_x, flow_y])

            mag, ang = cv.cuda.cartToPolar(flow_x, flow_y, angleInDegrees=True)

            mean_mag = np.median(mag.download())
            mean_ang = np.median(ang.download())

            flow = flow.download()


   # get the actual pixel coords of the flow
        flow_coords = flow + empty

        xvals = flow_coords.ravel()[::2] - (w / 2)
        yvals = flow_coords.ravel()[1::2] - (h / 2)

        # calculate the distances from center points
        dists = np.sqrt(np.square(xvals) + np.square(yvals))

        dist_diff = dists >= empty_dists
        zoom_in_factor = np.count_nonzero(dist_diff) / len(dist_diff)

        if show:
            angs.append(mean_ang)
            if len(angs) > 10:
                angs.pop(0)
            mags.append(mean_mag)
            if len(mags) > 10:
                mags.pop(0)
            zooms.append(zoom_in_factor)
            if len(zooms) > 10:
                zooms.pop(0)

            draw_lines(frame, flow, grid=10)
            draw_text(frame, np.mean(mags), np.mean(angs), np.mean(zooms), frame_num)
            cv.imshow("Preview", frame)
            k = cv.waitKey(30) & 0xFF
            if k == 27:
                break

        prvs = next_frame

        data.append(
            {
                "frame": frame_num,
                "mag": mean_mag,
                "ang": mean_ang,
                "zoom": zoom_in_factor,
            }
        )

This is what the result looks like when running the detect.py code with one of our videos:

frame	mag	ang	zoom
2	7.865718	177.85184	0.5148555555555560
3	6.7536435	178.8259	0.5163555555555560
4	6.088938	176.2493	0.5136666666666670
5	5.7395935	175.17606	0.5128222222222220
6	5.3491364	169.88715	0.5126666666666670
7	5.234653	162.72421	0.5135666666666670
8	5.4581466	149.06877	0.5144111111111110
9	5.795512	139.33131	0.5172111111111110
10	6.4585404	130.55109	0.5151666666666670
11	6.7035637	121.55008	0.5129333333333330
12	7.265873	116.424255	0.5140555555555560
13	7.5837803	111.431595	0.514
14	7.8835945	107.09203	0.5138777777777780
15	8.027069	103.35013	0.5140444444444440
16	8.294752	104.49362	0.5143222222222220
17	9.019862	111.09737	0.5152777777777780
18	8.829447	109.35805	0.5158555555555560
19	8.673782	105.788025	0.5143888888888890
20	8.702745	104.03821	0.5165222222222220
21	8.655909	101.56949	0.5137111111111110
22	8.655781	101.051895	0.5134666666666670
23	8.407052	100.37541	0.5128
24	7.5121727	99.98055	0.5123333333333330
25	6.505313	100.03925	0.5110888888888890
26	5.7058544	99.79213	0.5109333333333330
27	4.8263493	97.97284	0.5188333333333330
28	4.2676554	95.798706	0.5275222222222220
29	3.3682094	91.92259	0.5299333333333330
30	3.77982	105.49237	0.9114777777777780
31	3.541996	109.67978	0.9387888888888890
32	3.2588053	127.96671	0.9778777777777780

Using optical flow to analyze UAV footage for camera motion and zoom levels offers a distinct approach compared to feature matching techniques like those based on homography. Both methods have their strengths and challenges, and choosing between them largely depends on the specific requirements and constraints of your application.

Key Principles of Optical Flow:

Brightness Constancy: The brightness of any point in the image remains constant over time, despite its movement. This means that if a particular feature or color moves from one location to another between frames, its intensity should remain the same.
Temporal Persistence: Objects and features in a scene typically move only a small amount between successive frames. This assumption allows for the estimation of motion vectors that are relatively small and hence more manageable to compute.
Spatial Coherence: Points in a region are likely to move in a similar manner to their immediate neighbors. This coherence helps in producing smoother and more continuous flow fields.

Types of Optical Flow:

Optical flow methods can be broadly classified into two categories:

Dense Optical Flow: This method calculates the motion vector for every single pixel in the image, providing a comprehensive flow field that represents the motion of all visible points. Algorithms like the Lucas-Kanade method in its global form, Horn-Schunck, and Farneback’s algorithm are examples of dense optical flow.
Sparse Optical Flow: This focuses only on the flow vectors at certain points of interest (features) in the image rather than every pixel. The Lucas-Kanade method in its original form is an example of sparse optical flow, where flow vectors are computed for features identified by corner detection or similar methods.

How Optical Flow Works:

To calculate optical flow, algorithms typically start by identifying points of interest in the first frame (in the case of sparse flow) or using all pixels (for dense flow). Then, they attempt to locate the same points in the subsequent frame by minimizing the difference in their appearance, under the constraints of brightness constancy and spatial coherence.

Optical flow is a powerful tool in computer vision, but it does have limitations, particularly with regard to handling large displacements, occlusions, and significant changes in lighting or appearance. Advanced techniques often combine optical flow with other methods to address these challenges.

Comparison: Feature Matching vs Optical Flow approach

Optical Flow Approach

Strengths:

Real-time Analysis: Optical flow, especially when implemented on a GPU as in your example, is well-suited for real-time applications. It can handle dynamic environments efficiently by calculating the motion between two consecutive frames at the pixel level.
Detailed Motion Vector: Optical flow provides a dense motion vector (flow field) that shows the movement of every pixel between frames. This is particularly useful for detecting subtle movements and can accurately measure camera panning and tilting, as well as zooming actions.
Zoom Detection: By comparing distances from the center points of flow vectors, your method calculates zoom factors effectively, which can distinguish between zoom-in and zoom-out actions based on the distribution of flow vector lengths.
Angle and Magnitude Calculations: Calculating the median angle and magnitude of flow vectors provides a measure of the dominant direction of movement and the speed of camera motion, respectively.

Challenges:

Noise Sensitivity: Optical flow can be sensitive to noise and rapid changes in lighting or appearance, which might require additional filtering or robustness measures.
Computational Resources: While GPU acceleration significantly speeds up the processing, it still demands substantial computational resources, which can be a limiting factor in some environments.

Feature Matching Approach (based on homography)