A Multi-Threading Approach for Faster OpenCV Video Streaming in Python

Overview

In this article we will see two Python code examples, with without multi-threading, for reading video frames from a camera. We’ll see the difference of obtained FPS with/without Multi-threading.

What is Multi Threading?

A thread is a unit of execution within a process. Multithreading refers to concurrently executing multiple threads by rapidly switching the control of the CPU between threads (called context switching). In our example, we’ll see that Multi-threading enables faster real-time video processing with increased FPS (frames per second).

Basic Of Threading in Python

The following code snippet shows how to create threads using the threading module in python:

# importing the threading module 
import threading 
# importing the time module 
import time
# Function to print "Hello", however, the function sleeps
# for 2 seconds at the 11th iteration
def print_hello(): 
  for i in range(20):
    if i == 10:
      time.sleep(2)
    print("Hello")
# Function to print numbers till a given number
def print_numbers(num): 
  for i in range(num+1):
    print(i)
# Creating the threads. Target is set to the name of the
# function that neeeds to be executed inside the thread and
# args are the arguments to be supplied to the function that
# needs to be executed.
print("Greetings from the main thread.")
thread1 = threading.Thread(target = print_hello, args = ())
thread2 = threading.Thread(target = print_numbers, args = (10,))  
# Starting the two threads
thread1.start() 
thread2.start() 
print("It's the main thread again!")

Let’s try to make sense of the output by tracing the execution of the code:

1. The main thread executes. “Greetings from the main thread” is printed, threads thread1 and thread2 are created and started.
2. A context switch occurs, and thread1 starts executing.
3. After the first ten iterations, thread1 goes to sleep, and thread2 starts executing, finishing up before the next context switch.
4. Now, the main thread gains control of the CPU and prints “It’s the main thread again!”
5. Another context switch takes place, and thread2 resumes execution and finishes.
6. Since there are no more instructions to be executed by the main thread, the program terminates.

Using thread.join()

What if it was required to block the main thread until thread1 and thread2 finished executing? Then thread.join() would come in handy as it blocks the calling thread until the threa​d, whose join() method is called, is terminated:

# importing the threading module 
import threading 
# importing the time module 
import time
# Function to print "Hello", however, the function sleeps
# for 2 seconds at the 11th iteration
def print_hello(): 
  for i in range(20):
    if i == 10:
      time.sleep(2)
    print("Hello")
# Function to print numbers till a given number
def print_numbers(num): 
  for i in range(num+1):
    print(i)
# Creating the threads. Target is set to the name of the
# function that neeeds to be executed inside the thread and
# args are the arguments to be supplied to the function that
# needs to be executed.
print("Greetings from the main thread.")
thread1 = threading.Thread(target = print_hello, args = ())
thread2 = threading.Thread(target = print_numbers, args = (10,))  
# Starting the two threads
thread1.start() 
thread2.start() 
thread1.join()
thread2.join()
print("It's the main thread again!")
print("Threads 1 and 2 have finished executing.")

Reason multi-threading helps with faster processing

The video processing code is divided into two sections: reading the next available frame from a camera and doing video processing on the frame, such as running a deep learning model for face recognition, and so on.

Reading the next frame and processing happen sequentially in a program without multi-threading. The program waits for the next frame to become available before doing the necessary processing on it. The time it takes to read the frame is mostly related to the time it takes to request, wait for, and transmit the next video frame from camera to memory. The time it takes to conduct computations on the video frame, whether on the CPU or GPU, occupies the majority of the time spent on video processing.

In a program with multi-threading, reading the next frame and processing it are not required to be sequential. While one thread performs the task of reading the next frame, the main thread can use the CPU or GPU to process the last read frame . In this way by overlapping the two tasks, total time for reading and processing the frames can be reduced.

OpenCV Code— Without Multi-Threading

# importing required libraries 
import cv2 
import time
# opening video capture stream
vcap = cv2.VideoCapture(0)
if vcap.isOpened() is False :
    print("[Exiting]: Error accessing webcam stream.")
    exit(0)
fps_input_stream = int(vcap.get(5))
print("FPS of webcam hardware/input stream: {}".format(fps_input_stream))
grabbed, frame = vcap.read() # reading single frame for initialization/ hardware warm-up
# processing frames in input stream
num_frames_processed = 0 
start = time.time()
while True :
    grabbed, frame = vcap.read()
    if grabbed is False :
        print('[Exiting] No more frames to read')
        break
# adding a delay for simulating time taken for processing a frame 
    delay = 0.03 # delay value in seconds. so, delay=1 is equivalent to 1 second 
    time.sleep(delay) 
    num_frames_processed += 1
cv2.imshow('frame' , frame)
    key = cv2.waitKey(1)
    if key == ord('q'):
        break
end = time.time()
# printing time elapsed and fps 
elapsed = end-start
fps = num_frames_processed/elapsed 
print("FPS: {} , Elapsed Time: {} , Frames Processed: {}".format(fps, elapsed, num_frames_processed))
# releasing input stream , closing all windows 
vcap.release()
cv2.destroyAllWindows()

OpenCV Code — With Multi-Threading

# importing required libraries 
import cv2 
import time 
from threading import Thread # library for implementing multi-threaded processing
# defining a helper class for implementing multi-threaded processing 
class WebcamStream :
    def __init__(self, stream_id=0):
        self.stream_id = stream_id   # default is 0 for primary camera 
        
        # opening video capture stream 
        self.vcap      = cv2.VideoCapture(self.stream_id)
        if self.vcap.isOpened() is False :
            print("[Exiting]: Error accessing webcam stream.")
            exit(0)
        fps_input_stream = int(self.vcap.get(5))
        print("FPS of webcam hardware/input stream: {}".format(fps_input_stream))
            
        # reading a single frame from vcap stream for initializing 
        self.grabbed , self.frame = self.vcap.read()
        if self.grabbed is False :
            print('[Exiting] No more frames to read')
            exit(0)
# self.stopped is set to False when frames are being read from self.vcap stream 
        self.stopped = True
# reference to the thread for reading next available frame from input stream 
        self.t = Thread(target=self.update, args=())
        self.t.daemon = True # daemon threads keep running in the background while the program is executing 
        
    # method for starting the thread for grabbing next available frame in input stream 
    def start(self):
        self.stopped = False
        self.t.start()
# method for reading next frame 
    def update(self):
        while True :
            if self.stopped is True :
                break
            self.grabbed , self.frame = self.vcap.read()
            if self.grabbed is False :
                print('[Exiting] No more frames to read')
                self.stopped = True
                break 
        self.vcap.release()
# method for returning latest read frame 
    def read(self):
        return self.frame
# method called to stop reading frames 
    def stop(self):
        self.stopped = True
# initializing and starting multi-threaded webcam capture input stream 
webcam_stream = WebcamStream(stream_id=0) #  stream_id = 0 is for primary camera 
webcam_stream.start()
# processing frames in input stream
num_frames_processed = 0 
start = time.time()
while True :
    if webcam_stream.stopped is True :
        break
    else :
        frame = webcam_stream.read()
# adding a delay for simulating time taken for processing a frame 
    delay = 0.03 # delay value in seconds. so, delay=1 is equivalent to 1 second 
    time.sleep(delay) 
    num_frames_processed += 1
cv2.imshow('frame' , frame)
    key = cv2.waitKey(1)
    if key == ord('q'):
        break
end = time.time()
webcam_stream.stop() # stop the webcam stream
# printing time elapsed and fps 
elapsed = end-start
fps = num_frames_processed/elapsed 
print("FPS: {} , Elapsed Time: {} , Frames Processed: {}".format(fps, elapsed, num_frames_processed))
# closing all windows 
cv2.destroyAllWindows()

Download All the Codes

https://github.com/SihabSahariar/Multi-threading-OpenCV-

Resources

1. Python OpenCV streaming from camera — multithreading, timestamps
2. Video Streaming from IP Camera in Python Using OpenCV cv2.VideoCapture
3. How to capture multiple camera streams with OpenCV?
4. OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?
5. Storing RTSP stream as video file with OpenCV VideoWriter
6. OpenCV video saving
7. Python OpenCV multiprocessing cv2.VideoCapture mp4

References

1. https://github.com/PyImageSearch/imutils/tree/master/imutils/video
2. https://www.pyimagesearch.com/2015/12/21/increasing-webcam-fps-with-python-and-opencv/
3. https://forum.opencv.org/t/videocapture-opens-video-sources-by-multi-thread/8045