Video storage

Overview

The primary input of U-Query are videos - either static (video files) or live (RTSP streams).

Video files have to be indexed first before being analysed by U-Query. For RTSP streams, indexing can be done continuously.

U-Query can store the input of both static and live videos. In case of live videos, U-Query creates small (one GOP length) .mp4 files from RTSP stream inputs and stores them in a structured way. Video retention time can be configured.

Video storage data is located under ${U_QUERY_DATA}\indexer.

This folder contains the video storage, video configuration and the metadata index for static videos (the index of live videos is typically stored in kafka).

Videos can be organized in a free folder structure. If a folder contains a video.prototxt file, the folder is treated as a video store with the following structure:

Videos in the video store folder are organized in the following structure: yyyy-MM-dd/hh/mm_ss_fff.mp4

For example:

2023-07-12
|-- 00
|   |-- 00_00_000.mp4
|   |-- 00_02_000.mp4
|   ...
|   |-- 59_58_000.mp4
|-- 01
|   |-- 00_00_000.mp4
|   |-- 00_02_000.mp4
|   ...
|   |-- 59_58_000.mp4
|  ...

All configuration data of a video store is contained in the video.prototxt file.

Add an RTSP stream

New live streams can be added to the system with the following steps:

1. Create a new folder for the stream.

Go to the indexer data folder at: ${U_QUERY_DATA}/indexer (U_QUERY_DATA is set in the .env file of the U-Query deployment) and create a new folder with the name of the stream.

2. Create stream configuration file.

Create a new file with the name video.prototxt containing the configuration of the stream:

rtsp_url: "rtsp://1.2.3.4:554/live.sdp"
retention_in_seconds: 86400
retention_check_period_in_seconds: 60
indexer {
  detector_fps: 5 
  kafka_index {
    boostrap_servers: "broker:9092"
    topic_prefix: "cam1"
  }
}

• retention_in_seconds: Specifies how long U-Query will store videos. Calculate this carefully based on how much disk space is available for video storage.

• detector_fps: Controls the analysis frame rate. GPU resource consumption is directly proportional to this frame rate, depending on the use case: if the FPS of the RTSP stream is high (>=10), consider setting this to a lower value so as not to overload the GPU. 5 is a good value for most use cases.

• bootstrap_servers and topic prefix: Control where the output stream will be written to.

3. Restart the indexer server.

Restart the indexer server to use the new configuration.

docker restart u_query_indexer

Add video files

The quickest way to test a U-Query use case is to get video footage from a given camera, index it once, then try out different queries.

Uploading and indexing a new video file from the UI is not supported yet - follow the steps below for manual configuration.

1. Convert the video to MP4 with H264.

If the file is not in .mp4 format, it will have to be converted first by the H264 codec.

Use ffprobe to query the codec used in the video file first.

If the codec is H264 but not MP4 use this. This will be fast as if it not transcoding the video file.

ffmpeg -an -i input.avi -movflags +faststart output.mp4

If the codec is not H264 use this. This will take time as it will transcode the video file.

ffmpeg -an -i input.avi -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -level 3 -movflags +faststart output.mp4

2. Index the video file with the U-Query indexer.

The U-Query indexer is available as a docker container.

First make sure U-Query services are running on the machine (This process requires a healthy triton container running)

Let's assume we want to index /home/u-query/input.mp4

Run this command.

docker run --net u_query_net --ipc host -v /home/u-query/:/work ultinous/u_query_indexer_indexer:0.10.1 --triton_host triton --triton_port 8001 --cvinput /work/input.mp4 --out_folder /work/out --debug_video

The --debug_video option can be left out in this case the process is faster and does not generate an annotated debug video.

This an example output of a successful run:

2023-07-28 12:12:39,451 - 1 - INFO - Indexer started with args: Namespace(cvinput='/work/1.mp4', fps=1000, out_folder='/work/out', kafka_broker=None, prefix='test', debug_video=True, json_dump=False, triton_host='triton', triton_port='8001', vidstore_host=None, vidstore_port=None, obj_det_model='yolov7x')
2023-07-28 12:12:39,468 - 1 - INFO - Creating YOLO7 detector, model=yolov7x, input_dim=640
2023-07-28 12:12:39,474 - 1 - INFO - Opening video source: /work/1.mp4
2023-07-28 12:12:39,509 - 1 - INFO - FPS: 8
2023-07-28 12:12:39,509 - 1 - INFO - Total number of frames: 1428
2023-07-28 12:13:46,334 - 1 - INFO - VideoReader: 1000 of 1428 (70.0%)
2023-07-28 12:14:05,773 - 1 - INFO - Indexer finished, main thread exiting.

The process creates an out folder containing .pb index files. If --debug_video was given it will also produce a debug.mp4 file.

3. Add the video store to the video storage.

Create a new folder under ${U_QUERY_DATA}\indexer with the name of the new video store. Create the finished folder and place and rename the mp4 file under using the structure: yyyy-MM-dd/hh/mm_ss_fff.mp4 Use the start time of the video to fill the date and time info in the folders.

Create the index folder and copy all the index file (.pb) generate the process above.

Create the video.prototxt config file with the following content:

indexer: {
  file_index: {}
}

4. Restart the indexer.

docker restart u_query_indexer

Remove a video store

To remove a video store, stop the indexer service first:

docker stop u_query_indexer

Delete the video store folder and restart the indexer:

docker start u_query_indexer

Removing mp4 files manually

Mp4 files can be removed manually from the storage without breaking the store. This can be useful e.g. for removing personal data.