Protocol Documentation¶

Table of Contents¶

u_query/proto/backend_api.proto
u_query/proto/common_model.proto
u_query/proto/index.proto
u_query/proto/indexer.proto
u_query/proto/video_storage.proto
libs/proto/utils.proto
Scalar Value Types

u_query/proto/backend_api.proto¶

Host¶

Indexer, video storage

Field	Type	Label	Description
storage_ip_address	string	required	IP address
storage_port	uint32	required	Storage port Default: 54101
indexer_ip_address	string	required	IP address
indexer_port	uint32	required	Indexer port Default: 54100

HostHealth¶

Field	Type	Label	Description
indexer_health	com.ultinous.ServiceHealth	required	Indexer health
storage_health	com.ultinous.ServiceHealth	required	Video storage health

IndexedVideo¶

Field	Type	Label	Description
host_id	string	required	Host ID
video_info	com.ultinous.uquery.model.common.VideoInfo	required	Video info
indexing_info	IndexingStatusInfo	required	Index info

IndexedVideos¶

Field	Type	Label	Description
indexed_videos	IndexedVideos.IndexedVideosEntry	repeated	Indexed video mapping

IndexedVideos.IndexedVideosEntry¶

Field	Type	Label	Description
key	string	optional
value	IndexedVideo	optional

IndexingStatusInfo¶

Field	Type	Label	Description
params	com.ultinous.uquery.model.common.IndexingParams	optional	Indexing parameters
status	com.ultinous.uquery.model.common.IndexingStatus	required	Indexing status
start	com.ultinous.Timestamp	optional	Index start time
end	com.ultinous.Timestamp	optional	Index end time

NewAddRtspVideoRequest¶

Field	Type	Label	Description
video_metadata	com.ultinous.uquery.model.common.VideoMetadata	required
rtsp_metadata	com.ultinous.uquery.model.common.RTSPMetadata	required

PasswordChangeRequest¶

Field	Type	Label	Description
original_password	string	required
new_password	string	required

Query¶

Field	Type	Label	Description
type_id	string	required	Query type GUID
name	string	required	Query name
description	string	optional	Query description
param_values	com.ultinous.uquery.model.query.QueryParamValues	optional	Query parameters

QueryStatus¶

Field	Type	Label	Description
query_id	string	required	Query GUID
query	Query	required	Query details
state	com.ultinous.uquery.model.common.QueryExecutionState	optional	Execution state

QueryTypeDescriptor¶

Field	Type	Label	Description
name	string	required	Query type name
description	string	optional	Query type description
version	int32	required	Version
recommended	bool	optional	Recommended flag

QueryTypeParamsSpec¶

Field	Type	Label	Description
params	com.ultinous.uquery.model.query.Param	repeated	Query parameters

UserRequest¶

User

Field	Type	Label	Description
username	string	required
password	string	required

UserResponse¶

Field	Type	Label	Description
id	string	required
username	string	required

V2AddLocalVideoRequest¶

Field	Type	Label
path	string	required
start_time	com.ultinous.Timestamp	required
video_metadata	com.ultinous.uquery.model.common.VideoMetadata	required

VideoUploadMetadata¶

Field	Type	Label	Description
indexer_host_id	string	optional	Indexer host ID
start_time	com.ultinous.Timestamp	required	Start time
video_metadata	com.ultinous.uquery.model.common.VideoMetadata	required	Video metadata

Top

u_query/proto/common_model.proto¶

ClassifierDescriptor¶

Field	Type	Label	Description
name	string	required	Classifier name
description	string	optional	Description
applicable_object_types	string	repeated	Object types
classes	string	repeated	Classes
version	int32	required	Version
stability	ClassifierDescriptor.TYPE	required	Stability
min_object_height	uint32	optional	Min height Default: 80
applicable_to_full_crop	bool	optional	Full crop Default: true
applicable_to_upper_body	bool	optional	Upper body Default: false

Classifiers¶

Field	Type	Label	Description
types	Classifiers.TypesEntry	repeated	Classifiers by UUID

Classifiers.TypesEntry¶

Field	Type	Label	Description
key	string	optional
value	ClassifierDescriptor	optional

EncodedClassifier¶

Field	Type	Label	Description
bytes	bytes	required	Binary data

FileVideoConfig¶

Field	Type	Label	Description
file_path	string	required	File path
start_time	com.ultinous.Timestamp	required	Start time

GetFrameRequest¶

Field	Type	Label	Description
video_id	string	required	GUID
time	com.ultinous.Timestamp	required	Frame timestamp

GetVideoSegmentRequest¶

Field	Type	Label	Description
video_id	string	required	GUID
start_time	com.ultinous.Timestamp	required	Start time
duration_ms	uint32	optional	Duration Default: 10000
wait_for_future_packets	bool	optional	Wait for future packets Default: false

IndexingParams¶

Field	Type	Label	Description
fps	uint32	optional	FPS Default: 4
object_detection	bool	optional	Object detection Default: true
detector_speed_range_tradeoff	IndexingParams.DETECTOR_SPEED_RANGE_TRADEOFF	optional	Tradeoff Default: BALANCED
generic_attribute_analysis	bool	optional	Generic analysis Default: true
max_vector_calculations_per_frame	uint32	optional	Max vectors Default: 2
upper_body_attribute_analysis	bool	optional	Deprecated. Upper body analysis Default: true
max_upper_body_vector_calculations_per_frame	uint32	optional	Deprecated. Max upper body vectors Default: 2
face_analysis	bool	optional	Face analysis Default: true
max_face_vector_calculations_per_frame	uint32	optional	Max face vectors Default: 2
lp_analysis	bool	optional	License plate analysis Default: true
max_lp_per_frame	uint32	optional	Max license plates Default: 2
background_analysis	bool	optional	Background analysis Default: true
max_bg_vector_calculations_per_frame	uint32	optional	Max background vectors Default: 1

IndexingStatus¶

Field	Type	Label	Description
status	IndexingStatus.STATUS	optional	Status Default: NO_INDEX
message	string	optional	Message
progress	uint32	optional	Progress
start	com.ultinous.Timestamp	optional	Earliest timestamp in the index.
end	com.ultinous.Timestamp	optional

IndexingStatuses¶

Field	Type	Label	Description
indexes	IndexingStatuses.IndexesEntry	repeated	UUID -> VideoStatus

IndexingStatuses.IndexesEntry¶

Field	Type	Label	Description
key	string	optional
value	IndexingStatus	optional

Mp4Metadata¶

QueryExecutionState¶

Field	Type	Label	Description
state	QueryExecutionState.State	required	State
live	bool	optional	Live query Default: true
progress_in_percent	uint32	optional	Progress
started	com.ultinous.Timestamp	optional	Start timestamp
ended	com.ultinous.Timestamp	optional	End timestamp
message	string	optional	Message

QueryExecutionStates¶

Field	Type	Label	Description
execution_states	QueryExecutionStates.ExecutionStatesEntry	repeated	Query states by UUID

QueryExecutionStates.ExecutionStatesEntry¶

Field	Type	Label	Description
key	string	optional
value	QueryExecutionState	optional

RTSPInfo¶

Information about an RTSP stream.

Field	Type	Label	Description
frame	com.ultinous.JPEGImage	required	Snapshot frame
fps	float	optional	Frame rate of the stream.
codec	string	optional	Codec of the streams (eg.: h264)
encoding	VideoEncodingInfo	optional	Encoding parameters of the stream

RTSPMetadata¶

Field	Type	Label	Description
rtsp_url	string	required	RTSP URL
retention_in_seconds	uint32	optional	Retention time Default: 86400

RTSPTestResult¶

Result of a testRTSP call.

Field	Type	Label	Description
error_message	string	optional	Human readable message if connection failed.
rtsp_info	RTSPInfo	optional	Rtsp info if connection succeeded.

RTSPVideoConfig¶

Field	Type	Label	Description
rtsp_url	string	required	RTSP URL
retention_in_seconds	uint32	optional	Retention time Default: 86400

RawMetadata¶

Field	Type	Label	Description
utc_ts_ms	int64	optional	UTC timestamp
pts_us	uint64	required	Presentation timestamp
dts_us	uint64	optional	Decoding timestamp

Resources¶

Indexer node compute resources. Unit of resources are not specified it is implementation dependent however resources can be compared to each other For example load percentage can be calculated using the total and available resources.

Field	Type	Label	Description
decoding	float	required	Video decoding resources.
inference	float	required	Deep learning inference resources.

StreamVideoRequest¶

Field	Type	Label	Description
video_id	string	required	GUID
format	StreamVideoRequest.Format	required	Format
start_time	com.ultinous.Timestamp	required	Start time
end_time	com.ultinous.Timestamp	optional	End time

StreamVideoResponse¶

Field	Type	Label	Description
data	bytes	required	Video data
raw_metadata	RawMetadata	optional	Raw metadata
mp4_metadata	Mp4Metadata	optional	MP4 metadata

UpdateVideoRequest¶

Field	Type	Label	Description
video_id	string	required	GUID
video_metadata	VideoMetadata	required	Video metadata

VideoEncodingInfo¶

Field	Type	Label	Description
fps	float	required	Frame rate of the stream.
codec	string	required	Codec of the streams (eg.: h264)
size	com.ultinous.Size2D	required	Size of the video frame

VideoInfo¶

Field	Type	Label	Description
video_metadata	VideoMetadata	required	Metadata
rtsp_metadata	RTSPMetadata	optional	RTSP metadata
is_live	bool	required	Live status
status	VideoStatus	optional	Deprecated status
start	com.ultinous.Timestamp	optional	Deprecated start
end	com.ultinous.Timestamp	optional	Deprecated end

VideoInfos¶

Field	Type	Label	Description
videos	VideoInfos.VideosEntry	repeated	UUID -> VideoInfo

VideoInfos.VideosEntry¶

Field	Type	Label	Description
key	string	optional
value	VideoInfo	optional

VideoMetadata¶

Field	Type	Label	Description
name	string	required	Name of the video source
technical_name	string	optional	Placeholder for external id
description	string	optional	Human readable description

VideoStatus¶

Field	Type	Label	Description
status	VideoStatus.STATUS	required	Status
message	string	optional	Status message
progress	uint32	optional	Progress
start	com.ultinous.Timestamp	optional	Start time
end	com.ultinous.Timestamp	optional	End time

VideoStatuses¶

Field	Type	Label	Description
videos	VideoStatuses.VideosEntry	repeated	UUID -> VideoStatus

VideoStatuses.VideosEntry¶

Field	Type	Label	Description
key	string	optional
value	VideoStatus	optional

ClassifierDescriptor.TYPE¶

Name	Number	Description
STABLE	1	Stable class
UNSTABLE	2	Unstable class

IndexingParams.DETECTOR_SPEED_RANGE_TRADEOFF¶

Name	Number	Description
HIGH_SPEED	1	High speed
BALANCED	2	Balanced
HIGH_RANGE	3	High range

IndexingStatus.STATUS¶

Name	Number	Description
NO_INDEX	1	No index
INITIALIZING	2	Initializing
INDEXING	3	Indexing
FINISHED	4	Finished
ERROR	7	Error
STOPPED	8	Error
PAUSED	9	Paused

QueryExecutionState.State¶

Name	Number	Description
NOT_EXECUTED	6	Initializing
INITIALIZING	1	Initializing
RUNNING	2	Running
DONE	3	Done
STOPPED	4	Stopped
ERROR	5	Error

StreamVideoRequest.Format¶

Name	Number	Description
RAW	0	Raw format
MP4	1	MP4 format

VideoStatus.STATUS¶

Name	Number	Description
CONNECTING	1	Connecting
RECONNECTING	2	Reconnecting
RECORDING	3	Recording
FINISHED	4	Finished
STOPPED	8	Finished
ERROR	7	Error
PAUSED	9	Paused

Top

u_query/proto/index.proto¶

Detection¶

Field	Type	Label	Description
video_id	string	optional	GUID of the video. This is optional because it is not part of the serialized index. It is filled by the reader when the detection is read from the index.
frame_time	int64	required	milliseconds since the epoch (strictly monotonous increasing)
track_id	int64	required	track id is unique within a stream
frame_width	int32	required	Frame info
frame_height	int32	required
obj_type	string	required	Generic detection info

DetectionCluster¶

Field	Type	Label	Description
fv	com.ultinous.FeatureVector	optional
detections	Detection	repeated

DetectionMeta¶

Field	Type	Label
video_id	string	required
frame_time	int64	required
track_id	int64	required

DetectionPair¶

Output structures TODO: move this to its proper place

Field	Type	Label	Description
left	Detection	required
right	Detection	required

FaceAttributes¶

Field	Type	Label
key_points	FaceKeyPoints	optional
confidence	float	optional
face_rec_features	com.ultinous.FeatureVector	optional
face_features	com.ultinous.FeatureVector	optional

FaceKeyPoints¶

Field	Type	Label
right_eye	com.ultinous.Point	optional
left_eye	com.ultinous.Point	optional
nose	com.ultinous.Point	optional
mouth_right	com.ultinous.Point	optional
mouth_left	com.ultinous.Point	optional

FrameInfo¶

Field	Type	Label	Description
frame_width	int32	required
frame_height	int32	required

IndexRecord¶

for every analyzed frame there is a FrameInfo after the detections ts: frame_ts_ms key: track_id or empty for frame info

Field	Type	Label	Description
detection	NewDetection	optional
frame_info	FrameInfo	optional

IntKey¶

Field	Type	Label	Description
id	int64	required

LicensePlateAttributes¶

Field	Type	Label	Description
confidence	float	optional	TODO: lp keypoints
licence_plate	string	optional

NewDetection¶

Field	Type	Label	Description
obj_type	string	required	object type (eg.: COCO class name) extended with "bg" as background
box	Rect	required
confidence	float	required
features	com.ultinous.FeatureVector	optional	generic features (eg.: CLIP features)
person_attributes	PersonAttributes	optional
vehicle_attributes	VehicleAttributes	optional
meta	DetectionMeta	optional	beam needs this

PersonAttributes¶

Field	Type	Label	Description
face_attributes	FaceAttributes	optional
upper_body_features	com.ultinous.FeatureVector	optional

Rect¶

Field	Type	Label
x	int32	required
y	int32	required
width	int32	required
height	int32	required

VehicleAttributes¶

Field	Type	Label	Description
license_plate	LicensePlateAttributes	optional

Top

u_query/proto/indexer.proto¶

Indexing¶

Field	Type	Label	Description
video_id	string	required
cfg	IndexingConfig	optional	not set if indexing is stopped

IndexingConfig¶

Field	Type	Label	Description
params	com.ultinous.uquery.model.common.IndexingParams	required
bootstrap_servers	string	required	output kafka
start	com.ultinous.TimestampOrSpecial	optional	starts indexing from max(timestamp, highest ts from index + 1)
end	com.ultinous.Timestamp	optional

IndexingInfo¶

Field	Type	Label	Description
indexing	Indexing	required
status	com.ultinous.uquery.model.common.IndexingStatus	required	Status info.
start	com.ultinous.Timestamp	optional	Deprecated. Earliest timestamp in the index.
end	com.ultinous.Timestamp	optional	Deprecated.

IndexingInfos¶

Field	Type	Label	Description
infos	IndexingInfo	repeated

Indexings¶

Field	Type	Label	Description
indexings	Indexing	repeated

IndexerService¶

A video indexer. Input video comes from a kafka, output index goes to kafka (can be a different kafka instance) Main functions: - Index live or historic videos

Stored state: - Indexing config

Method Name	Request Type	Response Type	Description
getHealthStatus	.google.protobuf.Empty	.com.ultinous.ServiceHealth	Service health check
getTotalResources	.google.protobuf.Empty	.com.ultinous.uquery.model.common.Resources	Gets the total compute resources of this indexer hardware.
getAvailableResources	.google.protobuf.Empty	.com.ultinous.uquery.model.common.Resources	Gets the available compute resources of this indexer hardware.
getResourceUse	Indexing	.com.ultinous.uquery.model.common.Resources	Calculate resource usage for a given indexing config. Call this before setIndex to make sure there are enough resources.
getIndexingResourceUse	.com.ultinous.uquery.model.common.IndexingParams	.google.protobuf.FloatValue
getDecodingResourceUse	.com.ultinous.uquery.model.common.VideoEncodingInfo	.google.protobuf.FloatValue
init	Indexings	.google.protobuf.Empty
set	Indexing	.google.protobuf.Empty	Sets indexing for a given video. This starts a resource (video decoding, dnn inference), intensive process. Call getIndexingResourceUse to calculate the necessary resources.
list	.google.protobuf.Empty	IndexingInfos	Get all video config and status
getIndexingStatus	.google.protobuf.StringValue	.com.ultinous.uquery.model.common.IndexingStatus
listIndexingStatuses	.google.protobuf.Empty	.com.ultinous.uquery.model.common.IndexingStatuses
calcCLIPFeatureVectorFromImage	.com.ultinous.JPEGImage	.com.ultinous.FeatureVector	Feature vec calculation for testing purpose (eg.: compare feature vector of different implementations)
getObjectTypes	.google.protobuf.Empty	.com.ultinous.Strings	Returns all object types (eg.: person, car, dog, ...)
getHardwareIds	.google.protobuf.Empty	.com.ultinous.Strings	Returns all hardware ids (eg.: GPU)

Top

u_query/proto/video_storage.proto¶

AddLocalVideoRequest¶

Field	Type	Label
id	string	required
path	string	required
start_time	com.ultinous.Timestamp	required

AddRTSPVideoRequest¶

Field	Type	Label	Description
id	string	required
rtsp_metadata	com.ultinous.uquery.model.common.RTSPMetadata	required

LocalVideoMetadata¶

Field	Type	Label	Description
path	string	required
start_time	com.ultinous.Timestamp	required

UploadAndAddVideoRequest¶

Field	Type	Label	Description
metadata	VideoUploadMetadata	optional
chunk	bytes	optional

VideoConfig¶

Field	Type	Label
video_id	string	required
rtsp	com.ultinous.uquery.model.common.RTSPMetadata	optional
local	LocalVideoMetadata	optional
uploaded	google.protobuf.Empty	optional
stopped	bool	optional

VideoConfigs¶

Field	Type	Label	Description
configs	VideoConfig	repeated

VideoUploadMetadata¶

Field	Type	Label	Description
id	string	required
start_time	com.ultinous.Timestamp	required

VideoStorage¶

A video storage that can store both live and historic videos. Main functions: - Add video data from live (RTSP) or historic source (video files). - Access video data in various forms (jpeg image, short video sequence, direct packet level access from kafka) - Access config and status information

Stored state: - Encoded video data in kafka - VideoConfig for each video

Method Name	Request Type	Response Type	Description
getHealthStatus	.google.protobuf.Empty	.com.ultinous.ServiceHealth	Service health check.
testRTSP	.google.protobuf.StringValue	.com.ultinous.uquery.model.common.RTSPTestResult
init	VideoConfigs	.google.protobuf.Empty
addRTSPVideo	AddRTSPVideoRequest	.google.protobuf.Empty
addLocalVideo	AddLocalVideoRequest	.google.protobuf.Empty
stopVideo	.google.protobuf.StringValue	.google.protobuf.Empty
resumeVideo	.google.protobuf.StringValue	.google.protobuf.Empty
deleteVideo	.google.protobuf.StringValue	.google.protobuf.Empty	Deletes a video (with all config and recorded video data)
uploadAndAddVideo	UploadAndAddVideoRequest stream	.google.protobuf.Empty
getVideoInfo	.google.protobuf.StringValue	.com.ultinous.uquery.model.common.VideoInfo
listVideoInfos	.google.protobuf.Empty	.com.ultinous.uquery.model.common.VideoInfos
getVideoStatus	.google.protobuf.StringValue	.com.ultinous.uquery.model.common.VideoStatus
listVideoStatuses	.google.protobuf.Empty	.com.ultinous.uquery.model.common.VideoStatuses
getFrame	.com.ultinous.uquery.model.common.GetFrameRequest	.com.ultinous.JPEGImageWithTimestamp	Gets the frame for a given timestamp. If there is not frame for the exact timestamp it will return the closest.
getVideoSegment	.com.ultinous.uquery.model.common.GetVideoSegmentRequest	.google.protobuf.BytesValue
streamVideo	.com.ultinous.uquery.model.common.StreamVideoRequest	.com.ultinous.uquery.model.common.StreamVideoResponse stream

Top

libs/proto/utils.proto¶

FeatureVector¶

Field	Type	Label	Description
vals	float	repeated

JPEGImage¶

Field	Type	Label	Description
content	bytes	required

JPEGImageWithTimestamp¶

Field	Type	Label	Description
image	JPEGImage	required
timestamp	Timestamp	required

KafkaTopic¶

Field	Type	Label	Description
broker	string	required	eg.: "localhost:9092"
topic	string	required

Point¶

Field	Type	Label	Description
x	int32	required
y	int32	required

Polygon2D¶

Field	Type	Label	Description
points	Point	repeated

ServiceHealth¶

Generic service health response.

Field	Type	Label	Description
message	string	optional	Human readable message describing the current state.
status	ServiceHealth.STATUS	required	Current status.

Size2D¶

Field	Type	Label	Description
w	uint32	required
h	uint32	required

Strings¶

Field	Type	Label	Description
strs	string	repeated

Timestamp¶

Field	Type	Label	Description
millis	int64	required	Millisecond resolution UTC time.

TimestampOrSpecial¶

Field	Type	Label	Description
timestamp_ms	int64	optional	Millisecond resolution UTC time.
special	TimestampOrSpecial.CONSTANT	optional	Special value.

ServiceHealth.STATUS¶

Name	Number	Description
STARTING	1	The service is staring, calling endpoints can cause error as the service is not initialized yet.
HEALTHY	2	Everything works normally.
WARNING	3	Everything works normally but there are warnings requiring actions, (eg.: low on resource)
UNHEALTHY	4	The service does not work normally.
UNREACHABLE	5	The service is unreachable.

TimestampOrSpecial.CONSTANT¶

Name	Number	Description
NOW	0	Special value for current time.

File-level Extensions¶

Extension	Type	Base	Number
enum_description	string	.google.protobuf.EnumOptions	50100
enum_value_description	string	.google.protobuf.EnumValueOptions	50100
field_description	string	.google.protobuf.FieldOptions	50100
description	string	.google.protobuf.MessageOptions	50100

Scalar Value Types¶

.proto Type	Notes	C++	Java	Python	Go	C#	PHP	Ruby
double		double	double	float	float64	double	float	Float
float		float	float	float	float32	float	float	Float
int32	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
int64	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.	int64	long	int/long	int64	long	integer/string	Bignum
uint32	Uses variable-length encoding.	uint32	int	int/long	uint32	uint	integer	Bignum or Fixnum (as required)
uint64	Uses variable-length encoding.	uint64	long	int/long	uint64	ulong	integer/string	Bignum or Fixnum (as required)
sint32	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sint64	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.	int64	long	int/long	int64	long	integer/string	Bignum
fixed32	Always four bytes. More efficient than uint32 if values are often greater than 2^28.	uint32	int	int	uint32	uint	integer	Bignum or Fixnum (as required)
fixed64	Always eight bytes. More efficient than uint64 if values are often greater than 2^56.	uint64	long	int/long	uint64	ulong	integer/string	Bignum
sfixed32	Always four bytes.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sfixed64	Always eight bytes.	int64	long	int/long	int64	long	integer/string	Bignum
bool		bool	boolean	boolean	bool	bool	boolean	TrueClass/FalseClass
string	A string must always contain UTF-8 encoded or 7-bit ASCII text.	string	String	str/unicode	string	string	string	String (UTF-8)
bytes	May contain any arbitrary sequence of bytes.	string	ByteString	str	[]byte	ByteString	string	String (ASCII-8BIT)