Protocol Documentation
Table of Contents
Top
u_query/proto/backend_api.proto
AddLocalVideoRequest
AddRTSPVideoRequest
ClassifierId
Field |
Type |
Label |
Description |
classifier_id |
string |
required |
|
EncodedQueryType
Encoded query type. Query type is defines a use case (Synonyms in other systems: solution, rule, etc).
Some examples: intrusion detection, tailgating, near miss detection, dynamic search
Query types cannot be executed directly, a query has to be created from the query type where all parameters are filled.
Field |
Type |
Label |
Description |
bytes |
bytes |
required |
Encoded binary query type data |
ExecutionId
Field |
Type |
Label |
Description |
execution_id |
string |
required |
|
IndexedVideo
Video and index info.
IndexedVideos
IndexedVideos.IndexedVideosEntry
IndexerHost
Indexer host connection info
Field |
Type |
Label |
Description |
ip_address |
string |
required |
IP address of the indexer host. |
indexer_port |
uint32 |
optional |
Port of the indexer service. Default: 54100 |
video_storage_port |
uint32 |
optional |
Port of the video storage service. Default: 54101 |
IndexerHostHealth
IndexerHostId
Field |
Type |
Label |
Description |
indexer_host_id |
string |
required |
|
IndexerHosts
Indexer host connection info
IndexerHosts.IndexersEntry
IndexingRequest
IndexingResourceUseRequest
IndexingStatusInfo
Queries
All executable queries.
Queries.QueriesEntry
Field |
Type |
Label |
Description |
key |
string |
optional |
|
value |
Query |
optional |
|
Query
Represent an executable query: query type + filled parameters.
QueryId
Field |
Type |
Label |
Description |
query_id |
string |
required |
|
QueryStatus
QueryStatuses
Field |
Type |
Label |
Description |
queries |
QueryStatus |
repeated |
|
QueryTypeDescriptor
Provides information about a query type to display and edit its parameters.
As new query types can be added to the system its important to provide a unified way of presenting parameter for the user.
Field |
Type |
Label |
Description |
name |
string |
required |
Name of the query type (use case). Can be used as display name. It must be unique. Eg.: "Intrusion detection" |
description |
string |
optional |
Human readable description. Eg.: "Send alert if a person is detected in the given area." |
version |
int32 |
required |
Version number of the query type. Uploading a new version does not overwrite previous versions. |
recommended |
bool |
optional |
The latest version is recommended, but older versions are also available. When registering, the value of the field is irrelevant. |
QueryTypeId
Field |
Type |
Label |
Description |
query_type_id |
string |
required |
|
QueryTypeParamsSpec
This structure contains the parameters of the query type along with a UI model.
The application can use this model to present any query type to the user.
QueryTypes
Description info for all registered query types.
Field |
Type |
Label |
Description |
types |
QueryTypes.TypesEntry |
repeated |
All registered query type information by GUID. |
QueryTypes.TypesEntry
RenameQueryExecutionRequest
Field |
Type |
Label |
Description |
query_id |
string |
required |
|
name |
string |
required |
|
UpdateQueryRequest
Update and existing query.
Field |
Type |
Label |
Description |
query_id |
string |
required |
Query GUID. |
query |
Query |
required |
The new query. |
UploadAndAddVideoRequest
VideoFilter
Field |
Type |
Label |
Description |
host_id |
string |
optional |
|
indexed |
bool |
optional |
|
VideoId
Field |
Type |
Label |
Description |
video_id |
string |
required |
|
ClassifierAPI
API for managing classifiers
IndexerAPI
API for indexer host handling: get health, add/remove/list indexers, get indexer resource usage.
QueryAPI
API for managing query types and queries.
StatusAPI
Health
VideoAPI
API for video loading, indexing, and video access
Top
u_query/proto/common_model.proto
ClassifierDescriptor
Provides information about a classifier.
Field |
Type |
Label |
Description |
name |
string |
required |
Name of the classifier. Can be used as display name. It must be unique. Eg.: "Helmet" |
description |
string |
optional |
Human readable description. Eg.: "Tells if a person is wearing a helmet or not." |
applicable_object_types |
string |
repeated |
List the name of all applicable object types (eg.: COCO object types). |
classes |
string |
repeated |
List all classes. Eg.: ["helmet","no_helmet"] |
version |
int32 |
required |
Version number of the Classifier. Uploading a new version will overwrite the current version. The new version can only expand the ‘applicable_object_types’ and ‘classes’ |
stability |
ClassifierDescriptor.TYPE |
required |
Stability of the class. See TYPE for details. |
min_object_height |
uint32 |
optional |
Minimum object height in pixels to apply the classifier. Default: 80 |
applicable_to_full_crop |
bool |
optional |
Default: true |
applicable_to_upper_body |
bool |
optional |
Default: false |
Classifiers
Description info for all registered classifiers.
Classifiers.TypesEntry
EncodedClassifier
Field |
Type |
Label |
Description |
bytes |
bytes |
required |
Encoded binary classifier data |
FileVideoConfig
GetFrameRequest
Get frame by timestamp.
Field |
Type |
Label |
Description |
video_id |
string |
required |
GUID of the video. |
time |
com.ultinous.Timestamp |
required |
Timestamp of the requested frame. |
GetVideoSegmentRequest
IndexingParams
Indexing parameters. Default values are suitable for most of the use cases.
Performance and indexing feature trade off can be controlled with these parameters.
Cost optimization can be achieved by fine tuning these parameters to the required set of use cases.
Field |
Type |
Label |
Description |
fps |
uint32 |
optional |
Detector FPS. Typically 1-6. If movement of objects are not required by the use case (eg.: counting parking cars) FPS can be set to 1. For moving objects higher FPS is recommended. For persons and slow moving cars 4 FPS is good. For fast moving cars the FPS should be increased to 8. The hardware consumption is directly proportional with the FPS. Use the lowest possible FPS to spare hardware resources. Default: 4 |
object_detection |
bool |
optional |
Turn on/off object detection. For most of the use cases object detection is needed. If only background analysis is needed it can be set to false. Default: true |
detector_speed_range_tradeoff |
IndexingParams.DETECTOR_SPEED_RANGE_TRADEOFF |
optional |
Detector speed/range tradeoff (neglected if the detector does not support it) Default: BALANCED |
generic_attribute_analysis |
bool |
optional |
Generic attribute analysis for objects (feature vectors). This is required for free text search and classification (eg.: gender, smoking, on the phone) on objects. Default: true |
max_vector_calculations_per_frame |
uint32 |
optional |
Maximum number of generic feature vectors to calculate per frame (if generic_attribute_analysis is true). Default: 2 |
upper_body_attribute_analysis |
bool |
optional |
If set, features for upper budy will be calculated. This is required by upper body based classification (eg.:gender, smoking, on the phone, etc). Default: true |
max_upper_body_vector_calculations_per_frame |
uint32 |
optional |
Maximum number of upper body feature vectors to calculate per frame (if upper_body_attribute_analysis is true). Default: 2 |
face_analysis |
bool |
optional |
Turn on/off face analysis. If set face detection and face feature vector extraction will be used. It has to be set true for face recognition based use cases. Default: true |
max_face_vector_calculations_per_frame |
uint32 |
optional |
Maximum number of face vectors to calculate per frame (if face_analysis is true). Default: 2 |
lp_analysis |
bool |
optional |
Turn on/off licence plate analysis (licence plate detection and recognition) Default: true |
max_lp_per_frame |
uint32 |
optional |
Maximum number of licence plate recognitions per frame (if lp_analysis is true). Default: 2 |
background_analysis |
bool |
optional |
Turn on/off background analysis. Background analysis scans the full frame regardless object detections. It can be used to detect changes in the background (eg.: gate opened/closed or door blocked) Default: true |
max_bg_vector_calculations_per_frame |
uint32 |
optional |
Maximum number of background feature vectors per per frame (if background_analysis is true). Default: 1 |
IndexingStatus
Field |
Type |
Label |
Description |
status |
IndexingStatus.STATUS |
optional |
The indexing status. Default: NO_INDEX |
message |
string |
optional |
Human readable message describing the current state. |
progress |
uint32 |
optional |
For historic sources: progress percent for indexing. |
QueryExecutionState
Field |
Type |
Label |
Description |
state |
QueryExecutionState.State |
required |
State of the query execution. |
live |
bool |
optional |
true: live; false: ad-hoc Default: true |
progress_in_percent |
uint32 |
optional |
Progress information for ad-hoc queries if available. |
started |
com.ultinous.Timestamp |
optional |
Start timestamp of processing |
ended |
com.ultinous.Timestamp |
optional |
Timestamp of completion of processing |
message |
string |
optional |
Human readable message describing details of the the current state (eg.: error details). |
QueryExecutionStates
QueryExecutionStates.ExecutionStatesEntry
Field |
Type |
Label |
Description |
rtsp_url |
string |
required |
|
retention_in_seconds |
uint32 |
optional |
1 day Default: 86400 |
RTSPVideoConfig
Field |
Type |
Label |
Description |
rtsp_url |
string |
required |
|
retention_in_seconds |
uint32 |
optional |
1 day Default: 86400 |
Field |
Type |
Label |
Description |
utc_ts_ms |
int64 |
required |
UTC timestamp of the frame. |
pts_us |
uint64 |
required |
Presentation timestamp of the frame. |
dts_us |
uint64 |
optional |
Decoding timestamp of the frame. |
Resources
Indexer node compute resources.
Unit of resources are not specified it is implementation dependent however resources can be compared to each other
For example load percentage can be calculated using the total and available resources.
Field |
Type |
Label |
Description |
decoding |
float |
required |
Video decoding resources. |
inference |
float |
required |
Deep learning inference resources. |
StreamVideoRequest
StreamVideoResponse
UpdateVideoRequest
VideoInfo
VideoInfos
VideoInfos.VideosEntry
Field |
Type |
Label |
Description |
name |
string |
required |
Name of the video source (does not have to be unique). |
technical_name |
string |
optional |
Placeholder for an external id to connect with other systems (eg.: VMS) |
description |
string |
optional |
Human readable description of the source. |
VideoStatus
Field |
Type |
Label |
Description |
status |
VideoStatus.STATUS |
required |
Status of the video. |
message |
string |
optional |
Human readable message describing details of the the current state (eg.: error details). |
progress |
uint32 |
optional |
Progress percent for file loading state. |
ClassifierDescriptor.TYPE
Name |
Number |
Description |
STABLE |
1 |
The class is stable over a long time period (at least for minutes). Eg.: gender of a person, color of car Eg.: gender of a person, color of car |
UNSTABLE |
2 |
The class can change quickly (within a seconds) Eg.: a person take off the helmet, fall, or pull out a gun |
IndexingParams.DETECTOR_SPEED_RANGE_TRADEOFF
Name |
Number |
Description |
HIGH_SPEED |
1 |
Highest speed, lowest range. |
BALANCED |
2 |
Balanced speed and range. |
HIGH_RANGE |
3 |
Highest range, lowest speed. |
IndexingStatus.STATUS
Name |
Number |
Description |
NO_INDEX |
1 |
No index is available. |
INITIALIZING |
2 |
Indexing has been started, initializing and waiting for data. |
INDEXING |
3 |
Indexing is being performed. |
FINISHED |
4 |
Indexing finished for historic source. |
ERROR |
7 |
Indexing is in error state, see message for details. |
QueryExecutionState.State
Name |
Number |
Description |
INITIALIZING |
1 |
Query is initializing normally. |
RUNNING |
2 |
Query is running normally. |
DONE |
3 |
Done is valid only for ad-hoc queries (live queries runs until they stopped). |
STOPPED |
4 |
Execution is stopped by the user. |
ERROR |
5 |
Query execution has failed with an error. |
Name |
Number |
Description |
RAW |
0 |
Raw video byte stream. |
MP4 |
1 |
MP4 video stream. |
VideoStatus.STATUS
Name |
Number |
Description |
CONNECTING |
1 |
The video is created but no (live or historic) data has been loaded yet. |
RECONNECTING |
2 |
Reconnecting to RTSP source. |
RECORDING |
3 |
Video data is being loaded. |
FINISHED |
4 |
Recording finished for historic source. |
ERROR |
7 |
Video is in error state, see message for details. |
Top
u_query/proto/index.proto
Detection
Field |
Type |
Label |
Description |
video_id |
string |
optional |
GUID of the video. This is optional because it is not part of the serialized index. It is filled by the reader when the detection is read from the index. |
frame_time |
int64 |
required |
milliseconds since the epoch (strictly monotonous increasing) |
track_id |
int64 |
required |
track id is unique within a stream |
frame_width |
int32 |
required |
Frame info |
frame_height |
int32 |
required |
|
obj_type |
string |
required |
Generic detection info |
object type (eg.: COCO class name) extended with "bg" as background |
| box | Rect | required | |
| confidence | float | required | |
| features | com.ultinous.FeatureVector | optional | generic features (eg.: CLIP features) |
| person_attributes | PersonAttributes | optional | Object type specific attributes. Some objects (like person, vehicles) have more detailed attributes. |
| vehicle_attributes | VehicleAttributes | optional | |
DetectionCluster
DetectionPair
Output structures
TODO: move this to its proper place
FaceAttributes
FaceKeyPoints
IntKey
Field |
Type |
Label |
Description |
id |
int64 |
required |
|
LicensePlateAttributes
Field |
Type |
Label |
Description |
confidence |
float |
optional |
TODO: lp keypoints |
licence_plate |
string |
optional |
|
PersonAttributes
Rect
Field |
Type |
Label |
Description |
x |
int32 |
required |
|
y |
int32 |
required |
|
width |
int32 |
required |
|
height |
int32 |
required |
|
VehicleAttributes
Top
u_query/proto/indexer.proto
Indexing
Field |
Type |
Label |
Description |
video_id |
string |
required |
|
cfg |
IndexingConfig |
optional |
not set if indexing is disabled |
IndexingConfig
IndexingInfo
IndexingInfos
IndexerService
A video indexer. Input video comes from a kafka, output index goes to kafka (can be a different kafka instance)
Main functions:
- Index live or historic videos
Stored state:
- Indexing config
Top
u_query/proto/video_storage.proto
AddLocalVideoRequest
AddRTSPVideoRequest
UploadAndAddVideoRequest
VideoStorage
A video storage that can store both live and historic videos.
Main functions:
- Add video data from live (RTSP) or historic source (video files).
- Access video data in various forms (jpeg image, short video sequence, direct packet level access from kafka)
- Access config and status information
Stored state:
- Encoded video data in kafka
- VideoConfig for each video
Top
libs/proto/utils.proto
FeatureVector
Field |
Type |
Label |
Description |
vals |
float |
repeated |
|
JPEGImage
Field |
Type |
Label |
Description |
content |
bytes |
required |
|
JPEGImageWithTimestamp
KafkaTopic
Field |
Type |
Label |
Description |
broker |
string |
required |
eg.: "localhost:9092" |
topic |
string |
required |
|
Point
Field |
Type |
Label |
Description |
x |
int32 |
required |
|
y |
int32 |
required |
|
Polygon2D
Field |
Type |
Label |
Description |
points |
Point |
repeated |
|
ServiceHealth
Generic service health response.
Field |
Type |
Label |
Description |
message |
string |
optional |
Human readable message describing the current state. |
status |
ServiceHealth.STATUS |
required |
Current status. |
Strings
Field |
Type |
Label |
Description |
strs |
string |
repeated |
|
Timestamp
Field |
Type |
Label |
Description |
millis |
int64 |
required |
Millisecond resolution UTC time. |
ServiceHealth.STATUS
Name |
Number |
Description |
STARTING |
1 |
The service is staring, calling endpoints can cause error as the service is not initialized yet. |
HEALTHY |
2 |
Everything works normally. |
WARNING |
3 |
Everything works normally but there are warnings requiring actions, (eg.: low on resource) |
UNHEALTHY |
4 |
The service does not work normally. |
UNREACHABLE |
5 |
The service is unreachable. |
Scalar Value Types
.proto Type |
Notes |
C++ |
Java |
Python |
Go |
C# |
PHP |
Ruby |
double |
|
double |
double |
float |
float64 |
double |
float |
Float |
float |
|
float |
float |
float |
float32 |
float |
float |
Float |
int32 |
Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. |
int32 |
int |
int |
int32 |
int |
integer |
Bignum or Fixnum (as required) |
int64 |
Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. |
int64 |
long |
int/long |
int64 |
long |
integer/string |
Bignum |
uint32 |
Uses variable-length encoding. |
uint32 |
int |
int/long |
uint32 |
uint |
integer |
Bignum or Fixnum (as required) |
uint64 |
Uses variable-length encoding. |
uint64 |
long |
int/long |
uint64 |
ulong |
integer/string |
Bignum or Fixnum (as required) |
sint32 |
Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. |
int32 |
int |
int |
int32 |
int |
integer |
Bignum or Fixnum (as required) |
sint64 |
Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. |
int64 |
long |
int/long |
int64 |
long |
integer/string |
Bignum |
fixed32 |
Always four bytes. More efficient than uint32 if values are often greater than 2^28. |
uint32 |
int |
int |
uint32 |
uint |
integer |
Bignum or Fixnum (as required) |
fixed64 |
Always eight bytes. More efficient than uint64 if values are often greater than 2^56. |
uint64 |
long |
int/long |
uint64 |
ulong |
integer/string |
Bignum |
sfixed32 |
Always four bytes. |
int32 |
int |
int |
int32 |
int |
integer |
Bignum or Fixnum (as required) |
sfixed64 |
Always eight bytes. |
int64 |
long |
int/long |
int64 |
long |
integer/string |
Bignum |
bool |
|
bool |
boolean |
boolean |
bool |
bool |
boolean |
TrueClass/FalseClass |
string |
A string must always contain UTF-8 encoded or 7-bit ASCII text. |
string |
String |
str/unicode |
string |
string |
string |
String (UTF-8) |
bytes |
May contain any arbitrary sequence of bytes. |
string |
ByteString |
str |
[]byte |
ByteString |
string |
String (ASCII-8BIT) |