autopilot/misc/rtsp_ai_player/README.md

rtsp_ai_player

rtsp_ai_player is an application that listens to an RTSP stream, analyzes images with an AI model, and shows the results visually. It also controls a gimbal camera to zoom in on the recognized objects. Application uses YOLOv8 AI models converted to the ONNX format.

How to convert Azaion AI model file to ONNX format

yolo export model=azaion-2024-08-13.pt dynamic=False format=onnx imgsz=640,640

How to use application locally on a Linux PC.

Speed up compilations

echo "export MAKEFLAGS=\"-j8\"" >> ~/.bashrc
echo "export PATH=/usr/lib/ccache:\$PATH" >> ~/.bashrc

Install OpenCV 4.10.0

sudo apt update
sudo apt install libgtk-3-dev libpng-dev cmake ffmpeg libavcodec-dev libavformat-dev libavfilter-dev
wget https://github.com/opencv/opencv/archive/refs/tags/4.10.0.zip
unzip 4.10.0.zip
cd opencv-4.10.0
mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/opt/opencv-4.10.0 -DBUILD_opencv_world=ON -DOPENCV_GENERATE_PKGCONFIG=ON -DBUILD_PERF_TESTS=OFF -DBUILD_TESTS=OFF ..
make -j8 && sudo make install

Install ONNX runtime 1.18.0

wget https://github.com/microsoft/onnxruntime/releases/download/v1.18.0/onnxruntime-linux-x64-1.18.0.tgz
sudo tar xf onnxruntime-linux-x64-1.18.0.tgz -C /opt

Install ffmpeg and mediamtx RTSP server:

sudo apt update
sudo apt install ffmpeg

# For Amd64-like platforms:
wget https://github.com/bluenviron/mediamtx/releases/download/v1.8.4/mediamtx_v1.8.4_linux_amd64.tar.gz
mkdir mediamtx
tar xf mediamtx_v1.8.4_linux_amd64.tar.gz -C mediamtx

# For OrangePi5 use this lib instead (since it's an ARM platform):
wget https://github.com/bluenviron/mediamtx/releases/download/v1.9.1/mediamtx_v1.9.1_linux_arm64v8.tar.gz
mkdir mediamtx
tar xf mediamtx_v1.9.1_linux_arm64v8.tar.gz -C mediamtx

If you use video file from the local RTSP server:

cd mediamtx
./mediamtx

Play Azaion mp4 video file from RTSP server ... :

ffmpeg -re -stream_loop -1 -i $HOME/azaion/videos/for_ai_cut.mp4 -c copy -f rtsp rtsp://localhost:8554/live.stream

... or play simple video file from RTSP server:

ffmpeg -re -stream_loop -1 -i $HOME/azaion/videos/table.mp4 -c copy -f rtsp rtsp://localhost:8554/live.stream

Test RTSP streaming with ffplay:

ffplay -rtsp_transport tcp rtsp://localhost:8554/live.stream

Compile and run rtsp_ai_player with YOLOv8 medium AI model:

cd autopilot/misc/rtsp_ai_player
qmake6 && make
./rtsp_ai_player ~/azaion/models/onnx/yolov8m.onnx

Compile and run rtsp_ai_player with Azaion AI model:

cd autopilot/misc/rtsp_ai_player
qmake6 && make
./rtsp_ai_player ~/azaion/models/azaion/azaion-2024-08-13.onnx

Compile and run rtsp_ai_player with YOLOv8 medium model and gimbal camera:

cd autopilot/misc/rtsp_ai_player
qmake6 CONFIG+=gimbal && make
./rtsp_ai_player ~/azaion/models/onnx/yolov8m.onnx

How to use application on Orange PI 5.

Install ffmpeg and mediamtx to Ubuntu PC:

sudo apt update
sudo apt install ffmpeg
wget https://github.com/bluenviron/mediamtx/releases/download/v1.8.4/mediamtx_v1.8.4_linux_amd64.tar.gz
mkdir mediamtx
tar xf mediamtx_v1.8.4_linux_amd64.tar.gz -C mediamtx

Launch RTSP server in Ubuntu PC

cd mediamtx
./mediamtx

Play RTSP video stream in Ubuntu PC

ffmpeg -re -stream_loop -1 -i SOME_MP4_VIDEO_FILE -c copy -f rtsp rtsp://localhost:8554/live.stream

Test RTSP stream with ffplay in Ubuntu PC:

ffplay -rtsp_transport tcp rtsp://localhost:8554/live.stream

Compile and launch RTSP AI PLAYER in Orange PI 5:

Modify autopilot/misc/rtsp_ai_player/aiengineconfig.h and change 192.168.168.91 to IP address of RTSP source

cd autopilot/misc/rtsp_ai_player
mkdir build
cd build
qmake6 CONFIG+=opi5 .. && make -j8 && ./rtsp_ai_player [RKNN_MODEL_FILE]