CV capabilities of the Kurento Media Server Infrastructure


The Kurento Media Server runs as an MCU in both the Kurento Media Server Infrastructure and the OpenVidu Media Server as the main media control unit.

What is the Kurento Media Server? https://doc-kurento.readthedocs.io/en/6.10.0/user/about.html#kurento-media-server

Kurento offers a multimedia framework that eases the task of building multimedia applications with the following features:

  • Dynamic WebRTC Media pipelines: Kurento allows custom media pipelines connected to WebRTC peers like web browsers and mobile apps.
  • Client/Server Architecture: Apps developed with Kurento follow a client/server architecture. Kurento Media Server (KMS) is the server and offers a WebSocket interface implementing the Kurento Protocol, which allows Client Applications to define pipeline topologies.
  • Java and JavaScript Client Applications: The typical use case of a KMS deployment consists of a three-layer architecture, where the user’s browser interacts with the KMS server by means of an intermediate Client Application.
  • Third party Modules: The Kurento Media Server has an extensible architecture based on plugins.

The Kurento Media Server Infrastructure has 3 components,

  1. The Media Server itself
  2. Signalling server to control communication between clients and Media Server.
  3. Client side code-base for WebRTC peer creation.

To install KMS (Kurento Media Server ) use the following link on an Ubuntu 16.04:


DISTRO=”xenial”

sudo apt-key adv — keyserver keyserver.ubuntu.com — recv-keys 5AFA7A83

sudo tee “/etc/apt/sources.list.d/kurento.list” >/dev/null <<EOF
# Kurento Media Server — Release packages
deb [arch=amd64] http://ubuntu.openvidu.io/6.10.0 $DISTRO kms6
EOF

PACKAGES=(
# Development tools
build-essential
cmake
debhelper
default-jdk
gdb
git
maven
pkg-config
valgrind
wget
libboost-dev
libboost-filesystem-dev
libboost-log-dev
libboost-program-options-dev
libboost-regex-dev
libboost-system-dev
libboost-test-dev
libboost-thread-dev
libevent-dev
libglib2.0-dev
libglibmm-2.4-dev
libopencv-dev
libsigc++-2.0-dev
libsoup2.4-dev
libssl-dev
libvpx-dev
libxml2-utils
uuid-dev
# Kurento external libraries
gstreamer1.5-plugins-base
gstreamer1.5-plugins-good
gstreamer1.5-plugins-ugly
gstreamer1.5-plugins-bad
gstreamer1.5-libav
gstreamer1.5-nice
gstreamer1.5-tools
gstreamer1.5-x
libgstreamer1.5-dev
libgstreamer-plugins-base1.5-dev
libnice-dev
openh264-gst-plugins-bad-1.5
openwebrtc-gst-plugins-dev
kmsjsoncpp-dev
ffmpeg
)

sudo apt-get update

sudo apt-get install -y “${PACKAGES[@]}”

sudo apt-get install -y kurento-media-server-dev

We have now installed the development version of Kurento Media Server.

If we are running on Gcloud, AWS or any cloud providers, it is important to add a TURN configuration so that we do not block the server peer behind the NAT.

To configure a TURN server in KMS, uncomment the following lines in the WebRtcEndpoint configuration file, located at /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini:

turnURL=<user>:<password>@<serverIp>:<serverPort>

The parameter serverIp should be the public IP address of the TURN server. It must be an IP address, not a domain name.

See some examples of TURN configuration below:

turnURL=kurento:kurento@WWW.XXX.YYY.ZZZ:3478

Next, what if we want to add custom filters using OpenCV or other libraries to KMS. Here is what we can do.

git clone https://github.com/Kurento/kms-omni-build.git
cd kms-omni-build
git submodule init
git submodule update --recursive --remote
mkdir release
cd release
TYPE=Releasecmake -DCMAKE_BUILD_TYPE=$TYPE -DCMAKE_VERBOSE_MAKEFILE=ON ..
make

Now that the KMS Omni build package is built. Next we initialize a filer.

kurento-module-scaffold.sh <nameOfFilter> . opencv_filter
cd <nameOfFilter>

Add -fPIC to CFLAGS in line 51 and 52 and to CMakeLists.txt

nano CMakeLists.txt

If you want to prevent problems with (optional) debian package builder later on, add -Wno-error=date-time too

To build the module,

mkdir build
cmake ..

Edit the implementation file to add custom filters and code.

nano ../src/server/implementation/objects/<nameOfFilter>OpenCVImpl.cpp

In the editor, within the process function, add the following code.

cv::Mat matBN(mat.rows, mat.cols, CV_8UC1);cv::cvtColor(mat, matBN, /*COLOR_BGRA2GRAY*/ 10);cvtColor(matBN, mat, /*COLOR_GRAY2BGRA*/ 9);

The process function takes in a Mat as an argument and modifies the address of the Mat within the function. The Mat can be modified in any way as desired and the modified element would reference the address. In the above code, Mat is converted to a single channel image and converted to Grayscale. and then it is converted to a 4 channel grayscale image.

To build the filter after modifying the process code, run

make

A .so file will be generated, which needs to be copied under the Release folder

we created earlier. Once this is copied, the media server will be able to load the module dynamically at run time.

cp src/server/libkms<nameOfFilter>module.so <path_to_kms_omni_build>/kms-omni-build/release

Now run the following commands in the

cd <path_to_kms_omni_build>/kms-omni-build/release
export GST_DEBUG='3,Kurento*:4,kms*:4,rtpendpoint:4,webrtcendpoint:4' # For logging
kurento-media-server/server/kurento-media-server \  --modules-path=. \  --modules-config-path=./config \  --conf-file=./config/kurento.conf.json \  --gst-plugin-path=.

This runs the KMS Server with the filter loaded, you should see a message that says that the libkms<nameOfFilter>module.so has been loaded succesfully. Now everytime a peer media stream passes through the media server, it has the ability to use this filter to provide filters and other algorithms.

We also need to generate a node module for the module created for the Kurento Media Server. This node module will be used in our custom node script to call the module in the Kurento Media Server.

To build the js client/ node module we run the following command in the build folder of the module.

cd <nameOfFilter>/build
cmake .. -DGENERATE_JS_CLIENT_PROJECT=TRUE

This will create a folder called “js” in the build directory.

Open the package.json file in the “js” directory and add the following to the “dependencies”

"kurento-client": "github:Kurento/kurento-client-js",

Keep in mind the path of this “js” directory, this will need to be installed via npm in our custom node code.


The signalling server and client side can be found at the following link: https://github.com/Kurento/kurento-tutorial-node/tree/master/kurento-hello-world

git clone https://github.com/Kurento/kurento-tutorial-node
cd kurento-hello-world

On line https://github.com/Kurento/kurento-tutorial-node/blob/master/kurento-hello-world/server.js#L28,

add the following

kurento.register('kurento-module-<nameOfFilter>');

Next, in function createMediaElements (https://github.com/Kurento/kurento-tutorial-node/blob/master/kurento-hello-world/server.js#L223) replace the function with the following,

pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) 
{        
    if (error) 
    {
            return callback(error);        
    } pipeline.create('<nameOfFilter>.<nameOfFilter>', function(error, filter) 
    {
     if (error) 
        {
         return callback(error);
        }
     return callback(null, webRtcEndpoint, filter);
});
return callback(null, webRtcEndpoint);    
});

Then in line: https://github.com/Kurento/kurento-tutorial-node/blob/master/kurento-hello-world/server.js#L182 , change the signature of the function

createMediaElements(pipeline, ws, function(error, webRtcEndpoint, filter)

Next, in line function connectMediaElements, (https://github.com/Kurento/kurento-tutorial-node/blob/master/kurento-hello-world/server.js#L243)

function connectMediaElements(webRtcEndpoint, filter, callback) 
{   
    webRtcEndpoint.connect(webRtcEndpoint, function(error) 
 {        
         if (error) 
         {            
            return callback(error);        
         }   
         filter.connect(webRtcEndpoint, function (error) {
         if (error) 
         { 
            return callback(error);
          }
          return callback(null);     
         });
    });
}
  

Then in line https://github.com/Kurento/kurento-tutorial-node/blob/master/kurento-hello-world/server.js#L195, Change the signature of the function as below,

connectMediaElements(webRtcEndpoint, filter, function(error)

The server and client infrastructure for the Kurento Media Server is now ready. Run the server with

npm install
npm install <path_to_js_folder>
npm start

Leave a Reply