JP4343808B2 - Server in bidirectional image communication system, processing method thereof, and program - Google Patents

Description

  The present invention relates to a server-related technique in a server / client video conference system, and more particularly to a technique that enables data transmission / reception using a communication band between a server and a client effectively.

  In a video conferencing system using an IP network, when the communication bandwidth of the network fluctuates and the transmission speed decreases, the transmission delay increases, so that there is a problem that the quality of moving images and audio on the receiving side is hindered. .

  In order to solve such problems, Quality Meeting (http://avs.kddlabs.co.jp/qmeet/body.html) commercialized by KDDI monitors the bandwidth that can be communicated, and the client side uses an encoder. The bit rate is changed, and communication according to the band is performed (see Patent Document 1).

  However, even if the above-described conventional technology is used, the following problems occur in the server-client type video conference system.

  For example, in the configuration in which the client 1 and the client 2 are connected to the server, when the client 1 transmits the encoded data to the client 2 via the server, the above-described conventional technology can consider only the bandwidth between the client 1 and the server. Therefore, when the bandwidth between the client 2 and the server is narrower than the bandwidth between the client 1 and the server, the client 2 may not be able to receive the video of the client 1. Therefore, the bit rate of the encoder of the client 1 must be set according to the narrower band.

Therefore, when three or more clients are connected to the server, the encoder bit rate must be set to the line with the narrowest bandwidth in order to realize bidirectional communication between the three clients. Therefore, there is a problem that the bandwidth between the server and each client cannot be effectively used. In order to solve this problem, it is conceivable that the encoded data is decoded at the server and encoded at an appropriate bit rate for each client. However, this method requires complicated processing on the server side, and the load on the server is reduced. There is a problem of growing.
JP 2003-116133 A "Check Point H.323 / MPEG4 Textbook", IE Institute, 2001

  The present invention has been made in view of the above points, and an object of the present invention is to provide a technique that enables effective use of a bandwidth between a server and each client without imposing a great burden on the server.

  Bi-directional having a plurality of clients that encode video using a video compression method that generates a key frame compressed within the frame and an inter-frame compressed interpolated frame, and a server that replicates and distributes the video A server used in an image communication system, which is based on a means for receiving encoded data from a client, a means for copying the received encoded data and storing it in a buffer, and a transmission state from the server to the client. The video at the client based on the transmission state from the server to the client by selecting either the encoded data for the interpolated frame received from the client or the encoded data for the key frame received from the client and transmitting to the client Determine the refresh rate value for encoding, and at least Is solved by a server, characterized in that it comprises a means for transmitting the refresh rate value to one client.

  The server obtains the transmission state as a response time from transmission of encoded data to reception of a response to the encoded data from a client, and when the response time is less than a frame interval time, Encoding data for the latest interpolated frame received from the source client after receiving a response is transmitted, and if the response time is equal to or longer than the frame interval time, the latest data received from the source client after receiving the response The encoded data for the key frame is transmitted.

  In addition, when there is at least one client whose response time is equal to or longer than the frame interval time, the server predetermines the minimum value of the response time among the clients whose response time is equal to or longer than the frame interval time. The larger refresh rate value is compared, the larger value is set as the refresh rate value to be used by the client, and the refresh rate value is transmitted to the client.

  Instead of performing the above comparison, the minimum value of the response time may be used as a refresh rate value to be used by the client, and the refresh rate value may be transmitted to the client.

  As described above, according to the present invention, either encoded data for an interpolated frame or encoded data for a key frame is selected as encoded data to be transmitted to the destination client, depending on the transmission state to each client. Therefore, data transmission / reception suitable for the network bandwidth between the server and the client becomes possible. In addition, since the refresh rate value, which is the key frame transmission interval at the time of encoding, can be appropriately changed according to the transmission state to each client, even a client that can receive only key frames as a result of the above selection is a frame suitable for the band. Video can be received at a rate. In addition, even when the band fluctuates, encoded data dynamically corresponding to the fluctuating band can be transmitted and received.

  Furthermore, according to the present invention, since it is not necessary to monitor the bandwidth as in the prior art, even when the bandwidth of the network fluctuates rapidly, appropriate encoded data is transmitted at a rate suitable for the bandwidth. Can be sent to.

  Embodiments of the present invention will be described below with reference to the drawings. In the present embodiment, the video is encoded by an intra-frame compressed frame (referred to as a key frame in the present specification and claims) and an inter-frame compressed frame (in the present specification and claims). A video compression method for generating a frame is used. Examples of such a video compression method include MPEG4 and H.264. 263 (see Non-Patent Document 1) and the like.

  FIG. 1 shows a configuration of a video conference system according to an embodiment of the present invention. As shown in FIG. 1, the video conference system according to the embodiment of the present invention includes a server 3, a client A (4), a client B (5), and a client C (6). The client A and the client B are networks. 1 is connected to the server 3 via the network 1, and the client C is connected to the server 3 via the network 2.

  A camera is connected to each client, and video from the camera is encoded at a preset bit rate. The camera connected to each client is, for example, a USB camera or an NTSC TV camera connected via a capture board.

  The network 1 and the network 2 are the Internet, an intranet, or the like. In the network 1, the maximum bit rate between the server and the client is an upstream bit rate 1, and the downstream maximum bit rate is a downstream bit rate 1.

  Similarly, the uplink maximum bit rate between the server and the client in the network 2 is assumed to be the uplink bit rate 2, and the downlink maximum bit rate is assumed to be the downlink bit rate 2. Here, the downlink bit rate 2 is smaller than the uplink bit rate 1, and the downlink bit rate 1 is sufficiently larger than the uplink bit rate 1 and the uplink bit rate 2 (at least twice as much as those).

  The server 3 has a general function as a server in a server / client video conference system, and also has a function according to the present invention described later. The server 3 has a general hardware configuration including a CPU, a memory, a hard disk, a network communication device, and the like, and each unit according to the present invention is realized by executing a program installed in the server 3. The The program can be installed on the server from a recording medium such as a CD-ROM, or can be downloaded and installed on the server via a network.

  Each client is installed with a video conference program including a function for encoding video. The function of encoding video has a function of setting an encoding parameter and a function of using a refresh rate value received from a server as one of the encoding parameters.

  FIG. 2 is a flowchart showing the operation of the video conference system shown in FIG. With reference to FIG. 2, an outline of the operation of the video conference system according to the present embodiment will be described. Before the following operation, encoding parameters (bit rate, frame rate, refresh rate value indicating the interval between key frames compressed in the frame, etc.) are set. A predetermined maximum refresh rate value is set as the refresh rate value.

  First, camera images are acquired at each client (step 1). Then, the refresh rate value received from the server is set (step 2). The refresh rate value may be set only when an updated refresh rate value is received from the server.

  Next, a camera video encoding process is performed at each client (step 3). Here, the client A and the client B each encode the camera video at the same bit rate as the upstream bit rate 1, and the client C encodes the camera video at the same bit rate as the upstream bit rate 2. The frame rate is the same for all clients (for example, 30 fps).

  Next, the encoded video data is transmitted from each client to the server (step 4). Here, for example, RTP / UDP or TCP obtained by encapsulating RTP with HTTP is used as a communication protocol, and the encoded data is transmitted as a video packet divided in frame units.

  Next, the encoded data received by the server 3 is duplicated and stored for each client (step 5). Then, the encoded data is selected based on the transmission state from the server to the client, and the refresh rate value to be set in each client is determined (step 6). Then, the server 3 transmits the encoded data and the refresh rate value to each client (Step 7). The refresh rate value may be transmitted only when the refresh rate value is changed.

  The selection of the encoded data and the determination of the refresh rate value are performed based on the transmission state. In this embodiment, the server receives a response from the client to the encoded data transmitted from the server to the client. It is supposed to be time to do. This process will be described in detail later. Thereafter, the client that has received the encoded data performs a process of decoding and displaying the received data (step 8).

  Hereinafter, the processing in the server from duplication and accumulation to transmission of the video packet in steps 5 to 7 will be described in more detail with reference to FIGS. This processing is realized by a program according to the present invention. First, the basic process will be described, and the subsequent change of the refresh rate value will be described.

(Basic processing)
First, the case of FIG. 3 will be described. FIG. 3 shows a case where the video packet received from the client C is transmitted to the client A and the client B. As described above, the downstream bit rate 1 in FIG. 1 is sufficiently larger than the upstream bit rate 2, and the client C encodes the camera video at the same bit rate as the upstream bit rate 2. Therefore, in the case of FIG. The downlink bit rate 1 to A or the client B is sufficiently larger than the bit rate of data to be transmitted.

  In FIG. 3, I indicates a video packet for an I picture (key frame compressed in a frame), P indicates a P picture (an interpolation frame for forward prediction), and B indicates a video packet for a B picture (an interpolation frame for bidirectional prediction). .

  When the server receives a video packet from the client C, the server duplicates the video packet by the number of connected clients in the order in which the video packet is received, and stores it in a buffer prepared for each client. The buffer is realized as a predetermined area in the memory in the server, for example.

  The buffer amount prepared here is set so as to include at least one I-picture video packet. For example, in the case of FIG. 3, the buffer amount that can store I, B, P, and B is set.

  Next, processing for selecting encoded data (video packet) based on the transmission state in the server will be described.

  First, the video packet of the I picture stored in the buffer is transmitted to each client. Each client returns a response immediately after receiving the video packet, and notifies the server that the video packet of the I picture has been received. In this case, since the downlink bit rate is sufficiently secured, the time from video packet transmission to response reception is less than the frame interval time.

  In this case, the buffer is cleared and the next B picture video packet is awaited. When a video packet of B picture from client C arrives, it accumulates in a buffer, and transmits a video packet of B picture to each client in the same manner as an I picture video packet, and waits for a response. By repeating this, a video packet can be transmitted to each client. Here, the identification of the transmission and response packets is determined by matching the time stamps added to the packets.

  As described above, when the time from video packet transmission to response reception is less than the frame interval time, the same video packet as the video packet transmitted from the transmission source client is transmitted to the transmission destination client. Become.

  Next, the case shown in FIG. 4 will be described. FIG. 4 shows a case where the video packet received from the client A is transmitted to the client C. As described above, the downstream bit rate 2 in FIG. 1 is smaller than the upstream bit rate 1, and the client A encodes the camera video at the same bit rate as the upstream bit rate 1. Therefore, in the case of FIG. The downstream bit rate 2 is smaller than the bit rate of the transmitted data.

  First, as in the case of FIG. 3, an I picture video packet is received from the client A, copied and stored in the buffer, and the I picture video packet stored in the buffer is transmitted to the client C.

  However, in the case shown in FIG. 4, since the downlink bit rate is low, even if the video packet of the next B frame arrives at the server from the client A, the server has not yet received a response from the client C.

  In this case, unlike the case of FIG. 3, the video packet of the B picture is held in the buffer following the video packet of the I picture without clearing the buffer. Similarly, the subsequent P packet and B packet video packets are also held in the buffer. During this time, if there is a response from the client C, when the server receives the video packet of the next I picture from the client A, all the buffers are cleared and the video packet of the I picture is similarly transmitted to the client C. However, even if the video packet of the next I picture is received from the client A, if there is no response from the client C, transmission is not performed, and buffer accumulation and clearing are repeated until a response is received. In the case of FIG. 4 as well, the identification of the transmission and response packets is made by matching the time stamps added to the packets. Note that information indicating that a video packet has been received from the client A and a time stamp corresponding thereto are held separately from storing the video packet in the buffer.

  As described above, when the time from video packet transmission to response reception is equal to or longer than the frame interval time, only the video packet of I picture among the video packets transmitted from the transmission source client is transmitted to the transmission destination client. Will be.

(Refresh rate value change processing)
Now, as in the case of FIG. 3, when the bandwidth is sufficient and the time from video packet transmission to response reception is less than the frame interval time, all received frames can be transmitted to the client, so the refresh rate value is increased ( For example, it is possible to maintain high image quality by increasing the compression efficiency in 5 seconds.

  On the other hand, as in the case of FIG. 4, when the response time is equal to or longer than the frame interval time, the client receives only the I frame. Here, since the server receives the I frame at the refresh rate interval, when the refresh rate value is larger than the response time as in the case of FIG. 4, the client can only perform at the refresh rate time interval larger than the response time. I cannot receive I frames. That is, in this case, although the client can receive an I frame at a response time interval that is a shorter time interval, it cannot do so due to the restriction of the refresh rate, and the bandwidth from the server to the client is not effectively utilized. .

  Therefore, in the case of FIG. 4, the server transmits a refresh rate value smaller than the refresh rate value at that time to each client, and the client in the state as shown in FIG. Get the frame.

  If the refresh rate value is reduced, the compression efficiency is lowered and the image quality is lowered. Therefore, even if the refresh rate value is decreased, it is preferable to set the refresh rate value as large as possible. Therefore, in the present embodiment, the following is performed.

  For each client, a predetermined maximum refresh rate value is preset as a refresh rate value. The server also maintains its maximum refresh rate value. The server also holds a predetermined minimum refresh rate value (for example, 1 second). The minimum refresh rate value is a value determined from the image quality allowed in the subjective evaluation.

  During video communication by each client, the server holds the above response time for each client (for example, updating at regular time intervals) in a memory or the like, and a client whose response time is equal to or longer than the frame interval time. Check if it exists. If there is a client whose response time is longer than the frame interval time, the minimum response time among such clients (the response time of the client if there is one such client) and the minimum refresh The rate value is compared, the larger value is determined as the refresh rate value (if it is the same, the same value may be used), and the refresh rate value is transmitted to each client. Each client performs video encoding using the refresh rate value received from the server.

  By comparing the minimum value of the response time among the clients in the above state and the minimum refresh rate value and determining a larger value as the refresh rate value, all the clients in the above state have their own response time. An I frame can be received at an interval or a time interval close thereto. In addition, the refresh rate value can be reduced more than necessary by comparing the minimum response time value and the minimum refresh rate value among the clients in the above state and determining a larger value as the refresh rate value. The image quality can be kept as good as possible. In addition, when it is known that the response time of the client in the above state is approximately larger than the minimum refresh rate value, the response time is not compared without comparing the minimum response time value with the minimum refresh rate value. The minimum value may be determined as the refresh rate value.

  As an example, since the state shown in FIG. 4 has been reached, the server determines the refresh rate value as the same value as the response time of the client C, and shows the communication status after transmitting the refresh rate value to each client. In the case of FIG. 5, since the client C can receive I frames at about half the time interval as compared with the case of FIG. 4, the frame rate at the client C is about twice that of FIG. The client C can be used more effectively and can receive better video.

  The update processing of the refresh rate value may be performed every time the server always checks the response time and the minimum response time changes, or may be performed at regular time intervals. The refresh rate value increases as the minimum response time increases, but when the minimum response time exceeds the maximum refresh rate value, the refresh rate value is set to the maximum refresh rate value.

  Further, the updated refresh rate value may be sent only to a specific client instead of being sent to all clients participating in video communication. For example, when the clients A to D participate in video communication, it is known that the communication state between the clients A and B is good, and the image quality between the clients A and B is to be preferentially kept good. For example, when the client D enters the above state, the updated refresh rate value is sent only to the clients C and D.

  As described above, the encoded data received by the client is the same encoded data as the transmission source or encoded data including only an I frame. Therefore, regardless of which encoded data arrives, it can be decoded and displayed without the need for separate processing on the client side. In this way, transmission / reception suitable for the bandwidth between the server and each client is possible. Even when only I frames are received, transmission and reception suitable for the bandwidth between the server and each client can be performed by appropriately changing the refresh rate value. In addition, even when the band fluctuates, encoded data dynamically corresponding to the fluctuating band can be transmitted and received.

  Furthermore, since there is no need to monitor the bandwidth as in the prior art, even when the bandwidth of the network fluctuates rapidly, the encoded data can be transmitted to the destination client at a rate suitable for that bandwidth.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

It is a figure which shows the structure of the video conference system in embodiment of this invention. It is a flowchart showing operation | movement of the video conference system shown in FIG. 6 is a diagram for explaining the operation of the server when a video packet received from client C is transmitted to client A and client B. FIG. FIG. 6 is a diagram for explaining the operation of the server when a video packet received from client A is transmitted to client C. It is a figure for demonstrating operation | movement of the server after changing a refresh rate value.

Explanation of symbols

1, 2 Network 3 Server 4 Client A
5 Client B
6 Client C

Claims (6)

Bidirectional image having a plurality of clients that encode video using a video compression method that generates intraframe-compressed key frames and interframe-compressed interpolated frames, and a server that duplicates and distributes video A server used in a communication system,
Means for receiving encoded data from a client;
Means for copying the received encoded data and storing it in a buffer;
Based on the transmission state from the server to the client, the encoded data selection transmission for selecting the encoded data for the interpolated frame received from the client or the encoded data for the key frame received from the client and transmitting to the client. Means,
Refresh rate value determination and transmission means for determining a refresh rate value that is a key frame transmission time interval in the client based on a transmission state from the server to the client, and transmitting the refresh rate value to at least one client. ,
The encoded data selection transmission means acquires the transmission state as a response time from when the encoded data is transmitted to the client until the server receives a response to the encoded data from the client, and the response time If the response time is less than the frame interval time, the encoded data for the latest interpolation frame received from the transmission source client is transmitted after the response is received. After sending the encoded data for the latest key frame received from the sending client,
In a case where there is at least one client whose response time is equal to or longer than the frame interval time, the refresh rate value determination transmission unit is configured to reduce the minimum response time among the clients whose response time is equal to or longer than the frame interval time. And a predetermined minimum refresh rate value, a larger value is set as a refresh rate value to be used by each client, and the refresh rate value is transmitted to each client . Bidirectional image having a plurality of clients that encode video using a video compression method that generates intraframe-compressed key frames and interframe-compressed interpolated frames, and a server that duplicates and distributes video A server used in a communication system,
Means for receiving encoded data from a client;
Means for copying the received encoded data and storing it in a buffer;
Based on the transmission state from the server to the client, the encoded data selection transmission for selecting the encoded data for the interpolated frame received from the client or the encoded data for the key frame received from the client and transmitting to the client. Means,
Refresh rate value determination and transmission means for determining a refresh rate value that is a key frame transmission time interval in the client based on a transmission state from the server to the client, and transmitting the refresh rate value to at least one client. ,
The encoded data selection transmission means acquires the transmission state as a response time from when the encoded data is transmitted to the client until the server receives a response to the encoded data from the client, and the response time If the response time is less than the frame interval time, the encoded data for the latest interpolation frame received from the transmission source client is transmitted after the response is received. After sending the encoded data for the latest key frame received from the sending client,
In a case where there is at least one client whose response time is equal to or longer than the frame interval time, the refresh rate value determination transmission unit is configured to reduce the minimum response time among the clients whose response time is equal to or longer than the frame interval time. Is a refresh rate value to be used by each client, and the refresh rate value is transmitted to each client . Bidirectional image having a plurality of clients that encode video using a video compression method that generates intraframe-compressed key frames and interframe-compressed interpolated frames, and a server that duplicates and distributes video A program for causing a server used in a communication system to execute a process,
Means for receiving encoded data from a client;
Means for copying the received encoded data and storing it in a buffer;
Based on the transmission state from the server to the client, the encoded data selection transmission for selecting the encoded data for the interpolated frame received from the client or the encoded data for the key frame received from the client and transmitting to the client. Means,
A program for determining a refresh rate value that is a key frame transmission time interval in a client based on a transmission state from a server to a client, and functioning as a refresh rate value determination transmission unit that transmits the refresh rate value to at least one client And
The encoded data selection transmission means acquires the transmission state as a response time from when the encoded data is transmitted to the client until the server receives a response to the encoded data from the client, and the response time If the response time is less than the frame interval time, the encoded data for the latest interpolation frame received from the transmission source client is transmitted after the response is received. After sending the encoded data for the latest key frame received from the sending client,
In a case where there is at least one client whose response time is equal to or longer than the frame interval time, the refresh rate value determination transmission unit is configured to reduce the minimum response time among the clients whose response time is equal to or longer than the frame interval time. And a predetermined minimum refresh rate value, a larger value is set as a refresh rate value to be used by each client, and the refresh rate value is transmitted to each client . Bidirectional image having a plurality of clients that encode video using a video compression method that generates intraframe-compressed key frames and interframe-compressed interpolated frames, and a server that duplicates and distributes video A program for causing a server used in a communication system to execute a process,
Means for receiving encoded data from a client;
Means for copying the received encoded data and storing it in a buffer;
Based on the transmission state from the server to the client, the encoded data selection transmission for selecting the encoded data for the interpolated frame received from the client or the encoded data for the key frame received from the client and transmitting to the client. Means,
A program for determining a refresh rate value, which is a transmission time interval of key frames in a client, based on a transmission state from a server to a client, and functioning as a refresh rate value determination transmission unit for transmitting the refresh rate value to at least one client And
The encoded data selection transmission means acquires the transmission state as a response time from when the encoded data is transmitted to the client until the server receives a response to the encoded data from the client, and the response time If the response time is less than the frame interval time, the encoded data for the latest interpolation frame received from the transmission source client is transmitted after the response is received. After sending the encoded data for the latest key frame received from the sending client,
In a case where there is at least one client whose response time is equal to or longer than the frame interval time, the refresh rate value determination transmission unit is configured to reduce the minimum response time among the clients whose response time is equal to or longer than the frame interval time. Is a refresh rate value to be used by each client, and the refresh rate value is transmitted to each client . Bidirectional image having a plurality of clients that encode video using a video compression method that generates intraframe-compressed key frames and interframe-compressed interpolated frames, and a server that duplicates and distributes video A server used in a communication system performs a process,
Receiving encoded data from the client;
Duplicating the received encoded data and storing it in a buffer;
Based on the transmission state from the server to the client, the coded data for the interpolation frame received from the client, or the encoded data selection transmission to be transmitted to the client by selecting one of the encoded data for the key frames received from the client Steps,
A refresh rate value determination and transmission step of determining a refresh rate value, which is a transmission time interval of key frames in the client, based on a transmission state from the server to the client, and transmitting the refresh rate value to at least one client. ,
In the encoded data selection transmission step, the server obtains the transmission state as a response time from when the encoded data is transmitted to the client until a response to the encoded data is received from the client. When the time is less than the frame interval time, the encoded data for the latest interpolation frame received from the transmission source client is transmitted after the response is received, and when the response time is equal to or longer than the frame interval time, After receiving the response, send the encoded data for the latest key frame received from the sending client,
In the case where there is at least one client whose response time is equal to or longer than the frame interval time, in the refresh rate value determination transmission step, in the refresh rate value determination transmission step, the server responds among the clients whose response time is equal to or longer than the frame interval time. A method of comparing a minimum value of time with a predetermined minimum refresh rate value, setting a larger value as a refresh rate value to be used by each client, and transmitting the refresh rate value to each client . Bidirectional image having a plurality of clients that encode video using a video compression method that generates intraframe-compressed key frames and interframe-compressed interpolated frames, and a server that duplicates and distributes video A server used in a communication system performs a process,
Receiving encoded data from the client;
Duplicating the received encoded data and storing it in a buffer;
Based on the transmission state from the server to the client, the coded data for the interpolation frame received from the client, or the encoded data selection transmission to be transmitted to the client by selecting one of the encoded data for the key frames received from the client Steps,
A refresh rate value determination and transmission step of determining a refresh rate value, which is a transmission time interval of key frames in the client, based on a transmission state from the server to the client, and transmitting the refresh rate value to at least one client. ,
In the encoded data selection transmission step, the server obtains the transmission state as a response time from when the encoded data is transmitted to the client until a response to the encoded data is received from the client. When the time is less than the frame interval time, the encoded data for the latest interpolation frame received from the transmission source client is transmitted after the response is received, and when the response time is equal to or longer than the frame interval time, After receiving the response, send the encoded data for the latest key frame received from the sending client,
In the case where there is at least one client whose response time is equal to or longer than the frame interval time, in the refresh rate value determination transmission step, in the refresh rate value determination transmission step, the server responds among the clients whose response time is equal to or longer than the frame interval time. A method wherein a minimum value of time is a refresh rate value to be used by each client, and the refresh rate value is transmitted to each client .

Images