JP2004350300A - Decoding device for image signal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an error correction function strong for transmission errors, and to suppress retransmission signals and picture freeze generation for the dynamic image transmission of super low rate transmission. <P>SOLUTION: Encoded bitstreams, for which improvement for turning to a super low rate is executed to an existing dynamic image compression standard H.261, are received, and the error correction function is organically executed for them in the respective levels of codes, syntax, patterns, signals, and recognition with a protocol as a base by a communication path decoding part 35, a pattern signal level error correction part 37, and a processing part 38 of a recognition level, etc., in a stage of decoding processing. Further, by the global check of the encoded bit streams for which a required bit amount is considered, the error correction of the patterns, the signal, and the syntax is driven, and an error correction based on the evaluation of the recognition level is performed by using mode information, and area information (human being or the like) based on a 2D template. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention provides an image signal decoding apparatus having a function of receiving an image signal as an encoded bit stream compressed using a variable length code, decoding the image signal into an image signal, and correcting an error of the image signal. About.

  2. Description of the Related Art In recent years, multimedia use of information communication terminals has been rapidly progressing, and it is indispensable in the future business development to create various added values based on digital data transmission in telephones. In particular, in PHS (personal handy phone system), transmission of not only conventional voice and text information but also natural still images and moving images has been seriously studied, and multimedia of telephones is very near. It is expected to be commonplace in the future.

However, in transmitting such an image signal, in a non-compressed state, a TV image having a signal amount equivalent to about 166 Mbps is assigned to PHS (384 kbps, image allocation is 32 kbps or 48 kbps) or the transmission rate of an existing telephone line. (About 10 kbps).
Here, a moving image compression technique used for a conventional TV conference or video CD is required. However, even in this standard, for example, the current TV conference standard ITU-T / H. H.261 targets 64 kbps to 2 Mbps, and ISO / MPEG1 targets about 1.5 Mbps, and does not support ultra-low-rate transmission of 64 kbps or less. Note that H.264 recommended in November 1995 The H.263 standard covers 4.8 kbps to 64 kbps, but since the contents are not disclosed, the actual situation cannot be recognized. Also, the standard states that 261 is not guaranteed.

  Furthermore, such a moving image compression technique is mainly intended for a wired system. For example, the level of a data transmission error occurring in wireless communication in mobile communication (10-2 to -3) No countermeasures have been taken completely, and a method called demand refresh using a retransmission request by ARQ (automatic repeat request) is usually used in a wireless system. However, when such ARQ is used, it is inevitable that the picture freezes on the receiving side until the retransmission of the image information is completed. Therefore, in a place where the radio wave condition is bad, a state in which the picture is frozen is continued, and there is a problem that moving image transmission becomes substantially impossible.

On the other hand, in still image transmission, a method of dividing data into groups according to importance and controlling the rate has been proposed, but has not been put to practical use. At a transmission rate of about 384 kbps, there has been reported an example in which a transmission error is dealt with by changing the data transmission order and using an error correction code, but there is no report on ultra-low-rate wireless transmission of 64 kbps or less.
The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a strong error correction function against transmission errors as a function of a decoding device, thereby enabling ultra-low-rate transmission of PHS and digital mobile phones. Another object of the present invention is to provide an image signal decoding apparatus capable of minimizing the occurrence of a data retransmission mode and the occurrence of a picture freeze even when a moving image is transmitted by using.

  According to the present invention, when the mode information is included in the image signal, the mode control means sets the mode condition specified by the mode information to perform the decoding operation. become. The error correction means can perform the error correction process by giving priority to the mode condition when the decoding result of the image signal does not conform to the set mode condition. As a result, the amount of information as an image signal is reduced, but the incidental condition similar to the mode condition on the transmission side is set, so that the decoding capability can be improved in error correction.

  According to the second to fifth aspects, a use environment and a subject are set as the mode conditions in the above-described mode control, whereby the personal information storage means and the background information storage means are set according to the set use environment or the subject. The decoding process can be performed by using the stored data, and the decoding error can be corrected. More specifically, there are limited elements that are determined by the camera depending on whether the camera is fixed or movable in various usage environments in a car, indoors or outdoors, or a subject such as a person or landscape. The error correction function of the decoding process can be improved according to the set use environment mode.

  According to the sixth aspect, the coding mode can be set as the mode condition in the mode control. The mode control means sets the decoding condition based on the set coding mode information, and sets the error correction means. Can perform error correction by judging whether or not the coding mode conditions are satisfied, thereby improving the error correction capability. In this case, according to claims 7 and 8, as the encoding mode, it is possible to set each encoding mode of a moving image, a quasi-moving image, and a still image, or a model-based mode. And the setting is made in accordance with the transmission rate limitation.

  In the ninth aspect, as a function of the error correction means, the level of a request signal to be transmitted to the transmission side when a decoding stoppage occurs due to an error or the like included in an image signal during a decoding process. Can be selectively set in four cases, whereby the level of the reproduced image is switched stepwise according to the state of the communication path, etc., so that the stop of decoding is not continued. Can be.

  According to the tenth aspect, a personal image of a specific person can be registered as personal information in the personal information storage means as required, so that the personal information can be appropriately read and used at the time of mode control by the mode control means. At any time, by reading as needed, the registered state of the person can be confirmed, and the usability is improved. According to the eleventh aspect, the personal information stored in the personal information storage means is used in the model-based mode. For example, even if the communication situation deteriorates, the communication partner based on the registered personal information is used. Can be reproduced to some extent by the model-based mode control.

  According to the twelfth aspect, the color information estimating means estimates the color information of each block of the decoded pixel data from the past color information or the color information of surrounding blocks. In this case, the reason that such estimation is possible is that the probability that a normal image signal will accidentally generate a color different from that of the surrounding blocks in a tile-shaped shape corresponding to the grid of the pixel blocks is extremely high. It is based on the fact that it is small. Then, when the error amount between the estimated color information and the color information obtained by the decoding process for the target block is calculated by the error evaluating unit, the error correcting unit calculates a correct image if the value of the error amount is small. If the signal is determined to be a signal and the value of the error amount is large, error correction is performed by replacing the color information data of the target block with the estimated color information data. As a result, it is possible to improve the ability to correct errors when decoding color information at the signal level that cannot be corrected at the code level or the pattern level.

  In the above case, according to the thirteenth aspect, when the error amount between the color information data of the target block and the estimated color information data is large, a code describing the orthogonal transform coefficient data by returning to the code level upon error correction. Is searched for an ESC code indicating that it is described not as a variable-length code but as a fixed-length code, and error correction is performed. That is, the ESC code is searched by the transform coefficient decoding means, the similarity between the ESC code at that time and the variable-length code decoded before correction is obtained, and the bit coincidence ratio is equal to or more than a predetermined value (for example, 6 bits (5 bits) is stored in the ESC position storage means, and when an error in the signal level is detected, the process returns to another stored ESC position and performs the decoding process again to execute the decoding of the image signal. Decoding is performed, and re-evaluation is performed by the error correction means. As a result, when a decoding error at the signal level is detected, the signal level returns to the code level and the function of error correction can be exhibited with high accuracy.

  Further, according to claim 14, when the error of the color information data detected in the decoding process of the image signal is replaced with the estimated color information or the like, the code level is used in the decoding process of the next block. Since it is necessary to determine the position at which decoding of the coded bit stream is restarted, an EOB code indicating the end position of data in block units is searched for this purpose. That is, in the search for the EOB code by the EOB detection means, when the EOB code is not detected by the EOB detection means within a predetermined condition range in the target block, the EOB code is detected again by assuming a one-bit error, and When the condition range is satisfied, the decoding process can be restarted by specifying this as an EOB code.

  According to the present invention, when the error correction means includes an error in the color information of the pixel data obtained by performing the decoding process in units of blocks, the color information of the past color information data or the color of the surrounding blocks is used. The error correction is performed by estimating from the information data, and the error propagation preventing means transmits the image signal of the block unit which is the important area of the decoded image to the transmission side in accordance with the frequency of the error correction performed by the error correction means. Since the forced intra block data is periodically transmitted, even if errors due to error correction are accumulated, the image data is disturbed due to the accumulated error by periodically obtaining block data refreshed by the forced intra block data. Can be prevented.

According to claim 16, the area specifying means specifies the target area based on the area information data described in the user data area in the coded bit stream data. An error included in the image signal in the decoding process of the specified target area can be corrected based on the area information.
In this case, in claim 17, a plurality of basic templates are provided, and the area specifying means selects and sets the basic template specified by the area information described in the coded bit stream data from the transmitting side, and sets this as the target area. The part is deformed so as to correspond to. According to the eighteenth aspect, such a basic template is provided in a hierarchical manner, and the specified portion can be expressed as two-dimensional coordinates in the basic template.

  Furthermore, in claim 19, by providing the motion compensating means, when the area including the block currently being decoded is specified by the area specifying means, based on the motion vector of the block in the already decoded area. Motion compensation of the block can be performed. According to the twentieth aspect, by providing the color information compensating means, it is possible to correct the color information data of the block in the already decoded area based on the color information data such as luminance, color and texture of the block. Become like According to the twenty-first aspect, when an error occurs during the decoding of the image data in the background area, the error correction unit can correct the error based on the background information in the background information storage unit.

According to the present invention, when the object cannot be specified as a result of decoding the image signal, the pseudo area specifying means performs an evaluation calculation centering on the image center of gravity based on the pattern information data of the past frame. This makes it possible to specify a pseudo target area.
According to the twenty-third aspect, the code checking means can globally check the start code of a specific bit pattern indicating the head position of one frame with respect to the bit stream data accumulated in the buffer. In starting the decoding process, other information obtained based on the start position of the frame can be efficiently obtained.

  According to a twenty-fourth aspect of the present invention, a specific bit added at the beginning of a first block group of a plurality of block groups of one frame of pixel data is similarly globally checked by a code checking unit. Since the GOB (Group of Blocks) start code of the pattern can be detected, the efficiency of the decoding process can be further improved.

  Furthermore, in the twenty-fifth aspect, the above-described global check is performed by a fuzzy matching process in which when a bit error rate of a specific bit pattern is equal to or lower than a predetermined level, this is specified as a start code. , Even if is mixed, the start code can be specified with a high probability, and it is possible to prevent the search from being disabled as much as possible.

According to the twenty-sixth aspect, when specifying the start code by the code checking means, the start code can be determined while correlating with another start code based on data indicating the required number of bits following the start code. Therefore, even when the data indicating the required number of bits has an error, the position of the correct start code can be specified.
Further, in detecting the data of the required number of bits, the code checking means has a function of obtaining the detected bit pattern of the GB start code as an inverted pattern in which the upper and lower sides are exchanged. ing. Thus, when the data of the required number of bits set on the transmission side is within a predetermined bit range, the number of bits used for the data can be reduced.

Hereinafter, an embodiment in which the present invention is applied to a configuration assuming a car phone will be described with reference to the drawings.
FIG. 2 shows a functional block configuration of an encoder 1 as an encoder, and FIG. 1 shows a functional block configuration of a decoder 2 as a decoder. Hereinafter, the overall configuration will be schematically described with reference to FIGS.

  In the encoder 1 shown in FIG. 2, a camera 3 as an image pickup means shoots an object and outputs image data as a video source as a normal analog signal. In this case, the video source is not limited to an image source such as the camera 3, but a video signal or the like stored in advance can be used. A video source based on image data captured by the camera 3 is input to an A / D converter 5 via a changeover switch 4 and is converted into a digital signal. The A / D converter 5 is connected to a quadrature converter 7 via a changeover switch 6. Also, in this case, the A / D converter 5 is connected to the orthogonal transform unit 7 even through the subtractor 8 and the changeover switch 6. The switching control of the changeover switch 6 controls whether or not the subtractor 8 is interposed.

  The orthogonal transform unit 7 performs the H.264 conversion. Similar to that of the H.261 standard, the original image or the prediction difference image is subjected to a two-dimensional orthogonal transform (in this embodiment, a discrete cosine transform (DCT) according to the H.261 standard) in block units. The conversion output is output to the quantization unit 9. The quantization unit 9 is configured to quantize the transform coefficient data provided from the orthogonal transform unit 7 with 9-bit precision by linear quantization or quantization with a dead zone, and output it to the channel coding unit 10. . The communication path encoding unit 10 converts the image signal quantized and converted from the quantization unit 9 into an encoded bit stream in accordance with a predetermined conversion rule. The signal is output to a communication path 12 such as a wireless path.

  The inverse quantization unit 13 receives the quantized signal converted by the quantization unit 9, converts the image signal quantized by the quantization unit 9 into digital data before quantization again, and inversely converts the image signal. Output to the converter 14. The inverse conversion unit 14 converts the digital data into analog data, restores the image data to be substantially the same as the image data obtained from the video source, and outputs the image data to the prediction memory 16 via the adder 15. The prediction memory 16 is for predicting image data in the next frame based on image data taken in from a video source, and outputs it as a subtraction signal to the subtractor 8 via the loop filter 17 and the changeover switch 18. It has become.

  The loop filter 17 functions as a spatial low-pass filter that performs smoothing to suppress a sudden change in color or luminance between adjacent blocks. The output of the loop filter 17 is transmitted from the changeover switch 18 to another changeover switch. A signal 19 is supplied to the adder 15 via an adder 19 as an addition signal. The output of the prediction memory 16 is provided to a motion detection unit 20. The motion detection unit 20 converts the output of the prediction memory 16 into a frame based on the image data of the video source supplied from the A / D converter 5 and both data. A signal is supplied to the quantization unit 9 and a signal is supplied to the communication channel coding unit 10 so as to compensate for the movement between them.

  The encoding control unit 21 performs various controls of the encoding process. The encoding control unit 21 receives conversion information from the orthogonal transformation unit 7 and data such as a buffer usage rate from the FIFO buffer 11. In addition, based on these data, adaptive quantization control for setting the level of quantization conversion of the quantization unit 9 and adaptive rate control are performed. An attribute memory 22 for storing attribute data of an image signal obtained as a result of encoding is configured to receive attribute data to be stored from an encoding control unit 21. The attribute data currently being encoded is predicted based on the past attribute data stored in the storage 22, and is provided to the encoding control unit 21. As described above, H. An encoding processing unit 24 based on the H.261 standard is configured.

  Next, a characteristic configuration added in the present embodiment will be described. That is, the area extraction / recognition processing unit 25 is configured as follows. The target area extracting unit 26 executes a target area extracting process as described later based on the information of the motion vector given from the motion detecting unit 20, and outputs information to the template database 27. The 2D (two-dimensional) template matching unit 28 performs a matching process based on the data from the template database 27, and outputs a processing result to the target region extracting unit 26 and the channel encoding unit 10.

  The model-based prediction unit 29 retrieves data from the three-dimensional shape database 30 and performs image reproduction processing in the model-based mode based on the data stored in the person memory 31 and the background memory 32. The signal is supplied to the subtractor 8 as a subtraction input via the conversion unit 10 and the changeover switch 18. The person memory 31 and the background memory 32 are configured to receive person data and background data to be stored from the target area extraction unit 26.

  Further, the encoder 1 is connected to a mode control unit 33 for controlling the mode of the encoder 1, and various details of the mode control are performed as described later. . Further, a human interface 34 for transmitting and receiving information to and from a user is connected to the mode control unit 33. Specifically, the human interface 34 includes a display unit, a voice input / output unit, an operation input unit, and the like, and is controlled so as to quickly and appropriately perform mode control. The mode control unit 33 is also connected to the decoder 2 side, and performs the mode control of each unit of the decoder 2 in the same manner as described in the operation section. The human interface 34 can also be used in the decoder 2.

  The personal image information can be registered and stored in the personal memory 31 as personal identification information through the human interface 34, and the personal identification information is input via the human interface 34 at an appropriate timing. It is configured to be called from the person memory 31 based on the information, and to be displayed as registered personal identification information even in cases other than communication.

  Next, in the decoder 2 shown in FIG. 1, the communication path decoding unit 35 that receives the coded bit stream of the image signal data transmitted via the transmission path has a hierarchical coding attribute (macro block attribute). Type, etc.) and output. It is connected to a decoding processing unit 36 having a basic configuration conforming to the H.261 standard. Further, the configuration for executing the error correction function includes a pattern / signal level error correction unit 37, a recognition level processing unit 38, and a memory 39 as storage means for storing various data. In the decoding process, the mode control unit 33 controls the above by exchanging signals with the above-described units, and the finally decoded image signal is supplied to the changeover switch 40 and the D / A The data is output via the converter 41.

  In the above-described communication path decoding unit 35, the coded bit stream data of the image signal transmitted via the external communication path or the transmission path 12 such as in the air is input to the FIFO buffer 42, and the data of at least one frame is transmitted. Is accumulated. The coded bit stream data stored in the FIFO buffer 42 is subjected to grammatical interpretation of the code level in the parser 43. The fuzzy matching unit 44 searches the bit stream data stored in the FIFO buffer 42 globally, specifies PSC and GBSC described by a specific code by fuzzy matching processing, and also specifies ESC and GBC described by the specific code. EOB and the like are searched for and specified, and their positions are stored in the storage unit 45, and are used when the error correction function is executed as described later.

  Further, the parser 43 searches for other codes or data or interprets grammar based on the position of the specific code searched by the fuzzy matching unit 44, and in that case, the variable length code (VLC )). The error determining unit 47 determines whether or not there is an error when performing a grammatical interpretation in the parser 43 based on the position of the specific code specified by the fuzzy matching unit 44. Is corrected at the code level and output. Then, the parser 43 outputs mode information obtained by decoding the bit stream data to the mode control unit 33.

In the decoding processing unit 36, an inverse quantization unit 48 receives a decoded output from the parser 43, and performs inverse quantization conversion on the quantized image signal data and outputs the data to an inverse transformation unit 49. I do. The inverse transform unit 49 inversely transforms the orthogonal transform data corresponding to the frequency signal included in the image signal into data corresponding to the position signal, and outputs the data as decoded data.
The output of the inverse conversion unit 49 is input to the error determination / correction unit 50 via the path input to the error determination / correction unit 50 of the pattern / signal level error correction unit 37 and the changeover switch 51 and the adder 52. Routes are provided. The prediction memory 53 receives data from the error determination / correction unit 50 and the motion compensation unit 54, and is connected to the addition input of the adder 52 via a loop filter 55 and a changeover switch 56.

  In the pattern / signal level error correction section 37, the error determination / correction section 50 supplies data to the frame memory 57, and receives estimated image data via the pixel value estimation section 58 and the image estimation section 59. I have. The motion vector estimation unit 60 obtains data from the memory 39 and outputs the estimated motion vector data to the motion compensation unit 54. Further, the attribute estimating unit 61 reads out the past attribute data stored in the memory 39 and supplies the past attribute data to the decoding control unit 62 of the decoding processing unit 36 to obtain information on the decoding operation of the inverse quantization unit 48 and the inverse transform unit 49. give.

The memory 39 includes a storage unit including a mode information unit 39a, an area information unit 39b, a pattern attribute unit 39c, a 2D (two-dimensional) motion vector unit 39d, and a personal identification information unit 39e.
In the recognition level processing unit 38, the model-based prediction unit 63 receives 3D (three-dimensional) shape data 64 a from the three-dimensional shape database 64 and receives data of a person image from the person image database 65. The prediction is performed and the data is output to the error determination / correction unit 50. Data from the memory 39 is given to the person image database 65 and the three-dimensional shape database 64.

  The background memory 66 as the background information storage means and the person memory 67 as the person information storage means are configured to be able to store the background information and the person information of the image signal output to the D / A converter 41, Data is output to the frame memory 57, and data is exchanged with the person image database 65. The area reproducing unit 68 is provided with a 2D (two-dimensional) template 69a selected in the template database 69 based on the data from the memory 39, performs area reproduction, and outputs the area to the frame memory 57.

  Next, the operation of this embodiment will be described with reference to FIGS. In the following description, [A] the operation of the image signal encoding process in the encoder 1 will be described in the encoder, and the encoded bit in the decoder 2 will be described in [B] the decoder. The operation of the stream decoding process will be described. In the [A] encoder, various methods for reducing the code amount in the encoding process at an extremely low rate are listed for each item ([A-1] to [A-4]). The operation will be described mainly with reference to the flowchart of the encoding processing program (FIGS. 5 to 7) (section [A-5]), and in the [B] decoder, the decoding error in the decoding process will be described. Various methods of detection and correction are listed for each item (sections [B-1] to [B-8]), and the operation is described mainly with reference to the flowchart of the decoding processing program (FIGS. 8 to 11). (Section [B-9]).

[A] Encoder [A-1] Basic Operation First, the basic operation of the encoder 1 will be described. As a basic operation, the encoder 1 takes in data of an image captured by the camera 3, performs an encoding process of the image signal, and outputs the encoded signal to the communication channel 12 as encoded bit stream data. In this case, the encoding process for the first frame and the encoding process for the second and subsequent frames are performed separately.

First, in the first frame, an A / D converter 5 converts a video source that supplies a scene including an object captured by the camera 3 into a digital signal. According to the H.261 standard, intra-frame coding (INTRA) is performed in units of macroblocks according to the CIF format, and this is transmitted as an INTRA picture.
Further, after the second frame, only the motion compensation prediction difference for the first frame is subjected to a two-dimensional orthogonal transform (here, DCT (discrete cosine transform) conforming to H.261), and then the encoding control unit 21 performs buffering. After undergoing adaptive quantization according to the amount, coding rate, and other state values of the encoder 1, the data is temporarily stored in the output FIFO buffer 11 via the channel coding unit 10. The output FIFO buffer 11 allows encoded bit streams having different amounts of information in frame units to be transmitted to a communication path in accordance with a predetermined encoding rate.

  The coded bit stream is converted into data having a four-layer structure of PICTURE / GOB / MBK / BLK (see FIG. 30) according to the standard of the CIF (common intermediate format) format as described later. A set of variable length codes representing a sequence of DCT significant coefficients of the BLK (block) layer is generated with an attribute header attached to each layer. On the other hand, the motion-compensated prediction frame is generated as an output of the frame addition loop of FIG. 2, is compared with the original image, and sends new difference information to the encoding mechanism.

In the above-described encoding process, the encoder 1 according to the present embodiment performs an encoding process at an extremely low bit rate by performing a unique process as described below.
That is, in the present embodiment, H.264, which is an existing international standard for video compression for video conferencing and video telephony, is used. On the premise based on the H.261 standard, a mode control function and a region extraction function are added as unique processing steps. Under such a configuration, processes such as encoding control and syntax change are executed at each level of signal, pattern, code, and recognition, and finally, an ultra-low rate of about 64 kbps to 4.8 kbps. This is an implementation of image coding. FIG. 3 shows a conceptual correlation between these control contents.

  In the following, description of [A-2] mode control, [A-3] extraction of a target area, [A-4] encoding control operation, and [A-5] operation according to an encoding processing program will be given. The principle and method of the coding method will be described by classifying the coding method into large items and further classifying the large items into small items.

[A-2] Mode Control FIG. 3 conceptually shows the contents of the ultra-low rate encoding in the present embodiment. The mode control performed in the present embodiment includes, for example, (1) usage environment, Mode control is performed for each of the three categories of (2) subject and (3) coding control. In this case, the progress of the mode in each mode control is not necessarily performed completely independently. For example, encoding may be performed using two or more modes simultaneously in each category.

  These transition relations are governed by the state of the encoder 1 and the decoder 2, that is, the buffer amount of the FIFO buffer 11, the currently set mode state, the state such as the coding rate or the amount of motion generation, and determination information. There are two cases: a case where the state is changed, and a case where the state depends on the stochastic state transition. In the following, the operation of mode control based on these categories (1) to (3) and the difference in state transition will be described.

(1) Usage environment Regarding the mode control of the usage environment, as shown below, fixed and movable modes corresponding to each place of the car, indoor and outdoor are considered, and (a) fixed mode in the car, The modes can be classified into six modes: b) a movable mode in a car, (c) an indoor fixed mode, (d) an indoor movable mode, (e) an outdoor fixed mode, and (f) an outdoor movable mode.

(A) In-vehicle fixed mode This mode sets the use environment of the rear seat that is most conceivable as a TV phone for a vehicle, and limits various conditions from its positional relationship as shown in FIGS. 12 and 13, for example. There are features that can be. This can be summarized as follows.
1) The portion of the background inside the car can be almost fixed, and the scenery outside the car seen from the window becomes a moving area during traveling.

2) The distance L from the camera 3 to the person P can be limited (see FIG. 12). As a practical value, the distance L can be set to, for example, about 1 m or less.
3) As will be described in the section on the next subject mode, in this use environment, the subject is mostly a person mode centered on a person (see FIG. 13), and as a result of the moving area extraction, the area ratio is the largest. The probability that the region is a person becomes very high.

4) When the rear seat is used as a target, the number of persons P to be photographed is most likely to be one.
5) In the case of a car, the number of persons P that can enter the field of view of the camera 3 is considered to be at most about four.

(B) In-vehicle movable mode In this mode, there is a high possibility that a landscape mode as a subject mode is used in addition to the person mode.
(C) Indoor fixed mode This is almost the same as the vehicle fixed mode, but the outside scenery seen from the window is likely to be fixed, which is advantageous for reducing the amount of information.
(D) Indoor movable mode There is a high possibility that a landscape mode is used in addition to the person mode.

(E) Outdoor fixed mode This mode can be applied to traffic monitoring, security, pet monitoring, and the like. Also, in this case, since the camera 3 is fixedly installed, wired transmission becomes possible, and the transmission error rate is hardly reduced.
(F) Outdoor movable mode The use environment is the harshest condition. In this case, it is a prerequisite to perform ultra-low bit rate transmission in a wireless system because it is movable. In addition, when a portable device is used, there is a high possibility that a landscape mode is used in addition to the portrait mode. In addition, it is assumed that the camera 3 shakes more due to camera shake during a call or the like. In practice, it is assumed that the operation in the quasi-moving image mode or the still image mode is mainly performed.

(2) Subject In this case, four encoding modes are switched and set as follows according to the encoding target and the application.
(A) Person mode This person mode is set as a default mode used with the highest priority in normal communication. In this person mode, as shown in FIG. 14, mode switching as described below is further performed according to the background and the use environment. In this case, the mode switching condition is normally set and switched automatically based on the extraction result of the target area and the distance determination.

A) Head mode (when the target person P is one)
B) Upper body mode (when the target person P is one)
C) Full-body mode (when the target person P is one)
D) Multi-person mode For example, in each use environment shown in FIG. 14, (a) the head mode of a) in a car, (b) the upper body mode of a) indoors, and (c) the upper body mode outdoors. A) Upper body mode or C) Full body mode is set. In the multi-person mode d), a particularly detailed template (see FIG. 25, see the description of the template to be described later) is not prepared, and the scene is assumed to be a transient short time. It is left to another control according to the degree of generation of the encoded information amount.

(B) Landscape Mode For example, in a running car, the subject is often a person during conversation, and the mode determination tree has a high probability of setting the person mode. On the other hand, if the user intentionally wants to transmit the scenery or things outside the vehicle as an image, the coding is determined based on the presence or absence of motion, the amount of information generated due to the fineness of the texture, and the presence or absence of a person. Change the mode to the semi-moving image or still image mode.

(C) Still Object Mode When a stationary object is targeted, the amount of encoded information is relatively small, so that a normal moving image mode can be selected as the encoding mode.
(D) Drawing / character mode The high-resolution still image mode can be selected as the encoding mode.

(3) Coding control (a) Image center priority mode This mode is set when there is no foresight information for the object on the screen and there is no manual initial setting mode prior to this. You. That is, as a situation, it is assumed that the camera 3 is pointed at something that one simply wants to photograph. In this case, when the target is positioned at the center of the image by pointing the camera 3, it can be generally assumed that the area to be watched is also a part close to the center of the image (that is, the camera 3 is positioned with respect to the center of the target). The horizontal and vertical angles α and β of the posture can be assumed to be values close to zero).

  Therefore, an encoding control operation is performed such that the bit amount is preferentially assigned near the center of the image for encoding, and the assigned amount is gradually reduced toward the periphery. This is specifically performed by [A-3] quantization control for each area of encoding control, use of a background memory, control of the maximum number of significant transform coefficients, and the like, which will be described later. In this mode, the conditions for shifting to another mode are set corresponding to the following three cases.

1) When the movement of the target area is detected If the target area has been extracted, the mode shifts to the target area priority mode. Here, in the following case, the target area cannot be followed, so that the area center-of-gravity mode in the target area priority mode is selected.
A) When a 2D (two-dimensional) template is not determined
B) When the adaptation of the 2D template has not been completed
C) In the case of the landscape mode without a 2D template (the 2D template will be described later).
2) When the subject mode is the person mode In the case of the target area priority mode and the person mode, the 2D template is suitable if it does not deviate from the front image. In this case, identification of each part such as the head, eyes, and mouth is further started.
3) Specifying the target object category based on the target region extraction and the motion analysis result If there is no motion information or region information, the center region of the screen is preferentially encoded. In this case, giving priority to the screen center region corresponds to, as shown in FIG. 15, assuming that a block within a value calculated as a fixed distance from the screen center, for example, as an average distance Dav, is regarded as the target region.

  Further, here, in calculating the average distance Dav, the Eucrid distance D (n) shown in FIG. 3A or the absolute value distance D (n) shown in FIG. As shown in the following equation (1), the average value Dav of the distance D (n) (in this case, the absolute value distance is used as a reference) is calculated for the blocks (length 18 × width 22) set on the screen. As a result of this calculation, the average distance Dav is about 12, and the hatched area indicated by the oblique rectangle in FIG.

ただし、Ｎ＝２２×１８＝３９６、ｄｄ（ｉ，ｊ）は、絶対値距離では、
ｄｄ（ｉ，ｊ）＝｜ｉ−１１｜＋｜ｊ−９｜
となる。

Where N = 22 × 18 = 396, and dd (i, j) is an absolute value distance.
dd (i, j) = | i-11 | + | j-9 |
It becomes.

(B) Target area priority mode 1) Background memory use mode In the person mode, a background image or a first image prepared in advance is used in the person mode based on the area information obtained in [A-2] Extraction of target area described later. In this mode, control is performed as follows by using background information stored in the background memory 66 using a background image transmitted in a frame.
A) After that, do not transmit any background information at all.
B) Although background information is transmitted, the background change information is periodically dispersed and sent, and the background is gradually updated.

2) Region center-of-gravity mode In this region center-of-gravity mode, the image center in the above-described image center mode is considered to be the center of gravity of the target region, and the quantization control for each region is driven around the center. The quantization control for each area will be described later in the section of [A-3] Coding control, and will not be described here.
3) Target area tracking mode In this mode, when the target area moves on the image plane, the above-described 2) area center of gravity mode can be applied by detecting the position of the center of gravity of the target area that changes every moment. It is the mode that did.

(C) Motion priority mode This is a mode in which the frame rate is not reduced even when the number of motion blocks or the absolute value of motion increases. Here, the intensity of the motion is determined by an index AM (see Expression (10)) indicating the intensity of the motion in the description of the adaptive control according to the information generation amount in the section of [A-3] encoding control described later. By using the determination, the control is performed such that the transmission rate is maintained by driving the target area priority mode, the screen center priority mode, or a model base mode described later.

(D) Quasi-moving image (INTRA picture) mode In this mode, in order to reduce the amount of information, the transmission rate is maintained while maintaining the image quality by lowering the frame rate. In this case, compression using the correlation between frames may be difficult depending on the amount of motion. In such a case, only the INTRA picture is transmitted.

(E) Still image transmission mode This assumes the following two cases.
1) When the coding control fails In this case, first, the encoder 1 uses the image signal that has been subjected to picture freeze to transmit a new forced intra frame. After this is completed, the mode is shifted to the moving image transmission mode, and the moving image transmission is restarted.

2) When there is a retransmission request from the decoder 2 due to an error or the like, the picture is frozen on the decoder 2 side, and the encoder 1 transmits a forced intra frame in response to the retransmission request. This is a mode in which moving image transmission is resumed in the same manner as described above by shifting to the moving image transmission mode.
(F) Model-based prediction mode 1) Extraction of texture source The texture source is obtained from the analysis result of the moving area (in [A-3] extraction of the target area) to be described later for the first front image taken before the start of the call. This is performed based on the image information of the person region.

2) Model Selection and Adaptation A 3D (three-dimensional) model is defined in advance by giving depth information to each 2D template 69a (front image).
3) Model-based prediction and transmission distance information using a 2D template The distance information can be calculated from a table calculated using camera characteristics using the template number and scaling value of the 2D template 69a for the front image. For this front image, a model-based predicted image of a human image can be generated by using the distance information based on the region extraction result and the 2D movement amount on the image plane. Further, even when the radio wave condition is not good, 2D model-based image transmission can be performed.

4) Model-Based Prediction Using 3D Motion Information For example, a characteristic region of a 2D template 69a of a front image of a person is determined in advance as shown in FIG. 16, and the head and shoulders of the upper body template are determined based on the 2D motion information. The two parts below are regarded as rigid bodies, and a perspective transformation matrix representing a 3D position / posture can be obtained from a set of 2D positions of the N feature regions. Note that obtaining the perspective transformation matrix in this manner can be performed by using a well-known technique such as a spatial quantization method.

Further, in this case, such a thing may be stored in a table in advance, and such a position and orientation estimation for communication is performed by controlling a robot in a factory. This is based on the ground that there is a premise that there is no need to visually feel uncomfortable because higher accuracy is not required compared to position and orientation recognition.
5) Background Memory When the installation position of the camera 3 is fixed and the background image normally shot is fixed, for example, in a situation where the camera 3 is fixed in a car, the target area priority mode is set. The background memory use mode can be used.

6) Compensation of forced intra-frame transmission delay time based on model As shown in FIG. 17, in the case of a car telephone or the like, there is usually a session preparation time of typically 10 seconds or more before the start of a call. Therefore, using such a session preparation time, the front face is transmitted in a forced intra frame (about 30 k bits) at the start of the session (not at the start of the call). At the same time, analysis of the motion area, modification of the template, adaptation of the 3D model, and the like are completed by the start of the call.

  In addition, an approximate 3D motion vector is calculated based on the distance information L and the 2D motion information. Thus, the frame immediately before the start of the call is predicted on a model basis. Based on this, if the first frame is encoded in the moving image transmission mode after the start of the call, the code amount can be reduced as compared with the case where the first frame is transmitted as a forced intra frame (it is considered to be effective at least in the background area). .) Freezing, reduction in frame rate, and reduction in image quality can be significantly reduced.

(4) H. Compatibility with those of the H.261 standard In the encoder 1 in this embodiment, a bit stream of 64 kbps or less according to the protocol can be easily converted to H.264. For example, the protocol converters 70 and 71 shown in FIG. 18 can be configured to maintain compatibility.

In addition, H. At present, there is no means for uniform conversion from the bit stream of the image signal of the H.261 standard to the protocol in the decoder 2 of the present embodiment. Is connectable.
In the configuration of FIG. 18, for example, when an encoded bit stream transmitted from the encoder 1 is received, the protocol converter 70 adds a redundant code such as MBSTUFF to reduce the data amount of 9.6 kbps to H. . H.261 is converted to have a data amount of 64 kbps or more, and the syntax of the variable length code is changed to reduce the code amount. H.261 is performed by performing simple conversion control that can be performed at the code level, such as replacing with a code conforming to the H.261 standard. 261 encoder Sa.

On the other hand, in the protocol converter 71, as described above, there is no means for performing uniform conversion. H.261 decoder Sb. The output converted to an H.261 coded bit stream is converted so as to conform to a 9.6 kbps transmission rate by reducing the transmission code amount by adding syntax changes, mode information, etc., and reducing the frame rate. Output.
By doing so, H. 261 standard encoders and decoders Sb and the encoders and decoders 2 of the present embodiment can mutually convert image signals. Similarly, image communication can be made possible with an H.261 standard device.

(5) State determination and mode transition (a) Forced setting For example, in the above-mentioned encoding mode, mode switching between 1) texture source cutout mode and 2) model selection and adaptation mode is performed. Can be automatically switched depending on whether the target area extraction result is a person or not, but this can also be forcibly switched by a user's manual setting.

(B) Judgment of Category of Target Area In this judgment operation, the mode progresses in consideration of each condition according to a state transition diagram of mode control relating to the use environment and the subject as shown in FIG. .
1) Probabilistic state transition When the judgment information necessary for state transition is insufficient, autonomous state transition or transition using the transition probability table selected based on the probability given by default or incomplete information is performed. Perform a search. This is performed, for example, in the following cases.
A) When there is no forced mode given from outside
B) When area extraction based on moving area analysis has not been completed
C) When signal analysis results such as color signals have not been completed 2) Deterministic state transition based on determination information This is based on the probabilistic criterion, as shown in FIG. A moving area analysis for about a frame time is performed as described later, and after extracting a moving area and a motion occurrence state, a person area is extracted and a template is selected by extracting shape parameters. The conformity of the template is obtained by an evaluation calculation, and the process ends if the criteria are satisfied. If not, select the next template and repeat the process.

(C) Updating the transition probability value based on the propagation of the judgment result based on the state transition If a certain judgment result causes a significant error in the subsequent evaluation calculation, the process returns to the initial node of the moving object judgment via the moving area analysis module. At this time, the transition probability (in the figure, “0.9” is set on the person side and “0.1” is set on the other object side as the branch of the moving object n) is updated. For example, when an area determined to be a person based on a pure probability state transition in the initial determination outputs an evaluation function value that is unlikely to be a person due to subsequent 2D template comparison, color analysis results, speed / acceleration evaluation, etc. The transition probability for determining “likely” is changed and set so as to be reduced from “0.9” to about “0.2”.

(6) Human interface leading to mode selection Although the above-described state determination and mode transition enable optimal coding control, the mode control unit 33 reduces the probability of making an erroneous determination by guidance via the human interface 34. It can be drastically reduced. For example, if the following process is completed by the mode control unit 34 during the preparation time until the start of the session, it is possible to smoothly perform the subsequent mode transition related to the encoding control.
1) The user selects the person mode and looks at the position direction of the camera 3.
2) Via the human interface 34, a message "Please sit in front" is notified by voice or display.
3) When the person is out of the center of the screen, a message "Please slightly move to the right / left" is notified by voice or display via the human interface 34. (As a display method, a sub-screen may be provided at the corner of the screen to indicate the position.)
4) Transmit the first frame as forced intra.
5) Via the human interface 34, a message "Connected to the other party. Please speak" is notified by voice or display (calls can be started).
(7) Transmission of Mode Information The mode information and the accompanying template information have an information amount that does not hinder ultra-low-rate transmission, and can be transmitted. In this case, the transmission is performed in the user data area of the PIC layer. Is transmitted in a state described in PSPARE (see FIGS. 30 and 50).

[A-3] Extraction of target region A target region to be watched, such as a person, is extracted based on a motion vector and camera settings. Here, in order to reduce the amount of calculation, the calculation is performed on a macroblock (MBK) basis. The causal relationship between this operation and various modes is as shown in FIG. 19, as described above.

(1) Region Extraction Based on Generation of Motion Vector Here, the process of finally determining the optimum template is performed according to the flowchart of the routine for extracting a motion region shown in FIG. Also, reference is made to the explanatory diagram of the region analysis in FIG. 24 and the diagram of the basic template in FIG.
(A) Detection of maximum motion vector and estimation of main region In an encoding standard such as H.261, the presence or absence of generation of a motion vector is included as information in MTP (macroblock type) as data indicating the attribute of the MBK. In this case, H. According to the H.261 standard, when the MTP value indicates a value of 2, 3, 6, 8, or 10, it is data indicating that a motion vector is generated (see FIG. 33). Therefore, H. As the motion vector detection result of the H.261 coding, for example, an array of MBK attributes (MBA, MTP, CBP) as shown in FIGS. FIGS. 20 to 22 show MBA values and MTP values in portions corresponding to each MBK position on a screen in which 12 GOBs in 2 rows and 6 rows are arranged and 33 MBKs in 11 rows and 3 rows are arranged in each GOB. , CBP values.

  Here, an area including a block having the largest motion vector is defined as a main area. In the evaluation of the magnitude of the motion vector, a panning vector which is a motion component accompanying the movement of the camera 3 is subtracted in advance. There are various well-known calculation methods for the panning vector. For example, the panning vector can be obtained by calculating an average of motion vectors in a peripheral portion of the screen as a simple calculation method.

(B) Analysis in the Time Direction In the above case, it is difficult to identify the target object area by examining the block attribute array only for one frame. Therefore, as shown in FIGS. 24A and 24B, analysis in the time direction is performed for about three frame times (see also FIG. 19, step D1 in FIG. 23). In particular, regarding the region where the motion vector is generated, the center of gravity of the pattern (Step D2 in FIG. 23) obtained by superimposing three frames is obtained (Step D3). (Step D4).
A) When the distance from the center of gravity exceeds a certain threshold, two or more times three times. A) When the distance from the center of gravity is equal to or less than three times for blocks below a certain threshold value. Then, an isolated point region and a noise region are removed from the target region by the obtained motion vector (step D5), the center of gravity of the remaining target region is calculated again (step D6), and the height h of the target region and Find the width w. Thereafter, for example, if the head is a human head, the 2D template head is scaled by (h, w) to obtain the approximate distance L0 to the camera 3 (step D7). Thereafter, in order to perform template fitting with higher accuracy, the following is performed (step D8).

(C) Collation and Scaling of 2D Template First, a block pattern composed of motion blocks is determined, and a feature amount is extracted. Here, if the person mode is manually selected in advance, the basic template of the person mode can be selected from the 2D pattern database 27 as shown in FIG.

Here, as the basic template, a 2D template of a front image such as only a person's upper body, the whole body, or the head is prepared as a default pattern according to the mode predicted in advance in the decision tree (see FIG. 19). Next, the shape of the selected template is modified adaptively. This correction includes centroid alignment, scaling, and local stretching (see FIG. 24C). Here, in the selection and correction of the optimal template, the value of the evaluation function F shown in Expression (2) described later is calculated.
A) Start with a template L (<L0) that is slightly smaller than the template obtained in the analysis in the time direction.
B) Overlay a template on an image in which only a motion block is extracted, and calculate the number NMBK (L) of motion vector blocks included in the area.
C) Increase the scale (decrease the distance L).
D) If L becomes equal to or less than the fixed value Lmin, the process proceeds to the next step. Otherwise, return to a).
E) The point where the next evaluation function F takes the maximum value Fmax is defined as the optimum L.
F = −B × NMBK (L) (2)
Here, B is the second derivative of L of the curve of NMBK (L).
Or, if we translate this into a discrete representation,
F = −B (n) × NMBK (Ln) (2a)
B (n) = A (n) -A (n-1)
A (n) = NMBK (Ln) -NMBK (Ln-1)
It becomes.

(D) 2D Template Containing Quantization Scale Information Although a 2D template can be expressed in a binary form, as will be described later, each MBK attribute array value can be expressed in a template corresponding to a model in advance. This may be a quantization scale as shown in FIGS. 42 and 43, for example, as described later.
(E) Analysis of 2D Movement Pattern As shown in FIG. 25, a pattern in which a person moves from the screen is a horizontal and horizontal movement (corresponding to the figure arranged in the horizontal direction in the figure), and a movement in the camera optical axis direction. (Corresponding to the figures arranged vertically in the figure).

(2) Generation of 2D Template Based on 3D Model As shown in FIGS. 12 and 13, in the person mode in a limited environment, it is possible to create a 2D template equivalent to FIG. 25 by perspective transformation of the 3D model. it can. This makes it possible to acquire parts such as the upper body, head, eyes, mouth, and nose. However, other than the front image, it is necessary to obtain the position and orientation of the target person with respect to the camera 3, but the details will not be described here as they depend on a general method.

(3) Judgment of Category of Target Area (a) Judgment Based on State Transition Graph In the state transition diagram of mode control shown in FIG. 19, a judgment tree for specifying the category of the target area is included as a partial diagram. The transition between nodes on the decision tree is performed using two types of searches, the above-described probability search based on the default probability and the probability search based on the determination information.

(B) Judgment Using MBK Attribute Of the blocks that have changed, particularly the blocks that have undergone minute texture changes are expected to have a high CBP score value. This can be used as a basis for determining a partial area.
(4) Background Memory At the time when the forced intra frame is transmitted, the area extraction processing is always performed, whereby the processing of storing the information of the background part in the background memory 32 is performed in the encoder 1. Such processing is also performed in the background memory 66 of the decoder 2. This is to substitute the background information in the frame with the data stored in the background memory 32 in order not to increase the amount of data transmission when it is determined that the movement or color change has become severe.

  That is, as shown in FIGS. 26, 27, and 28, all background areas currently being encoded are set to FIXD, and use of the background memory is declared in the user data area. Based on this, the decoder 2 calls the latest background information from the background memory 66 and reproduces the image by overlaying it with the transmitted target area information such as a human image.

  FIG. 26 shows an outline of coding control for achieving a very low rate, and it is possible to extract a region by analyzing a motion region based on image signals of a plurality of frames to obtain a background region and a person region. For example, based on this, the quantization control for each region is performed and the number of significant coefficients is controlled. On the other hand, encoding control is performed according to the buffer amount while setting the encoding rate and adding mode information. In the case of extracting a person region, the region is specified in the unit of the MBK of the GOB in the CIF format as shown in FIG. 27A, so that the background region can be obtained similarly (FIG. 27B). reference).

  FIG. 28 shows a conceptual example in the case of using the data stored in the background memory 32. The data of the background area obtained by performing the area analysis on the encoder 1 side is stored in the background memory 32 (in the figure, the background memory 1). ), And during the transmission of the moving image in the moving image mode, data of a new background area is taken in and stored at any time. The decoder 2 also stores the data in the background area in the same manner. If the data transmission is interrupted due to the deterioration of the communication path during communication, the background memory is stored in the decoder 2. In addition to playing back the screen using the data of the background area stored in the memory, the retransmission request is transmitted to the encoder 1 side to supplement the image reproduction until the moving image transmission is restored.

[A-4] Encoding Control Operation This encoding control operation is schematically illustrated in FIG. FIG. It is a general one that shows the syntax of an encoded bit stream in the H.261 standard. In the following, details of the encoding control operation based on FIGS. 29 and 30 will be described for each item.

(1) Change of bit stream syntax The redundant header in the syntax of H.261 (see FIG. 30) is deleted in order to achieve a very low rate. For this reason, the arrangement of GOBs in the CIF structure is not changed. However, for 12 GOBs constituting a screen as shown in FIG. 31, GBSC (16 bits) and GOB number code (4 The amount of data is reduced by half by adding the number of bits to only the left GOB and eliminating the right half (see FIG. 32). This makes it possible to reduce 20 × 6 = 120 bits per frame.

(2) Adaptive switching of variable length coding (VLC) table (a) Replacement of codeword 26 shows a variable length code having an MTP (macro block type) attribute of the H.261 standard. For example, according to the experimental results of the inventor at an extremely low rate, the occurrence probabilities of the values of each MTP for a human image are in the order shown in FIG. Therefore, the number of bits for attribute description can be reduced by replacing the variable length code for the MTP value according to the order shown in FIG. This replacement pattern is limited to a few, for example, so that which one is to be selected is written in the user data area. It can be done flexibly.

In most experimental results, the MTP value is 2 or 3 in the object region, and in the case of the mode of the upper body of a person, the total of both reaches about 100 to 150. It has been found that the number of bits can be reduced by about 200 to 300 bits per frame by changing to 2 and 3 (currently 2 and 3).
(3) Adaptive control of attribute determination characteristics (a) Adaptive control of MTP determination using region information 1) For background region, FIX (fixed) or NOMC (no motion compensation) using region information it can.
2) Regarding the target area, it is determined that MC-CO (motion compensation and coding) is applied to a person, especially a face.

(B) Adaptive control of INTER / INTRA determination Normally, H.264 or higher kbps A characteristic curve of the INTER / INTRA determination recommended in the H.261 standard is as shown in FIG. In this case, PVAR indicates the sum of the inter-frame prediction error powers for the four Y blocks of the MBK, and QVAR indicates the value of the intra-frame variance of the input image for the four Y blocks of the MBK. Then, at a rate of 64 kbps or less, determination is made as follows.
1) In order to prevent the propagation of an error, a forced INTRA is periodically dispersed.
2) The threshold value TH_INTRA is set as follows, for example. The filling rate RB of the FIFO buffer 11 is obtained by the following equation (3), and the filling rate RB is determined based on the comparison reference values RB1 and RB2, and the threshold value TH_INTRA is set.
RB = BUFF / BUFF_MAX (3)

In this case, RB, BUFF, and BUFF_MAX indicate the current filling rate, the current number of bits used, and the maximum bit capacity for the FIFO buffer 11, respectively, and are determined separately for the following three cases.
<Case 1> 0 ≦ RB <RB1
<Case 2> RB1 ≦ RB <RB2
<Case 3> RB2 ≦ RB
The threshold value TH_INTRA is set as follows according to each case according to the determination result.

<Case 1> TH_INTRA = 64 × 256
<Case 2> TH_INTRA = 64 × 256 (main part of target area)
TH_INTRA = 64 × 512 (other than the main part of the target area)
TH_INTRA = 64 × 1024 (background area)
<Case 3> TH_INTRA = 64 × 256 (main part of target area)
TH_INTRA = 64 × 1024 (other than the main part of the target area)
TH_INTRA = 64 × 2048 (background area)
In the case described above, if the region information is not clear, as shown in the section of the image center priority mode or the target region centroid mode, a region whose distance from the center of gravity is within a certain range is defined as the target region. Then, the threshold value TH_INTRA is controlled as follows.

R_IR = IR_MBK / IRMBK_MEAN (4)
NTH = TH_INTRA × (R0 + R_IR) (5)
RPQ = PVAR / (QVAR × R_IR) (6)
Here, IR_MBK indicates the distance from the center of gravity of the target area to the MBK currently being encoded, and IRMBK_MEAN indicates the average distance from the center of gravity of the target area to all MBKs.

Then, for the value obtained from the above equation,
PVAR ≦ NTH or RPQ_VAR ≦ 1.00
Is determined as INTER, otherwise, determined as INTRA. In this case, for example, the value of R0 is set to 0.5.
(C) Judgment of presence / absence of motion compensation Normally, H.264 or more at 64 kbps FIG. 36 shows a characteristic curve for motion determination recommended in the H.261 standard. In FIG. 36, the horizontal axis indicates the value of the error sum FDSUM between frames without motion compensation, and the vertical axis indicates the value of the error sum MVSUM with motion compensation. In this case, in the drawing, the area where the motion compensation (MC_ON) is located is set to an area surrounded by parameters GD1 and GD2 indicating the slopes of two straight lines and parameters IEV1 and IEV2 indicating the threshold values for the two FDSUMs. ing. At a rate of 64 kbps or less, each value of the parameter set (GD1, GD2, IEV1, IEV2) of the characteristic curve is changed as follows depending on the buffer amount and area. Note that the buffer filling rate RB is expressed by the above equation (3).
RB = BUFF / BUFF_MAX (3)
In this case, RB, BUFF, and BUFF_MAX indicate the current filling rate, the current number of bits used, and the maximum bit capacity for the FIFO buffer 11, respectively.

<Case 1> 0 ≦ RB <RB1
In this case, the conventional motion determination is used for all areas. Therefore, each value of the parameter set of the determination characteristic is set as follows.
(GD1, GD2, IEV1, IEV2) = (0.50, 0.91, 256, 256 x 3)
<Case 2> RB1 ≦ RB <RB2
In this case, a normal motion determination characteristic is used inside a slightly larger rectangular area surrounding the target area (automatically set for each template in consideration of the motion range between frames) RM. Therefore, each value of the parameter set of the determination characteristic is set as follows.
(GD1, GD2, IEV1, IEV2) = (0.50, 0.91, 256, 256 x 3)
In addition, the threshold value of the motion determination is set high in an area QM other than the above (corresponding to the background area).
(GD1, GD2, IEV1, IEV2) = (0.40, 0.80, 256x2, 256x4)
<Case 3> RB2 ≦ RB ≦ RB3
In this case, in the main area of the RM,
(GD1, GD2, IEV1, IEV2) = (0.50, 0.91, 256, 256 x 3)
Outside of the main area of RM,
(GD1, GD2, IEV1, IEV2) = (0.40, 0.80, 256x2, 256x4)
In QM,
(GD1, GD2, IEV1, IEV2) = (0.20, 0.50, 256x4, 256x8)
<Case 4> RB3 <RB
In this case, select one of the following.
1) Move to the quasi-video mode and reduce the frame rate
2) Move to model base mode
3) Set to picture freeze mode

(4) Use of past coding attributes (a) Characteristics of each coding attribute of MBK layer The MBK attribute of each frame stored in the attribute memory 22 is expressed as 22 MB per frame when corresponding to the MBK position on the CIF. It becomes an array of × 18.
1) MBA (macro block address)
MBA is assigned as a significant encoded block in a portion corresponding to the target region from around the third frame. In particular, an MBK (macroblock) whose MBA indicates a value of 2 or more often corresponds to the outline of the target area. Therefore, as shown in the flowchart of FIG. 37 and FIGS. 38 and 39, the MBA pattern of one frame before is converted into a NOT_FIXED (NFX) pattern (steps E1 in FIG. 37, FIGS. 38A and 38B). ), The MBA array one frame ahead can be estimated and expressed using the average motion vector of the area information and the template information (see steps E2 to E6 and FIG. 39).

Then, the encoding is controlled using the estimated value of the MBA pattern, and the template information and the motion vector are encoded and transmitted (steps E7 and E8). On the other hand, on the decoder 2 side, the MBA array for one frame can be uniquely reproduced by the template information and the motion vector.
2) MTP (macro block type)
A) INTER / INTRA mixed (normal frame)
At very low rates, experiments have shown that most of the moving target areas (such as people) are labeled with MTP = 2,3. Therefore, based on the determination in the adaptive control according to the information generation amount described later, as shown in the flowchart of FIG. 40 and FIG. 41, if no particularly severe movement occurs, the average motion vector of the area information and the template information are used. Thus, the MTP array one frame ahead can be expressed (see steps F1 to F3 in FIG. 40 and FIG. 41).

Then, the encoding for one frame is advanced based on the predicted MTP value (step F4), and the template information and the motion vector for each area are encoded and transmitted (step F5). On the other hand, the decoder 2 can uniquely reproduce one frame of the MTP array based on the template information and the motion vector.
A) Forced intra-frame If the forced intra-frame is declared in user data, there is no need to describe data with MTP = 4 or data with MBA = 1 thereafter. Thus, this allows a saving of 5 × 22 × 18 = 1980 bits, or about 2 k bits.

3) QSC (quantization scale)
The QSC is fixed at 31 (quantization step 62) for all 32 kbps or less, and is set to 16 (quantization step 32) only for the main area determined from the area analysis. Therefore, QSC information is not transmitted even in a person mode of 16 kbps or less even at an extremely low rate. Therefore, the description of the QSC information in the GOB layer becomes unnecessary, and as a result, the information amount can be reduced by 12 × 5 = 60 bits. In addition, there is no need to transmit quantization scale change information in MBK units.

  In this case, FIG. 42A shows an example of quantization by region in the person mode, and FIG. 42B shows an example of quantization template. FIG. 43 shows an outline of data exchange between the encoder 1 and the decoder 2 when performing model-based transmission of a quantization template. In the flowchart of FIG. 9 shows a flow routine relating to the setting of the value of the optimization scale QSC.

4) MVD (difference motion vector)
Since MVD is a horizontal motion vector change in a frame, a value other than “0” occurs in non-rigid motion or rotational motion. For a person, a motion caused by a change in facial expression, a three-dimensional rotation of an edge portion of the head and upper body, and the like correspond to the motion. Since these occur instantaneously, the prediction efficiency when predicting in the form of a difference vector is not very good. Thus, such a difference vector is predicted as a motion vector in the form of a motion vector, and as an average 2D motion vector per area.

  Here, the MVD occurrence position is limited to the MBK declared to have motion compensation by MTP. Usually, most of this is represented by MTP = 2,3. This makes it possible to suppress the MVD information about once every two frames. Note that FIG. 45 shows a flowchart of a routine for reducing the code amount by the average motion vector for each region, and FIG. 46 shows the outline of the contents in a pattern transition diagram.

5) CBP (encoded block pattern)
The CBP indicates by bit information whether or not to encode each of the six BLKs (blocks) included in the MBK. Therefore, by converting the CBP value into a YUV vector, the CBP array one frame ahead can be expressed using the average motion vector of the area information and the template information. On the other hand, on the decoder 2 side, the CBP array for one frame can be uniquely reproduced by the template information and the motion vector. FIG. 47 shows a flowchart of a routine for motion compensation prediction for each area, and FIG. 48 shows an outline of the contents in a pattern transition diagram.

(B) Code amount reduction of MBK attribute using region information 1) Attribute encoding once per two frames As described above, motion compensation using template information and an average motion vector for each region is performed for one frame. The pattern attribute can be predicted, and can be uniquely reproduced on the decoder 2 side. Note that all the predicted pattern attributes can be reliably reflected on the encoding control not only on the decoder 2 but also on the encoder 1 side.

2) Upper / lower alternate prediction or interleaving Instead of inserting a pattern attribute of only prediction once in two frames, the upper and lower parts are set in GOB units as shown in FIG. 49 (a) (or FIG. 49 (b)). (Or the right side and the left side) can be alternately replaced with a prediction pattern. Interleaving for each GOB line (see FIGS. 3C to 3E) is also conceivable, but discontinuity may occur in the contour representing the pattern, so that it is not adopted when the target area is large.

(5) Quantization control Quantization control is basically performed by controlling the quantization step. Here, the method of setting the quantization step is described in H. It is not defined in the H.261 standard, and its constraint condition is that it is an even number from 2 to 62 (the quantization scale value is from 1 to 31) and a range that can be represented by 5 bits. Therefore, in this embodiment, the quantization control is performed by controlling the quantization step as described below.

(A) Quantization control for each area 1) Target area priority mode In this mode, a small quantization step is assigned to the extracted target area. The quantization step is fixed at 62 for the background area. When the subject is in the person mode, only the head region is set to 62 or less, and the other regions are set to 62 in principle (see FIG. 42A).

2) Screen center priority mode In this mode, the closer the screen is to the center of the screen, the finer the quantization step is taken. However, in order to unify the control formula of the quantization step, a method of correcting the current buffer capacity for the step calculation using the distance to the current MBK is adopted. Calculation is performed using 7) and (8).

R_IR = IR_MBK / IRMBK_MEAN (7)
BUF_R = BUFF_MB
× (5.00 + real (IR_MBK) / real (IRMBK_MEAN)) (8)
here,
BUFF_MB: Buffer amount monitored in MBK units BUFF_R: Virtual buffer amount based on distance calculation IR_MBK: Distance from target barycenter to MBK currently being encoded IRMBK_MEAN: Average distance from target barycenter to all MBKs This virtual correction buffer The value of the quantity BUFF_MB is used in a control equation according to the coding rate described later.

(B) Control According to Buffer Amount Normally, quantization control is performed in the above case, but quantization control based on the buffer amount is not performed in the case of forced intra-frame transmission. I have. Note that the forced intra frame is normally transmitted in the following cases.
1) First picture at the start or retransmission of the moving image mode 2) Quasi moving image mode 3) Still image mode (picture freeze)
4) Texture source image in model-based mode The quantization step depends on a control formula according to the coding rate described later.

(C) Control According to Coding Rate The determination formula of the quantization step (STEP FPIC) according to the coding rate (RATE) is set as follows.
1) Quantization step of forced intra frame
When 1152kbps <RATE → STEP_FPIC = 12.0,
When 384kbps <RATE <1152kbps → STEP_FPIC = 14.0,
When 64kbps <RATE <384kbps → STEP_FPIC = 16.0,
When RATE <64kbps → STEP_FPIC = 32.0
2) Normal quantization step ISTEP = 2 × INT (BUFF_MB / (200.0 × QX64) +2 (9)
BUFF_MB: Current amount of data in the buffer QX64: Value that satisfies the coding rate = QX64 × 64.00 [kbps] When the coding rate is 16 kbps or less, frequent change of the quantization scale means that This leads to an increase in the required number of bits. Therefore, in the case of 10 kbps or less, the quantization step is fixed at 62.

(D) Adaptive control according to the amount of information generation Quantization and frame rate control are performed based on the amount of motion and the degree of color change.
1) Judgment of Intensity of Motion As a degree of intensity of motion of a current frame with respect to a past frame, a value of an index called AM defined by the following equation (10) is obtained by calculation, and a result of determination based on this value is obtained. Performs quantization and frame control.

ただし、
Ｎｍｂ；動きの発生したブロックの数
Ｌ（Ｘ）；ベクトルＸのノルム関数．絶対距離，ユークリッド距離など
Ｖｉ；動きベクトル
Ｒｄ；伝送データレート
ＴＨＶ（Ｒｄ）；データレートに依存したしきい値定数
式（１０）で計算されるＡＭの値を用いて、新たに尺度ＡＭＴを計算する。この場合において、ＡＭＴは次のようにして計算される。

However,
Nmb; number of blocks where motion has occurred L (X); norm function of vector X. Absolute distance, Euclidean distance, etc. Vi; motion vector Rd; transmission data rate THV (Rd); threshold constant depending on data rate A new measure AMT is calculated using the value of AM calculated by equation (10) I do. In this case, AMT is calculated as follows.

A) When AM ≦ THV (Rd), AMT = 0
B) When AM> THV (Rd), AMT = AM
Here, the target range of Nmb and the corresponding THV are changed as follows according to the calculation capability of the encoding processor.
a) From the first MBK of the current frame to the MBK currently being decoded
b) From the first MBK in the current GOB to the MBK currently being decoded
c) all MBKs in the current GOB
d) all MBKs in the current frame
In the above cases a) and b), since a global operation is unnecessary, the amount of calculation is small and there is no processing delay, but the reliability of determination is low. On the other hand, since c) and d) perform global calculations, the amount of calculation increases, but the processing delay is a maximum of one frame time. However, the reliability of the judgment is high.

2) Judgment of intensity of color change As a degree of intensity of color change of the current frame with respect to the past frame, an index value AC defined by the following equation (11) is obtained by calculation, and judgment is made based on this value. Based on the result, quantization and frame control are performed.

ただし、
Ｎｃｂ：ＣＢＰのブロック属性が１になったブロックの数
Ｃ（ｉ）：ｉ番目のマクロブロックに関してＤＣＴ係数のＤＣ成分の変化とＣＢＰに基づいてＹＵＶベクトルから色変化を計算する関数
ＴＨＣ（Ｒｄ）：データレートに依存したしきい値定数
式（１１）で計算されるＡＣの値を用いて、新たに尺度ＡＣＴを計算する。この場合において、ＡＣＴは次のようにして計算される。

However,
Ncb: Number of blocks whose CBP block attribute has become 1 C (i): Function for calculating color change from YUV vector based on change of DC component of DCT coefficient and CBP for i-th macroblock THC (Rd) : Threshold constant depending on data rate A new measure ACT is calculated using the value of AC calculated by equation (11). In this case, ACT is calculated as follows.

A) When AC ≦ THC (Rd), ACT = 0
B) When AC> THC (Rd), ACT = AC
Here, the target range of Ncb and the corresponding THC are changed as follows according to the calculation capability of the encoding processor.
a) From the first MBK of the current frame to the MBK currently being decoded
b) From the first MBK in the current GOB to the MBK currently being decoded
c) all MBKs in the current GOB
d) all MBKs in the current frame
In the above cases a) and b), since a global operation is unnecessary, the amount of calculation is small and there is no processing delay, but the reliability of determination is low. On the other hand, since c) and d) perform global calculations, the amount of calculation increases, but the processing delay is a maximum of one frame time. However, the reliability of the judgment is high.

3) Calculation of virtual buffer
B) Virtual buffer increment based on the amount of movement
a) MBK of the target area without motion: BUF_M = 16 × (AMT / aM)
b) MBK of the moving target area: BUF_M = 0
c) MBK in the background area: BUF_M = 32 × (AMT / aM)
aM is a number corresponding to the average amount of motion per MBK, for example, aM = 16.
B) Virtual buffer increment based on color change
a) MBK of target area without color change: BUF_c = BMBK × (ACT / aC)
b) MBK of target area with color change: BUF_c = 0
c) MBK of background area: BUF_c = 2 × BMBK × (ACT / aC)
aC: a number corresponding to an average color change per MBK, for example, aC = 128
BMBK: Estimated average code amount per MBK, given by
BMBK = QX64 × 64000 / (Frate × NMBK)
Frate: current frame rate
NMBK: Number of MBK in one frame

(6) Control of the number of significant coefficients In step 261, the DCT transform coefficients after the quantization transform are zigzag-scanned in block units, and the obtained one-dimensional quantized coefficient sequence is represented by a non-zero level followed by a binary set of zero run length (called an event). expressing. Here, it is assumed that the coefficient of the high-frequency component does not contribute much visually at an extremely low rate, and by limiting the number of events per block, the number of corresponding VLCs is reduced, thereby reducing the number of bits as a whole. Can be planned.

  That is, when the DCT significant coefficients (non-zero) sequentially obtained from the low frequency components by the zigzag scan exceed a certain number, control is performed so that all the remaining DCT coefficients are forcibly regarded as zero. At this time, control is performed so that the upper limit number Ncf (≦ 64) as the threshold value is switched according to each of the coding rate, the area, the amount of motion generation, the buffer amount, and the coding mode. The information on the upper limit number does not need to be sent to the decoder 2 side, and is therefore not encoded.

The above-described control of the number of DCT significant coefficients is actually performed as follows. Here, for example, the following state is assumed.
Coding mode: person mode Coding rate: 8 kbps
RB = V_BUFF / BUFF_MAX
V_BUFF = BUF_R + BUF_M + BUF_C
BUFF: current buffer amount BUFF_MAX: maximum buffer capacity (RB1, RB2, RB3, RB4, RB5) = (0.2, 0.3, 0.5, 0.8, 1.0)
(Ncf0, Ncf1) = (16, 8)
In the determination, the control is divided into the following six cases according to the value of the buffer filling rate BF. The values indicated by RB1 to RB5 are threshold values for determination, and values corresponding to the control contents are set in advance.

<Case 1> 0 ≦ RB <RB1
The maximum number of significant coefficients is 64 for all areas. <Case 2> RB1 ≦ RB <RB2
The maximum number of significant coefficients is 64 for the target area and Ncf0 for the background area. <Case 3> RB2 ≦ RB <RB3
The maximum number of significant coefficients is Ncf0 for all regions. <Case 4> RB3 ≦ RB <RB4
The maximum number of significant coefficients is Ncf1 for all regions. <Case 5> RB4 ≦ RB <RB5
The background uses a background memory, and the portion not in the memory is represented only by the DC component. The maximum number of significant coefficients is Ncf1 for the target area. <Case 6> RB5 <RB
Select one of the following <1> to <3> depending on other conditions etc. <1> Shift to quasi-moving image mode <2> Shift to model base mode <3> Picture freeze (7) Frame rate Adaptive Switching (a) Description of frame rate change instruction In the bit stream syntax of the H.261 standard (see FIG. 30), an instruction to change the frame rate to the decoder 2 can be described by setting the value of TREF (temporal reference) of the PIC layer. However, in this embodiment, the change of the frame rate is treated as a subordinate means for making the rate extremely low. The following describes the method and the factors of implementation.

(B) Method of Changing Frame Rate After the moving image is A / D-converted on the encoder 1 side, the frame is thinned out by selecting whether to send or not send the raw image data to the encoding loop in frame units. The rate changes. Therefore, this thinning information is reflected in the above TREF.
(C) Factors for driving frame rate change The driving factors for changing the frame rate can be summarized as follows.
1) Switching according to buffer capacity 2) Switching according to transmission rate (Example: 8kbps → 5frame / sec, etc.)
In the moving image mode, the initial frame rate is set according to the transmission rate. For example, the coding rate QX64 is set to the following frame rate.
・ QX64 ≧ 18 → 30 frame / sec
・ 18 ≧ QX64 ≧ 10 → 30 frame / sec or 15 frame / sec
・ 10> QX64 ≧ 6 → 15 frame / sec
・ 6> QX64 ≧ 1 → 15 frame / sec or 10 frame / sec
・ 64> QX64 × 64 ≧ 32 → 10 ～ 7frame / sec
・ 32> QX64 × 64 → 10 frames / sec or less 3) Switching according to the amount of motion generation 4) Changing the mode

[A-5] Description of Operation According to Encoding Processing Program As described above, the functions of the ultra-low rate implemented in the encoding processing are illustrated in the actual encoding processing. The program is executed according to the flowcharts of the programs shown in FIGS. Hereinafter, an outline of the entire flow will be described.

  That is, first, the forced mode is set (step A1), and under this setting state, the target area extracting unit 26, the template database 27, the 2D template matching unit 28, the model base predicting unit 29, the three-dimensional shape database 30, and the like. To analyze the moving area and extract the moving area (steps A2 and A3). Next, the mode control unit 33 performs search and determination based on the state transition diagram, and subsequently sets the use environment mode, sets the subject mode, updates the state transition probability, and determines the encoding control mode (steps A4 to A4). A8) is sequentially executed.

  Thereafter, it is determined whether or not the current frame is a forced intra frame (step A9). If “YES”, the quantization step in the encoding control unit 21 is determined, and the orthogonal transform unit 7, H.264 is performed by the quantization unit 8, the channel coding unit 10, and the like. The INTRA encoding process of the H.261 standard is performed (steps A10 and A11). If "NO", the process directly proceeds to the next virtual buffer calculation (step A12), and the encoding control unit 21 calculates the virtual buffer.

  Next, if the current frame is in the moving image mode, the frame rate is determined (steps A13 and A14). If the current frame is in the model base mode, the region extraction / recognition processing unit 25 converts the 2D template based on the 3D model. Generation is performed (steps A15 and A16), and a 2D template is collated to perform a target area extraction process (steps A17 and A18). When the background memory mode is set, the background memory 32 is used (steps A19 and A20).

  Subsequently, it is determined whether or not the frame is an attribute prediction frame (step A21). If “NO”, a series of processes of steps A22 to A28 is performed. A series of processing of A35 is executed. In steps A22 to A28, the encoding control unit 21 and the target area extracting unit 26 determine the intra frame, determine the presence or absence of motion compensation, determine the motion amount, determine the color change amount, calculate the virtual buffer, and perform the quantization step. Determination and calculation of the maximum number of coefficients are executed. In steps A29 to A35, the MTP, MBA, QSC, MVD, and CBP arrays are predicted by the encoding control unit 21 and the attribute memory 22, and the like, the MBK attribute encoding suppression process, the MBK attribute array Model-based transmission is performed.

  Thereafter, H.K. 261-based motion compensation and DCT calculation are executed (step A36), and the orthogonal transform unit 7, the quantizing unit 9, the inverse quantizing unit 13, the inverse transforming unit 14, the channel encoding unit 10 and the like quantize each region. After performing the control and the control of the number of significant coefficients, a bit stream of the BLK layer is generated (steps A37, A38, A39). After the end of the BLK layer, if the frame is an attribute prediction frame, MBK attribute determination control is executed (steps A40 to A42), and thereafter, the MBK attribute data is stored and stored in the attribute memory 22 (step A43). ). Hereinafter, encoded bit stream data for one frame is generated according to steps A44 to A54.

[B] Decoder Next, the content of the decoding process of the encoded bit stream received by the decoder 2 will be described. FIG. 4 conceptually shows the contents of the decoding process. When roughly classified, the decoding process is performed in three stages of a code level, a pattern level, and an image signal level. In this process, various mode controls are performed. In addition, the error correction function is implemented to implement a decoding process based on ultra-low rate transmission and an error correction function corresponding to an error rate at a wireless communication level. In the following, a detailed description corresponding to each function will be made item by item based on the conceptual configuration shown in FIG.

A brief description will be given of how the decoder 2 realizes the function of autonomous error correction, which is a feature of the decoder 2. That is, normally, the average bit error rate is about 10 −6 to −8 in a wired transmission system, whereas a large value of about 10 −2 to −3 is assumed in a wireless system. I have.
On the other hand, MPEG and H.264. In a bit stream generated by a syntax based on a variable-length code like the H.261 standard (see FIG. 30), even in the worst case, even a single-bit error propagates a decoding error, and the subsequent data cannot be decoded. It is assumed that However, in a conventional wired TV conference system, an error correction of a practically acceptable level has been realized by a combination of a demand refresh (a combination of a freeze and a retransmission request) and an error correction code (such as a BHC code).

  In a storage system such as a video CD, an error correction code is sufficient, but in a wireless system, an error rate is high, and depending on radio wave conditions, transmission errors such as loss, inversion, and insertion may occur in units of several bits. Therefore, complete error correction is difficult only with a normal code-theoretic approach. Therefore, in the decoder 2 of this embodiment, a solution is attained by taking at least an autonomous error recovery means capable of restoring a signal value as close as possible to the original signal without interrupting the decoding process. It is.

  In the following, the content of the autonomous error correction function will be described separately for each of the following items. [B-1] Global check of coded bit stream, [B-2] Error correction of code and syntax levels, [B-3] Error correction of pattern level, [B-4] Error correction of signal level , [B-5] Strategy control of error correction using mode information, [B-6] Error correction of recognition level, [B-7] Person memory and 3D model-based method, [B-8] Error correction strategy, [B-9] The operation according to the decryption processing program is divided into large items, and each principle is further classified into small items to explain each principle and method.

[B-1] Global Check of Encoded Bit Stream (1) Form of Bit Error First, when correcting a bit error in a decoding process, which bit error of an image signal received via a communication path is The mode of occurrence will be briefly described below.
(A) One-time "0/1" inversion error: An error occurrence mode in which bit values are randomly inverted at a certain probability. (B) Burst error: An error occurrence mode in which data in a certain section is masked. , The value between them is either of the following << 1 >> or << 2 >>. << 1 >> Continuously output a fixed value of 0 or 1 << 2 >> Output a completely random value (c) Insertion and deletion of bits: This causes temporal expansion and contraction

(2) Basic Policy of Error Correction In the present embodiment, as a basic policy of error correction, the form of error occurrence is not particularly limited. However, although the above case (a) can be dealt with relatively stably, in the cases (b) and (c), it is extremely difficult to completely correct it. Is restored with an estimated value that is not so visually strange, but since accumulation of errors due to subsequent error propagation is expected, means for suppressing this is provided. In the worst case, a picture freeze and a retransmission request (ARQ: automatic repeat request) are exercised.

Also, before exercising the active correcting means on the decoder 2 side, information (data such as the required number of bits) having a large influence on the decoding process is applied to the encoder 1 as described above. The correction capability is improved by adding redundantly on the side.
(3) Transmission of Required Number of Bits Using the user data area in the protocol syntax, data having a large effect on image reproduction when data is lost is redundantly transmitted from the encoder 1 as described above. I have. In the present embodiment, as shown in FIG. 50, the required number of bits per PIC in the PIC layer and the required number of bits per GOB in the GOB layer are approximately 16 bits (up to 64 kbits) in each user data area. It is described in. As a result, it is possible to determine whether bits are missing or inserted on the decoder 2 side. Whether or not the required bit number data itself has an error is determined as follows.

(A) As described in the following section [B-2] Error correction of code and syntax levels, localization of PSC (picture start code) and GBSC (GOB start code) is performed by global header check in the buffer. I will go. Then, based on the result, the code amount described in PSPARE or GSPARE is checked.
(B) If the sum of the code amount descriptions of the GOB layer does not match the code amount of the PIC layer, it is determined that any of the descriptions has an error. If they match, it is determined that there is no error, and the process ends.

(C) The average code amount per frame and the average code amount per GOB are calculated from the transmission rate and the frame rate. In the case other than the forced intra frame, a code amount deviating from this value by a certain threshold or more is set as a candidate for a description error portion.
(D) For the candidate extracted in the above item (b), a comparison is made with the linear prediction based on the code amount over several past frames, and if the deviation from the value is not within a certain range, an error is detected. judge.

(4) Forward type data check In order to detect all data of a required number of bits in one frame described by the encoder 1 as described in (a) above before the start of decoding processing of one frame. The bit stream data stored in the input buffer (FIFO buffer 42) of the decoder 2 is checked globally in the order of several k bits. For that purpose, as described in the following section [B-2], localization of PSC and GBSC is required. Therefore, the localization processing is performed as follows.

    (A) A certain buffer area is completely scanned by fuzzy matching processing as shown in FIG. When the transmission rate is high, a lot of matching calculations are required to complete the localization by this method. However, at an extremely low rate, the amount of data per frame is several kbits or less. There is no practical problem even if the process of scanning globally is performed.

  In this case, in the fuzzy matching processing, the similarity S (refer to the equation in FIG. 51) indicated by the bit match rate between the bit pattern of the code to be searched and the bit pattern in the encoded bit stream is, for example, 0.9 (90%). ) If it is above, there is a criterion to judge that there is almost no mistake. Thus, for example, since the PSC has 20 bits, it is possible to determine that the matching has been performed while allowing an error of 2 bits. The value of the similarity S is preferably 0.9, but can be implemented even if set to a value as low as about 0.8 in practical use.

    (B) After searching for the PSC and the first GBSC, a rough position is found based on the estimated code amount using the average code amount and the code amount one frame before. Thereafter, scanning is performed within a certain range centered on the position. Then, if the number of GBSCs up to the PSC of the next frame is correct, the check processing ends. If not correct, the value of the GN (group number) code is checked to determine the missing GBSC, scanning is performed from the GBSC immediately preceding the GBSC, and the process is terminated if it can be detected.

[B-2] Level Error Correction of Code and Syntax (1) Localization of PIC and GBSC If only a bit stream is decoded sequentially, PIC data and There is a possibility that the range of GOB data cannot be known. Therefore, a hierarchical process in which detection is performed relatively stably by the above-described fuzzy matching process and error correction based on a pattern level (MBK attribute) and an image signal level (BLK) is started from a PSC and a GOB header that can be localized. Is provided. After that, it is linked to the following error correction depending on the code and the grammar.

(2) Form of occurrence of grammatical decoding stop due to bit error When a bit stream containing a bit error is sequentially decoded, the parser 43 stops the decoding process due to a grammatical error. However, the stop position at that time does not always coincide with the bit error occurrence position, and in some cases, the bit error position may go back several tens of bits before the stop position. Therefore, the following describes what error occurrence modes exist.

(A) When Stopping Immediately at a Bit Error Position When a bit error is mixed in a fixed-length header and matching becomes impossible, decoding is immediately stopped unless there is a branch condition to another in the syntax.
(B) When decoding is continued for a while after the bit error position and then stopped 1) When a bit error occurs in variable length code (VLC) The bit stream has a different binary tree structure as shown in FIG. This is a time series of output symbols obtained by transiting between nodes in a graph obtained by combining code trees based on logical conditions and numerical conditions. Therefore, if even one bit in the VLC is inverted, the time series of the event (corresponding to the encoding attribute and the value of the image signal) occurring on the encoder 1 side cannot be reproduced on the decoder 2 side, and a completely different event occurs. There is a possibility that the result will be output as a decoding result of the column.

  However, such an error cannot be detected only by a grammatical constraint that determines only a code. That is, it is detected only by a grammatical constraint via an error detection condition at a signal level or a pattern level described later or a numerical condition based on an image data format. In other words, the erroneous bit stream decoding is continued until such an error detection process is performed, so that the decoding stop position is located after the bit error position.

  In addition, even if such an error exists, decoding does not always stop, and for example, if bit inversion is replaced with another codeword of the same code length, the state output is Only a different value is output, and if the value does not adversely affect the subsequent conditional branch, no synchronization deviation occurs in the subsequent VLC decoding processing. Therefore, in this case, even if there is a possibility that only the color or texture data of a certain pixel block is changed and reproduced, decoding does not stop.

2) When Bit Error Occurs in Fixed-Length Code In this case, the same as the above-described case of bit inversion where the code length is invariable, and the value or attribute of the decoded output is different from that at the time of encoding, but the subsequent conditional branching The decoding is not stopped unless it has an adverse effect on the decoding.

(3) Detection of grammatical decoding error Based on the protocol syntax of H.261 (see FIG. 30), the positions where bit errors occur are classified and described as follows.
(A) Fixed-length header that always appears 1) PSC (picture start code; 20 bits)
As long as the fuzzy matching process is performed, even if a bit error of about 2 bits occurs in the PSC, the PSC can be detected without depending on the syntax or the decoding result up to that time. Therefore, PSC detection and localization is an initial process required to detect bit errors at other locations.
2) GBSC (group of blocks start code; 16 bits)
Like the PSC, it can be stably detected by the fuzzy matching process. However, if the PSC has not been localized, the localization may be erroneous.

(B) Fixed-length data that always appears 1) TR (temporal reference; 5 bits)
If the localization of the PSC has been performed, it is easy to check the value because it is the following 5-bit data. Error determination differs depending on the mode setting state of the decoder 2 as follows.
A) In the normal moving image playback mode (fixed frame rate), the value should be increased from the previous TR by a value corresponding to the frame rate corresponding to the transmission rate. In the case of the ultra-low-rate moving image mode of 32 kbps or less, the increment is about 3 to 5, and other than that, it can be determined as an error.
B) In the case of the quasi-moving image mode, the increment is about 10 to 31.
2) PEI (picture extra insertion information; 1 bit)
If the data is 1, the following user data PSPARE (8 bits) exists. If the data is 0, GBSC follows.
3) GN (group number; 4 bits)
An error is determined by the GBSC localization process. It is an error if the following conditions << 1 >> and << 2 >> are not satisfied. << 1 >> In the CIF structure, 1 ≦ GN ≦ 12, << 2 >> When the numerical value is increased by one from the previous GN. 4) GQUANT (GOB layer quantizer information; 5 bits)
The quantization scale (QSC) in ultra-low-rate transmission is fixed to 31 (quantization step 62) in the target area priority mode, and is set to 16 (quantization step 32) only for the main area determined from the area analysis. I do. Therefore, QSC information is not transmitted in the person mode. As a result, GQUANT (QSC information of the GOB layer) and MQUANT (quantization scale change information in MBK units) are not required, and no error occurs in this data. In other modes, when MQUANT is used, an error is determined by estimating a value in the same manner as CBP in pattern-level error correction described later.

5) GEI (GOB extra insertion information; 1 bit)
A) If the data is "1", then user data GSPARE (8 bits) follows. Therefore, it is determined that the localization of the GBSC and the definition and numerical value of the GSPARE are incorrect (synchronization loss or error).
B) If the data is "0", MBA continues. Therefore, the determination of the error is carried over to the MBK layer.

(C) Fixed-length data / header that appears conditionally 1) PSPARE (picture layer spare information; 8 bits)
An area in which user data can be described in units of picture layers. In this embodiment, the area is used for mode information (8 bits) and information on the required number of bits (8 bits × 2). This makes it possible to determine an error in the required bit amount.

2) GSPARE (GOB layer spare infomation; 8 bits)
An area in which user data can be described in GOB layer units. In this embodiment, the required number of bits in GOB is described. Since the required number of bits in the GOB layer is likely to be within 8 bits, the bit pattern of the data of the required number of bits is described as an inverted pattern in which the order is switched between the upper (MSB) side and the lower (LSB) side. It is supposed to. Therefore, the next GSPARE is required only when the required number of bits of the GOB layer exceeds 8 bits.

3) MBSTUFF (macroblock address stuffing; 11 bits)
Not used for very low rate transmission. H. 64 kbps or higher. Although it may be used in the H.261 standard, as shown in FIG. 53, even if an error of one bit occurs, the interpretation result may be grammatically different, so that it is dangerous to perform fuzzy matching processing. . Therefore, the MBSTUFF code is not used in this embodiment.

4) MQUANT (MBK layer quantizer information; 5 bits)
As described in GQUANT, MQUANT is not used in the person mode of ultra-low-rate transmission in the present embodiment. When MQUANT is used in another mode, an error is determined by estimating a value in the same manner as CBP in pattern-level error correction described later.
5) INTRA-DC (8 bits)
Grammatically, it appears as the first DCT coefficient data only when MTP indicates INTRA. This determination is left to a signal level error correction process described later.

6) ESC (6) + RUN (6) + LEVEL (8 bits)
If an error occurs in the ESC, another interpretation occurs in the subsequent decoding process of the BLK layer, so that it is quite difficult to identify and correct the error position grammatically. Therefore, the following measures have been taken.
A) In the decoding process of the DCT coefficient, the similarity Sesc with the ESC is always calculated, and when Sesc = 5/6 (5 bits out of 6 bits match the ESC), all the positions Pesc are stored, and the subsequent decoding is performed. Proceed with the process.

B) If the following conditions << 1 >> and << 2 >> are not satisfied before the EOB is found, it is determined that an error has occurred, the process returns to Pesc, which is interpreted as ESC, and decoding is performed again. << 1 >> Constraint of number of significant coefficients ≦ Ncoef, << 2 >> Cumulative number of quantized DCT coefficients in BLK ≦ 64
C) If an error is detected in the error correction process (BLK layer) at the image signal level, the process returns to Pesc, interprets it as ESC, and performs decoding again.

D) If an error is detected in the pattern-level error correction process (MBK layer), the process returns to Pesc, re-interprets as ESC, and performs decoding again.
7) EOB (end of block; 2 bits)
Since the word length is short, it is difficult to determine a candidate based on the similarity. However, since the number of appearances is large, the probability that a random error occurs in the EOB is not small. Therefore, if the following conditions << 1 >> to << 3 >> are not satisfied, it is determined that an error has occurred in the EOB.

<< 1 >> Number of significant coefficients ≦ Ncoef constraint << 2 >> Cumulative number of quantized DCT coefficients in BLK ≦ 64
<< 3 >> (number of BLKs in MBK) ≦ (number of BLKs described in CBP)
In this case, there are two correction methods, i.e., a) and i), but usually, the method i) is selected to simplify the calculation.

A) Bits are sequentially inverted immediately after the immediately preceding EOB, and an EOB pattern "10" is detected. Then, the detected EOB pattern is regarded as an EOB, and a decoding process is performed. If all the above three conditions << 1 >>, << 2 >>, << 3 >> are satisfied, this is determined to be a correct EOB.
B) Apply pattern-level error correction to all remaining MBKs in the GOB. If the above three conditions << 1 >>, << 2 >> and << 3 >> cannot be detected, the error correction is left to the signal level or pattern level error correction.

(D) Variable length code (VLC) that always appears
1) MBA (macroblock address; 1 to 11 bits)
Since the MBA appears under the following conditions << 1 >> and << 2 >>, the VLC table is collated under these conditions, and the rest is left to pattern level error correction.
<< 1 >> Immediately after EOB when the number of decoded MBKs ≦ 32 << 2 >> Immediately after GEI = 0 2) MTP (macroblock type; 1 to 10 bits)
Since the MTP appears under the condition "immediately after MBA", the VLC table is collated under this condition, and the rest is left to pattern level error correction.

(E) Variable length code (VLC) that appears conditionally
1) MVD (motion vector data; 1 to 11 bits)
Since the MVD appears under the following conditions << 1 >> and << 2 >>, the VLC table is collated under these conditions, and the rest is left to pattern level error correction.
<< 1 >> Immediately after MTP when MTP = 2,3,6,8,9,10 << 2 >> MVD shows two VLCs continuously appearing in the order of x component and y component 2) CBP (coded block pattern; 3 to 9 bits)
Since CBP appears under the following conditions, the VLC table is collated under these conditions, and the rest is left to pattern level error correction.

3) TCOEF (INTER) (transform coefficients; 2 to 17)
The VLC of the DCT coefficient for the INTER block of the BLK layer appears under the following conditions << 1 >> and << 2 >>. This error correction is left to signal level error correction.
<< 1 >> When MTP is 4 or 7 (INTRA block), after the second coefficient in the BLK layer << 2 >> When MTP is other than 4 or 7, the next code is not ESC in the BLK layer

[B-3] Pattern Level Error Correction The continuity between frames in the hierarchical coding attribute is used.
(1) Use of past MBK decoding attribute For the past MBK layer that has already been decoded, five attribute data of MBA, MTP, QSC, MVD, and CBP are stored in the attribute memory 39c as array data in frame units. ing. Since such attribute data has considerable continuity between frames, it has the property that it does not change so drastically except for scene changes and forced intra frames. This is particularly true when there is temporal continuity regarding the continuation of the target area such as a person area (see FIGS. 54 and 55).

  When a signal level error is detected in units of block pixels, if the error is not caused by a VLC transmission error with respect to the DCT coefficient, the error is likely to be an MBK attribute error. At this time, if the MTP, QSC, and CBP are significantly different from each other within the range of k MBKs adjacent to the motion compensation position in MBK units of the attribute of the previous frame, if the attribute is past attribute, Error correction is performed by applying a value. The evaluation function for comparison in this case is as described below.

(A) MBA Error Detection and Correction First, although MBA has temporal continuity at the contour on the left side of the target area, it is expected that the change will increase in the target area. However, conversely, the FIXED / NOT_FIXED pattern can be almost predicted from the movement of the area. Therefore, as shown in the flowchart of FIG. 56 and FIGS. 57 and 58, in the following, the area is shifted using the average motion vector of the area, so that NOT_FIXED (hereinafter, NFX) uniquely corresponding to the MBA pattern is used. Prediction calculation of a pattern is performed, and the similarity SNFX between the NFX pattern based on the current decoding result and the NFX pattern based on the prediction result is calculated in GOB units according to the following equations (12) and (13).

ここで、
L ；現在復号中のＭＢＡ値ａのＭＢＫアドレス（既に復号が完了した１つ前のＭＢＡ値のＭＢＫアドレスL0にａを加えた値）
Ls； NFXパターン比較の開始位置
ｓ(A,B) ； A=B のとき「１」，それ以外は「０」
NFXM(k) ；ＧＯＢの１番目のＭＢＫのＭＢＡ値が「１」以上ならば「１」，「０」なら「０」
NFXM＿(k) ；１フレーム前の NFXパターンから予測した NFXパターン
次に、上述の計算結果について信頼度ＲNFX を式（１４），（１５）にしたがって計算する。

here,
L: MBK address of MBA value a currently being decoded (a value obtained by adding a to MBK address L0 of the immediately preceding MBA value already decoded)
Ls; NFX pattern comparison start position s (A, B); "1" when A = B, "0" otherwise
NFXM (k); "1" if the MBA value of the first MBK of GOB is "1" or more, "0" if "0"
NFXM_ (k); NFX pattern predicted from the NFX pattern one frame before Next, the reliability RNFX is calculated according to the equations (14) and (15) for the above calculation result.

上述の結果に基づいて、例えば次のような４つの条件によりＭＢＡの検出と誤り訂正を行う。
１）ＲNFX0 ＜ ０．５
この場合には、ＮＦＸ予測の信頼度が低いと判断して保留する。すなわち、とりあえずは現状のＮＦＸパターンを正しいと判定し、次の属性の判定に進む。

Based on the above results, for example, MBA detection and error correction are performed under the following four conditions.
1) RNFX0 <0.5
In this case, it is determined that the reliability of the NFX prediction is low, and is suspended. That is, for the time being, it is determined that the current NFX pattern is correct, and the process proceeds to determination of the next attribute.

2) RNFX0 ≧ 0.5 and SNFX <TNFX1
It is determined that the current NFX pattern is a decoding error. The NFX value is copied from the predicted pattern and converted into an MBA pattern. (TNFX1 is, for example, about 0.3)
3) RNFX0 ≥ 0.5 and TNFX1 ≤ SNFX <TNFX2
Since it cannot be determined that the current MBA value is a decoding error, it is suspended. That is, it is determined that the NFX value is correct for the time being, and the process proceeds to the determination of the next attribute. (TNFX2 is, for example, about 0.7)
4) RNFX0 ≧ 0.5 and TNFX2 ≦ SNFX
It is determined that the current NFX value is a correct decoding result.

(B) MTP Error Detection and Correction At 10 frames / sec, the inter-frame motion vector is a three-frame vector at the video rate, so the maximum size is about plus 45 pixels. This is equivalent to a maximum of three MBK displacements. Therefore, the motion vector of the target area is motion-compensated with respect to the MTP pattern of one frame before by the motion amount (mx, my) converted into the MBK unit, and the comparison area composed of the MBK already decoded based on the MBK position is used as a reference. Are set as shown in FIGS. 59 and 60, and each MTP value is compared with a corresponding area by motion compensation. Here, the similarity SMTP is calculated based on the following equation (16), and the reliability evaluation value RMTP0 is calculated using equations (17) and (18) in order to evaluate the reliability of the MTP prediction up to that time. calculate.

ここで、
ｓmtp(A,B)；２つのＭＴＰの間の類似度を計算する関数で、ＭＴＰの含む６種類の各情報の一致に対して、以下のスコア値を設定して合計する
ＩＮＴＲＡ 属性の一致 → ３点
ＭＱＵＡＮＴ属性の一致 → １点
Ｍ Ｖ Ｄ 属性の一致 → ２点
Ｃ Ｂ Ｐ 属性の一致 → ２点
ＴＣＯＥＦＦ属性の一致 → １点
Ｆ Ｉ Ｌ 属性の一致 → １点
ＬMTP ；スコア設定の合計値（ここでは「１０」）
Ｋ ；比較領域に含まれるＭＢＫの個数，１番目は現在復号中のＭＢＫ
Ｋ０ ；ＭＴＰ予測の信頼度の計算領域に含まれるＭＢＫの個数でＫ以上の値
MTP(i)；ＧＯＢの１番目のＭＢＫのＭＴＰ値，ＦＩＸＥＤでは０
MTP ＿(i) ；１フレーム前のＭＴＰパターンから予測したＭＴＰパターン，通常は動き補償による比較領域のＭＴＰパターンをそのまま予測パターンとする
上述の設定により、例えばＭＴＰ＝１とＭＴＰ＝２との間の類似度は、ＭＶＤとＦＩＬ以外ではすべて一致しているので、
３＋１＋０＋２＋１＋０＝７（点）
となる。したがって、その場合には、
ｓmtp(1,2)＝ｓmtp(2,1)＝７
となる。同様にして、他の組み合わせについても計算することができる。

here,
smtp (A, B); a function for calculating the degree of similarity between two MTPs. The following score values are set and totaled for the coincidence of each of the six types of information including the MTP.
INTRA attribute match → 3 points
Match of MQUANT attribute → 1 point
MVD attribute match → 2 points
CBP attribute match → 2 points
TCOEFF attribute match → 1 point
FI L attribute match → 1 point LMTP; total score setting (here “10”)
K: the number of MBKs included in the comparison area, the first is the MBK currently being decoded
K0: the number of MBKs included in the calculation area of the reliability of MTP prediction and a value equal to or greater than K
MTP (i): MTP value of the first MBK of GOB, 0 in FIXED
MTP_ (i): An MTP pattern predicted from the MTP pattern one frame before, that is, an MTP pattern of a comparison area normally obtained by motion compensation is used as a prediction pattern as it is. For example, between MTP = 1 and MTP = 2 Are similar except for MVD and FIL,
3 + 1 + 0 + 2 + 1 + 0 = 7 (points)
It becomes. Therefore, in that case,
smtp (1,2) = smtp (2,1) = 7
It becomes. Similarly, calculation can be performed for other combinations.

Next, based on the above results, MTP detection and error correction are performed under the following four conditions, for example.
1) RMTP0 <0.5
It is determined that the reliability of the MTP prediction is low, and is suspended. That is, for the time being, it is determined that the current MTP value is correct, and the process proceeds to the determination of the next attribute.

2) RMTP0 ≥ 0.5 and SMTP <TMTP1
<< 1 >> It is determined that the current MTP pattern MTP (L) is a decoding error. The MTP value is copied from the prediction pattern. (TNFX1 is, for example, about 0.3)
<< 2 >> On the other hand, all MTPs having a similarity of 8 or more (dissimilarity of 2 or less) are referenced, and the corresponding VLC is referenced.

<< 3 >> The VLC to be referred to and the bit stream sequence are compared by fuzzy matching processing to select the VLC with the highest matching degree.
<< 4 >> If the degree of collation satisfies a certain criterion (0.8 or more), the VLC and MTP are adopted. If not, the MTP of the first estimated value is adopted.
3) RMTP0 ≧ 0.5 and TMTP1 ≦ SMTP <TMTP2
Since the current MTP value cannot be determined to be a decoding error, it is suspended. That is, it is determined that the current MTP value is correct, and the process proceeds to the determination of the next attribute. (TMBA2 is, for example, about 0.7)
4) RMTP0 ≥ 0.5 and TMTP2 ≤ SMTP
It is determined that the current MTP value is a correct decoding result.

(C) Error detection and correction of QSC All QSCs are fixed at 31 (quantization step 62) at a transmission rate of 32 kbps or less, and only the main area determined from area analysis in the target area priority mode is 16 (quantization step 32). And Therefore, QSC information is not transmitted in the person mode. As a result, the QSC information of the GOB layer and the quantization scale change information in MBK units become unnecessary, so that an error of the QSC does not occur.

(D) MVD Error Detection and Correction Since MVD is represented by a differential motion vector between adjacent MBKs in a frame, it is difficult to determine an error in the data format as it is. Therefore, a method is employed in which the MVD data is returned to the original motion vector MV form and then evaluated. Note that the MVD attribute data can be regarded as a signal value having statistical properties that are temporally and spatially continuous compared to the values of other MBK attributes, so that linear prediction from past frames and Linear interpolation becomes possible. Therefore, first, when there is no area information, the motion vector MV is calculated according to the following equation (19).

上式において、
ｖx(L,M)；Ｍフレーム時刻におけるＬ番目のＭＢＫに関する水平方向の動きベクトル
ｖy(L,M)；Ｍフレーム時刻におけるＬ番目のＭＢＫに関する垂直方向の動きベクトル
Ａ ；ｘまたはｙを表す添字
ｖA ＿(L,M) ；Ｍフレーム時刻におけるＬ番目のＭＢＫに関する動きベクトルの推定値
ａ(i) ；フレーム内の線形補間係数
ｂ(m) ；フレーム間の線形予測係数
ｕ ；フレーム内補間とフレーム間予測の比率（０≦ｕ≦１）
Ｋ ；復号中のＭＢＫを含む周囲のＭＢＫ領域のＭＢＫ個数
ｐ ；線形予測を行うための過去のフレーム数
そして、周囲ＭＢＫの設定の仕方は、前述のＭＴＰの比較領域の場合に準ずる。このようにして得た推定ベクトルの値について次式（２０）の誤差評価の式を用いて評価する。

In the above formula,
vx (L, M); horizontal motion vector for L-th MBK at M-frame time vy (L, M); vertical motion vector A for L-th MBK at M-frame time A; subscript representing x or y vA_ (L, M); Estimated value of motion vector for L-th MBK at M-frame time a (i); Linear interpolation coefficient in frame b (m); Linear prediction coefficient u between frames u; Inter-frame prediction ratio (0 ≦ u ≦ 1)
K: the number of MBKs in the surrounding MBK area including the MBK being decoded; p: the number of past frames for performing linear prediction. The manner of setting the surrounding MBK is the same as in the case of the above-described MTP comparison area. The value of the estimated vector obtained in this way is evaluated using the error evaluation equation of the following equation (20).

式（２０）により得られた誤差評価の値Ｅにより次のように判定する。
１）Ｅ≧２０
《１》復号したＭＶＤは誤りであると判定しｖＡをｖＡ＿で置き換える。

The following judgment is made based on the value E of the error evaluation obtained by the equation (20).
1) E ≧ 20
<< 1 >> The decoded MVD is determined to be an error, and vA is replaced with vA_.

<< 2 >> MVD corresponding to this is calculated, and the corresponding VLC is referenced within a range of ± 5 with that as the center value.
<< 3 >> The referenced VLC and the bit stream sequence are compared by fuzzy matching processing, and the VLC with the highest matching degree is selected.
<< 4 >> If the collation degree satisfies a certain criterion (0.8 or more), the VLC and MVD are adopted. If not, the MVD of the first center value is adopted.

2) 20> E ≧ 10
Hold. The decrypted MVD is held for the time being.
3) 10> E ≧ 0
It is determined that the decrypted MVD is correct.
Next, when there is area information, the motion vector is calculated based on the following equation (21) instead of equation (19), and the evaluation is performed in the same manner as described above.

ここで、
ｖRA(L,M-m) ；ｍフレーム前の時刻における領域の平均動きベクトル
（ｅ）ＣＢＰの誤り検出と訂正
１）領域情報がない場合
動き補償予測符号化処理においてＣＢＰデータはテクスチャーや色の時間的変化の度合いを示す数値と考えることができる。しかし、このＣＢＰデータは、ＭＶＤのような線形補間計算が行えるような代数構造とはなっていないので、図６１〜６４に示すように、ＣＢＰ値をいったんＹＵＶベクトルに変換してからＭＶＤと同様の評価計算を実施する。そこで、まずＹＵＶベクトルへの変換に際しては、次式（２２）にしたがって計算する。

here,
vRA (L, Mm); Average motion vector of the area at the time before m frames (e) Error detection and correction of CBP 1) When there is no area information In motion compensation prediction coding processing, CBP data is temporally represented by texture or color. It can be considered as a numerical value indicating the degree of change. However, since this CBP data does not have an algebraic structure that allows linear interpolation calculation like MVD, as shown in FIGS. 61 to 64, CBP values are once converted to YUV vectors, and then converted to YUV vectors. Perform evaluation calculation of. Therefore, at the time of conversion into a YUV vector, calculation is performed according to the following equation (22).

ここで、
c(L,M) ；Ｍフレーム時刻におけるＬ番目のＭＢＫに関するＹＵＶベクトル
c ＿(L,M) ；Ｍフレーム時刻におけるＬ番目のＭＢＫに関するＹＵＶベクトルの推定値
ac(i) ；フレーム内の線形補間係数
bc(m) ；フレーム間の線形予測係数
uc；フレーム内補間とフレーム間予測の比率（０≦uc≦１）
Kc；復号中のＭＢＫを含む周囲のＭＢＫ領域のＭＢＫ個数
pc；線形予測を行うための過去のフレーム数
LN(i,m) ；Ｍフレーム時刻における比較領域中のｉ番目のＭＢＫがＧＯＢ中のアドレスで何番目であるかを示す番号対応付けの関数，比較領域を設定すれば一意的に決めることができる
そして、周囲ＭＢＫの設定の仕方は、前述のＭＴＰの比較領域の場合に準ずる。このようにして得た推定ベクトルの値について次式（２３）の誤差評価の式を用いて評価する。

here,
c (L, M); YUV vector for L-th MBK at M-frame time
c _ (L, M); Estimated value of YUV vector for L-th MBK at M-frame time
ac (i): Linear interpolation coefficient within the frame
bc (m): Linear prediction coefficient between frames
uc; ratio between intra-frame interpolation and inter-frame prediction (0 ≦ uc ≦ 1)
Kc: Number of MBKs in the surrounding MBK area including the MBK being decoded
pc; number of past frames for performing linear prediction
LN (i, m): A function of number association indicating the number of the i-th MBK in the comparison area at the M-frame time in the address in the GOB, and it can be uniquely determined by setting the comparison area. The setting method of the surrounding MBK can be based on the case of the above-described MTP comparison area. The value of the estimated vector obtained in this way is evaluated using the error evaluation equation of the following equation (23).

ｄ１(A,B) ；ベクトルＡとベクトルＢとの間の絶対値距離
式（２３）により得られた誤差評価の値Ｅにより次のように判定する。なお、ＹＵＶベクトルの定義から、
１２≧Ｅｃ≧０
であるので、以下のような判定を行う。

d1 (A, B); absolute value distance between the vector A and the vector B is determined as follows based on the value E of the error evaluation obtained by the equation (23). From the definition of the YUV vector,
12 ≧ Ec ≧ 0
Therefore, the following determination is made.

A) Ec ≧ 7
<< 1 >> It is determined that the decoded CBP is an error, and c_ is replaced with c.
<< 2 >> CBP is calculated within a range of ± 1 with the replaced c as a center value (a plurality of CBPs can exist for one c), and the corresponding VLC is referred to.
<< 3 >> The referenced VLC and the bit stream sequence are compared by fuzzy matching processing, and the VLC with the highest matching degree is selected.

<< 4 >> If the matching degree satisfies a certain criterion (0.8 or more), the VLC and CBP are adopted. If not, the CBP of the first center value is adopted.
B) 7> Ec ≧ 4
Hold. For now, the decrypted CBP is held.
C) 4> Ec ≧ 0
It is determined that the decrypted CBP is correct.

2) When there is region information Next, when there is region information, the calculation of the YUV vector is performed based on the following expression (24) instead of expression (22). Perform in a similar manner.

cR(L,M-m) ；ｍフレーム前の時刻における動き補償領域のＹＵＶベクトル
ただし、ｕｃの値は、領域情報がない場合の値よりも若干小さい値に設定することが望ましい。また、ｐｃは逆に少し大きく設定する。

cR (L, Mm); YUV vector of the motion compensation area at the time m frames before. However, it is desirable to set the value of uc to a value slightly smaller than the value when there is no area information. On the other hand, pc is set slightly larger.

(F) Processing of pending items Now, as a result of the detection of five attributes and the evaluation of errors as described above, if, for example, three or more attributes to be retained occur, the attribute in which the retention appears Only, the attribute value is replaced with the error-corrected attribute value, and the evaluation is performed again with reference to the VLC.
[B-4] Error Correction at Signal Level The error correction at the pattern level described so far is error correction depending on the coding attribute in MBK units, and is not means for evaluating the value of the image signal itself. . The image signal becomes available only when the bit stream data of the BLK layer is decoded at the code level and restored as a quantized DCT coefficient block. Therefore, error correction at the level of the image signal thus restored will be described below.

(1) Error Correction Based on Properties of Block Pixels (a) Detection of Errors Normally, decoding errors in the BLK layer are caused by decoding errors of DC components of quantized DCT coefficient blocks due to grammatical constraints and VLC tree code properties. Lead to. As a result, even if the MBK is grammatically decoded, the color (the DC component of the UV block is dominant) and the texture are likely to be very different from the surroundings (see FIG. 66). ).

  On the other hand, as shown in FIG. 65 (b), a color different from that of the surrounding blocks is accidentally generated only in a tile-shaped pixel on a pixel block grid (see also FIG. 65 (a)) as a normal image signal. Is considered to be very small. According to this concept, when the MTP of the target MBK indicates INTER, the reliability is further improved. This is because if the MBK has a large change in pixel value, it is highly likely that the MBK is determined to be normally INTRA. Therefore, here, an error is detected by focusing on the color signal as follows.

        1) As the simplest method, color is evaluated after inverse quantization and inverse DCT are performed to restore the image signal block. This evaluation is based on a comparison between the linear interpolation from the surrounding MBK and the MBK estimated by the motion compensated linear prediction from the MBK of the past frame and the MBK restored from the current bit stream. Specifically, an error evaluation calculation is performed according to the following equations (25) to (29).

ただし、各記号は以下の通りである。
ｄ１(A,B) ；ベクトルＡとベクトルＢとの間の絶対値距離
Ｃｂ；ビットストリームに基づく復元ＭＢＫの色ベクトル
Ｃｅ；推定したＭＢＫの色ベクトル
Ｙ＄；Ｙブロックの平均画素値
Ｕ＄；Ｕブロックの平均画素値
Ｖ＄；Ｖブロックの平均画素値
YBLK$(n,k)；ＭＢＫ中のｎ番目のＹブロックのｋ番目の画素値（ n=1〜4, k=1〜64）
UBLK$(k)；ＭＢＫ中のＵブロックのｋ番目の画素値
VBLK$(k)；ＭＢＫ中のＶブロックのｋ番目の画素値
以上の計算の結果得られる誤差評価値Ｅｃに対して、
７６５（＝２５５×３）≧Ｅｃ≧０
という条件があるので、この範囲内で、例えば、Ｅｃ＞４００である場合に誤りであると判定する。なお、ＭＢＫの予測推定計算については、次の２）と同様にして行う。

However, each symbol is as follows.
d1 (A, B); absolute value distance between vector A and vector B Cb; color vector of restored MBK based on bit stream Ce; estimated color vector of MBK Y ＄; average pixel value of Y block U ＄; Average pixel value of U block V ＄; Average pixel value of V block
YBLK $ (n, k); k-th pixel value of n-th Y block in MBK (n = 1 to 4, k = 1 to 64)
UBLK $ (k); k-th pixel value of U block in MBK
VBLK $ (k); k-th pixel value of V block in MBK For the error evaluation value Ec obtained as a result of the above calculation,
765 (= 255 × 3) ≧ Ec ≧ 0
Within this range, for example, if Ec> 400, it is determined that an error has occurred. The MBK prediction estimation calculation is performed in the same manner as in the following 2).

        2) A YUV vector is constructed using only the DC component of the quantized DC coefficient block, and the error between the linear prediction from the surrounding MBK and the YUV vector estimated by the motion compensation prediction from the past frame is expressed by the following equation (30). Therefore, it is calculated and evaluated.

ただし、
c(L,M) ；Ｍフレーム時刻におけるＬ番目のＭＢＫに関するＹＵＶベクトル
c(L,M)＝(Y,U,V) であり、Ｙは４つのＢＬＫのＤＣ成分の平均、
Ｕ，ＶはそれぞれＵＢＬＫ，ＶＢＬＫのＤＣ成分
cR(L,M-m) ；領域情報がある場合はｍフレーム前の時刻における動き補償領域のＹＵＶベクトルであり、領域情報がない場合はc(L,M)と同等
c ＿(L,M) ；Ｍフレーム時刻におけるＬ番目のＭＢＫに関するＹＵＶベクトルの推定値
ac(i) ；フレーム内の線形補間係数
bc(m) ；フレーム間の線形予測係数
uc；フレーム内補間とフレーム間予測の比率（０≦uc≦１）
Kc；復号中のＭＢＫを含む周囲のＭＢＫ領域のＭＢＫ個数
pc；線形予測を行うための過去のフレーム数
そして、周囲ＭＢＫの設定の仕方は、次項のＭＢＫ属性の比較領域の取り方に準ずる。このようにして得た推定ベクトルの値について次式（３１）の誤差評価の式を用いて評価する。なお、誤りの判定条件は、１）の場合と同じである。

However,
c (L, M); YUV vector for L-th MBK at M-frame time
c (L, M) = (Y, U, V), where Y is the average of the DC components of the four BLKs,
U and V are DC components of UBLK and VBLK, respectively.
cR (L, Mm); if there is area information, it is the YUV vector of the motion compensation area at the time m frames before, and if there is no area information, it is equivalent to c (L, M)
c _ (L, M); Estimated value of YUV vector for L-th MBK at M-frame time
ac (i): Linear interpolation coefficient within the frame
bc (m): Linear prediction coefficient between frames
uc; ratio between intra-frame interpolation and inter-frame prediction (0 ≦ uc ≦ 1)
Kc: Number of MBKs in the surrounding MBK area including the MBK being decoded
pc: the number of past frames for performing linear prediction And the setting method of the surrounding MBK is in accordance with the method of setting the comparison area of the MBK attribute described in the next item. The value of the estimated vector obtained in this manner is evaluated using the error evaluation expression of the following expression (31). The error determination conditions are the same as in the case 1).

（ｂ）誤りの訂正
復号したＢＬＫが誤りであると判定した場合には、推定値で置き換えるようにする。一方、次のＢＬＫの復号を行うには、符号レベルでビットストリーム復号再開位置を決める必要がある。このためには符号レベルの誤り訂正におけるＥＯＢ検出プロセスを起動する。このとき、もし、再生した（訂正も含む）ＭＢＫの数がＧＯＢ中で２２以上であるならば（ＧＯＢをなす３列のブロックのうちの２列分の個数が２２個である）、残りのＭＢＫについてはすべて領域情報を利用した動き補償予測に置き換えるようになっている。

(B) Error Correction When it is determined that the decoded BLK is an error, it is replaced with an estimated value. On the other hand, in order to decode the next BLK, it is necessary to determine the bit stream decoding restart position at the code level. For this purpose, an EOB detection process in code level error correction is started. At this time, if the number of reproduced MBKs (including corrections) is 22 or more in the GOB (the number of blocks in two columns of the three columns of blocks forming the GOB is 22), the remaining All MBKs are replaced with motion compensated predictions using region information.

(2) Prevention of Error Propagation by Periodic Forced Intra Even if error correction as described in the previous section is performed, if decoding processing by frame addition is continued, normal image reproduction becomes difficult due to accumulation of error propagation. . Therefore, the following error propagation prevention measures are considered.
(A) By periodically inserting the forced intra MBK based on the area information, it is possible to prevent an image failure of an important part. In particular, as shown in FIG. 67, in the face part, forced intra blocks are intensively assigned to the mouth and ears, and cyclical circulation is performed while suppressing an increase in the data amount.

(B) After performing error correction on the decoder 2, the encoder 1 is requested to transmit the region-based forced intra in a distributed manner over a certain time period in the future. In this case, it is not necessary to perform picture freeze because a remarkable increase in the amount of information can be avoided as compared with ARQ (request signal) in frame units.
[B-5] Strategy Control of Error Correction Using Mode Information As described above, the mode information set in the encoder 1 is divided into three categories of use environment, subject, and encoding control. The use of the mode information enables the decoder 2 to specify an error correction route.

(1) Usage environment (a) Fixed in the vehicle By setting the background memory 66 and the person mode, it is possible to activate the error correction function of all levels of recognition, pattern, and signal.
(B) Movable in car There is a possibility of both the person mode and the landscape mode. In the case of the person mode, it is the same as the case of the above-mentioned fixed in the car. However, the data in the background memory 66 must always be updated and stored.

(C) Fixed indoor The background memory 66 can be used. Even when the subject is not a person, a template or a 3D model can be specially set depending on the purpose, so that error correction using area information similar to that in the person mode can be activated.
(D) Movable indoor The error correction function can be activated only in the person mode.

(2) Subject As described in the use environment described above, the mode is divided into the portrait mode and the landscape mode. Since the landscape mode is difficult to specify the shape and color, the activation of the error correction function other than relying on the background memory 66 is not possible. difficult. In the person mode, since feature amounts such as a template, a 3D model, a person image texture, eyes, nose, mouth, ears, hair, and skin color can be used, the correction function at all levels can be activated.

(3) Encoding control mode In a mode different from a normal real-time moving image reproduction request, for example, a still image mode or a video mail mode, there is no need to perform error correction in real time, so it is sufficient to activate ARQ. .
(4) Judgment of Error Occurrence State in Communication Channel When decoding is stopped because the image cannot be corrected by other error correcting means, the decoder 2 is forcibly moved from the decoder 2 side to the encoder 1 side. Send one of the following request signals. It should be noted that the transmission of such a request signal can be automatically selected and set according to the condition of the communication path, or can be set according to the user's preference. The processing is performed via the human interface 34.

(A) Retransmission request for video transmission starting from forced intra-picture This is the same as retransmission (ARQ) used in the existing system. However, since ultra-low-rate transmission leads to a remarkable increase in delay time and frequent picture freeze, it is not often used particularly in real-time transmission of 32 kbps or less. On the other hand, in the case of non-real-time transmission such as the video mail mode, this request is applied.

(B) New restart of moving image transmission starting from forced intra-picture When decoding processing is stopped, a predicted image is generated from a past frame for the remaining MBK, and the picture is frozen. Until the forced intra picture is normally transmitted (the start position is confirmed by the PSC header), the data in the input buffer is continuously discarded.
(C) A request to send a quasi-moving image only by forced intra-picture.

(D) A request to send a motion parameter in model-based coding.
[B-6] Error Correction of Recognition Level (1) Specification of Target Area The target area detection result and mode control information set by the encoder 1 are described in the user data area (PSPARE, GSPARE). Is transmitted as an image signal, and is detected by performing a decoding process on the decoder 2 side. Then, a template (2D information) is selected and transformed based on these pieces of information, and the area information extracted on the encoder 1 side is reproduced (see FIG. 68).

(A) Selection of basic template Since the same set of binary templates is provided between the encoder 1 and the decoder 2 in advance, they are common to each other by detecting an identification number designating the template. Template can be used.
(B) Deformation of Basic Template 1) Centering Centering is performed using a 2D vector (mx, my) expressing the horizontal and vertical shift amounts of the template center in pixel units.

2) Scaling The basic template is scaled at a ratio r with the center obtained as described above as the origin.
3) Modifications Modifications such as width, height, and posture angle are partially added to each basic template using shape parameters unique to each template.

(C) Identification of Site When the category of the object is known from the mode information and the template selection information, it is possible to know which part in the template corresponds to what part of the object. For example, when a template of a person's upper body is selected in the person mode, as shown in FIG. 16 used in the description of the encoder 1, a fine image area corresponding to a person's head, face, mouth, eyes, nose, etc. Can be specified.

(2) Activation of Error Determination In this embodiment, the cause of the error is specified as a bit error at the code level, and the image correction process is described. Therefore, error detection itself is not performed at the recognition level, but at the signal level, the code level, or the pattern level. The error correction of the recognition level is to provide various properties which are reflected in the area information and the image of the target object based on the area information, especially when errors are evaluated by them. Therefore, activation of the error correction process at the recognition level will be performed by another level module.

(3) Error Correction Using Region Information (a) 2D Motion Compensation Using Region Information If it is known in which region the MBK currently being decoded is included, as described in the description of pattern-level error correction, Motion compensation can be performed using the motion vector of the MBK in the area where decoding has already been completed. For example, if it is determined that the MBK being decoded is included in the face area of the head, a motion compensated prediction MBK can be created using the average head motion vector.

(B) Correction Using Part Information If the region is known, it is possible to correct not only motion but also color, luminance, and texture errors. For example, if the skin color is analyzed from the face area information before the previous frame, the error can be determined and corrected by the linear estimation value and the error evaluation as described in the section of the signal level and the pattern level. Become like

(C) Expression of region information The expression of the region information is based on the same template expression method as described in the description of the encoder 1. For example, in the person mode, a template is created for the subject in four layers of upper body, head, and face based on the whole body image. The face has a mouth, eyes, nose, cheeks, and forehead as main parts, and a relative position in the front face is set. The above-described skin color analysis is calculated by calculating an average color based on a portion excluding the mouth and eyes.

(D) Calculation of the relative position in the person area The basic template before the deformation is described as shown in FIG. 25 as described above. Thereby, the positions of all parts can be expressed as two-dimensional coordinates in the basic template.
(4) Error Correction by Background Memory If the area information becomes clear, the background area can be specified. Therefore, the image information of the background area is stored in the background memory 66 of the decoder 2. As a result, even when a bit error occurs during the decoding process of the background area, the same error correction as described above can be performed.

[B-7] Person Memory and 3D Model-Based Method (1) Registration of Person Image The first intra picture can be labeled by a person name and stored in the person memory 67 as person image data. Here, the personal identification name is assigned and registered as a label with the same meaning as the authentication number at the time of encoding by using the human interface 34. The area information in the intra picture is stored as a template number and deformation information. Further, the date and time of contact (call) with the person can be stored at the same time, or the voice information can be stored with the same label as far as the memory capacity allows, so that the function can be further improved.

(2) Calling of a Person Image The person image information stored by being registered in the person memory 67 can be called by a user at an appropriate timing as a business card image based on the person identification name. This makes it possible to remember the face of the other party who made the call once, and can be used, for example, as a means for authenticating the other party when making another call. It can also be used as a texture image in the model-based decoding mode when a transmission error becomes severe.

(3) Model-based decoding mode The above-described person image is texture-mapped to a 3D model of a person, and 3D motion information is added, thereby generating a person image as an auxiliary image reproducing unit when a transmission error becomes severe. . The 3D motion information may be provided so that the decoder 2 appropriately looks like a motion of a person. Also, control can be performed using the 3D motion information extracted on the encoder 1 side.

(4) Spatial limitation based on 3D model If 3D motion information is given from the encoder 1 side, region prediction can be performed for a case other than a front image that cannot be expressed by only the 2D template information described above. That is, as shown in FIG. 13, the 2D area can be specified on the image plane by the perspective projection of the 3D model.

[B-8] Error Correction Strategy Based on the error determination result at each level described above, the error correction function finally exercised can be summarized as follows. Each level number is a value that predicts the degree of fidelity of the reproduced moving image to the original image.
<Level 0> 3D model-based reproduction <Level 1> Picture freeze <Level 2> Model-based estimation <Level 3> Linear estimation / motion compensation in GOB units based on region information <Level 4> Linear estimation in MBK units based on region information -Motion compensation <Level 5> Correction of code level [B-9] Description of decoding operation according to flowchart Now, as described above, the actual decoding process of the error correction function performed in the decoding process will be described. , Each function is performed according to the flowcharts of the programs shown in FIGS. Hereinafter, an outline of the entire flow will be described.

  That is, first, fuzzy matching processing is performed on the bit stream data accumulated in the FIFO buffer 42 by the fuzzy matching unit 44 of the communication path decoding unit 35 to search for a PSC (step B1). The decoding process of the required number of bits, mode information, and area information is performed by 43 (steps B2 to B4). Then, the global check is performed again based on these data, and the GBSC in one frame is localized (step B5). Thereafter, a determination operation for a code level error is performed by the error total determination routine shown in FIG. 11 (step B6).

  Next, the use environment, the object (subject), and the decoding mode are set by the mode control unit 33 based on the obtained mode information (steps B7 to B9). If the mode is the person mode, the basic template is selected, corrected, deformed, and scaled to reproduce the area pattern (steps B10 to B13). Perform the correction function.

  First, the attribute estimating unit 61 detects and corrects errors in MBA and MTP (steps B18 and B19). If the person mode is set, the error determination / correction unit 50 suppresses QSC. , MVD error detection and correction (steps B20 to B22), and then the CBP error detection and correction are performed by the attribute estimating unit 61 (step B23). Based on these results, the error determination / correction unit 50 performs an overall error determination according to the routine shown in FIG. 11 (step B24), and subsequently, when an attribute error is detected in any of the processes described above. Returns to that step and executes error detection (steps B25 to B28). When these steps are completed, the data of the decoded attribute array is stored in the pattern attribute section 39c, which is an attribute memory.

  Thereafter, after steps B30 to B33, the decoding processing unit 36 performs H.264 decoding on the basis of the data of the attribute array described above. The BLK decoding process is performed according to the H.261 standard (step B34). Next, when the person mode is set, the pattern / signal level error correction unit 37 estimates the texture and color based on the part information, the feature amount, and the linearity of the color vector based on the surrounding and the past MBK. Estimation and color vector evaluation are performed (steps B36 to B38).

  Here, if an error has occurred and LBLK is 22 or more, motion compensation estimation is performed on the remaining MBK in the GOB, and thereafter, the GOB ends (steps B40 and B41), and an error occurs. If LBLK is equal to or less than 22 or no error has occurred, the ESC detection and correction and the EOB detection and correction are performed, and then the error total determination routine shown in FIG. B42 to B44), and thereafter, by repeating this, the decoding process is continued.

  According to the present embodiment, when the decoder 2 receives a video signal of a moving image through a digital communication channel having a high transmission error rate and performs decoding processing, the decoding of the existing moving image is performed. Compression standard H. 261, an encoded bit stream with an improved ultra-low rate is received, and in the decoding process, an error correction function is provided at each level of code, grammar, pattern, signal, and recognition based on the protocol. Is organically executed, and furthermore, by globally checking an encoded bit stream in consideration of a required bit amount, error detection of a pattern, a signal, and a grammar is driven, and area information (person, etc.) based on mode information and a 2D template is obtained. Error correction based on the evaluation of the recognition level.

  Further, according to the present embodiment, the encoder 1 changes the syntax and replaces the code word, predicts the coding attribute of the current frame based on the past coding attribute, and adaptively controls the attribute determination, Ultra-low-rate image transmission can be realized by comprehensively using the object region extraction and region-specific quantization control using a model and a model, and controlling the significant number of transform coefficients according to the use mode, transmission rate, and motion generation amount. It is something that will be. The encoder 1 according to the present embodiment is of a level that can be realized by only slightly changing the current image compression standard (H.261). It can be realized with a simple configuration.

FIG. 2 is a block diagram of a decoder showing an embodiment of the present invention. Block diagram of encoder Conceptual diagram of the encoding process Conceptual diagram of the decryption process Flowchart of encoding processing program (1) Flowchart of encoding processing program (part 2) Flowchart of encoding processing program (3) Flowchart of decryption processing program (1) Flowchart of decryption processing program (part 2) Flowchart of decryption processing program (3) Flowchart of error total judgment routine Explanatory diagram when the camera is fixed inside the car as the usage environment Illustration of three-dimensional positional relationship of people in the car Examples of camera images inside a car, indoors and outdoors including people Explanatory diagram of setting of area by distance scale Explanatory drawing of the template and the characteristic area of the person front view Explanatory diagram when compensating for transmission delay in model-based mode H. FIG. 1 is an explanatory diagram of a configuration for performing communication with a H.261 terminal. State transition diagram of mode control for usage environment and subject H. Example of macroblock attribute array based on H.261 syntax (1) H. Example of macroblock attribute array based on H.261 syntax (part 2) H. Example of macroblock attribute array based on H.261 syntax (part 3) Flowchart of routine for extracting moving area and determining template Explanatory diagram of extraction of moving area and determination of template Illustration of the operation of the basic template and its deformation Illustration of the principle of ultra-low rate conversion based on extraction of a person region (A) Example of GOB and MBK lattice in CIF format and (b) Background memory image Illustration of the effect of using and updating background memory Chart for explaining the concept of the entire coding control H. 261 standard bit stream syntax Explanatory drawing of GOB number and additional part of header (GBSC) Explanatory drawing comparing the case where the GOB header is reduced and the case where the GOB header is not reduced H. Comparison table of MTP variable length code (VLC) in H.261 standard Correspondence table between the occurrence probability of each MTP value and the code length for a human image INTER / INTRA determination characteristic diagram Motion block determination characteristic diagram Flowchart of MBA prediction routine Explanatory diagram showing the correspondence between MBA and NFX Illustration of prediction of MBA pattern from previous frame Flowchart of routine for reducing MTP information Explanatory drawing of MTP code amount reduction by average motion vector for each area Illustration of quantization by region and quantization template in person mode Illustration of model-based transmission of quantization template Flowchart of QSC setting routine Flowchart of routine for reducing MVD information Explanatory drawing of MVD code amount reduction by average motion vector for each area Flowchart of CBP prediction and evaluation routine Explanatory drawing of motion compensation prediction for each area of CBP Explanatory diagram when performing interleaving of MBK attribute prediction Illustration of required bit number transmission format and global check Illustration of fuzzy matching process of PSC Explanatory diagram of combining a binary tree and generating a bit stream using syntax Example of misinterpretation caused by bit error when MBSTUFF is used Illustration of continuity between frames in the hierarchical encoding attribute (encoder side) Illustration of continuity between frames in the hierarchical encoding attribute (decoder side) Flowchart of MBA error detection and correction routine Explanatory diagram showing the correspondence between MBA and NFX Illustration of prediction of MBA pattern from previous frame Explanatory drawing of evaluation of decoding result based on similar calculation of MTP Explanatory diagram showing an example of a scan order in evaluation calculation Flowchart of CBP error detection and correction routine Explanatory drawing showing the definition of CBP Explanatory diagram showing a process of converting a CBP value into a YUV vector Illustration of prediction of YUV vector Explanatory drawing showing an example of (a) GOB and MBK lattice in CIF format and (b) error pattern in block form Explanatory diagram showing the effect on bit stream interpretation errors and signal errors caused by bit errors at the image signal level Explanatory diagram showing an example of period-based periodic distributed forced INTRA Flowchart of routine for area reproduction or pseudo area setting

Explanation of reference numerals

1 is an encoder (encoder), 2 is a decoder (decoder), 3 is a camera, 5 is an A / D converter, 7 is an orthogonal transformer, 8 is a subtractor, and 9 is a quantizer. , 10 is a channel coding unit, 11 is a FIFO buffer, 12 is a communication channel, 13 is an inverse quantization unit, 14 is an inverse transform unit, 15 is an adder, 16 is a prediction memory, 17 is a loop filter, and 20 is a motion. A detection unit, 21 is an encoding control unit, 22 is an attribute memory, 23 is an attribute prediction unit, 24 is an encoding processing unit, 25 is an area extraction / recognition processing unit, 26 is a target area extraction unit, 27 is a template database, 28 Is a 2D template matching unit, 29 is a model-based prediction unit, 30 is a three-dimensional shape database, 31 is a person memory (personal information storage unit), 32 is a background memory (background information storage unit), 33 is a mode control unit, and 34 is Human interface A chair, 35 is a communication path decoding unit, 36 is a decoding processing unit, 37 is a pattern / signal level error correction unit, 38 is a recognition level processing unit, 39 is a memory (storage means), 39a is a mode information unit, 39b is an area information section, 39c is a pattern attribute section, 39d is a 2D motion vector section, 39e is a personal identification information section, 41 is a D / A converter, 42 is a FIFO buffer, 43 is a parser, 44 is a fuzzy matching section, 45 Is a storage unit, 46 is a comparison table, 47 is an error determination unit, 48 is an inverse quantization unit, 49 is an inverse transform unit, 50 is an error determination / correction unit, 52 is an adder, 53 is a prediction memory, and 54 is motion compensation. Unit, 55 is a loop filter, 57 is a frame memory, 58 is a pixel value estimation unit, 59 is an image estimation unit, 60 is a motion vector estimation unit, 61 is an attribute estimation unit, 62 is a decoding control unit, and 63 is model-based prediction. Department 64 is a three-dimensional shape database, 64a is 3D shape data, 65 is a person image database, 66 is a background memory (background information storage means), 67 is a person memory (person information storage means), 68 is an area reproducing unit, and 69 is a template. The database 69a is a 2D template.

Claims (27)

An image signal decoding device that receives an image signal transmitted as an encoded bit stream compressed with a variable length code, decodes the image signal to reproduce the image signal,
A mode condition for setting the incidental state with a predetermined code is stored, and when the received image signal includes mode information, the mode information corresponds to the mode information when performing the decoding processing operation. Mode control means for selectively setting the mode condition to be performed;
Correction means for detecting and correcting an error in the decoded data when the decoding processing of the image signal is performed under the mode conditions selected and set by the mode control means. Decryption device. The mode control means includes:
Person information storage means capable of storing and reading data corresponding to the person on the screen,
Background information storage means capable of storing and reading background information located in a fixed area excluding the target area in the screen,
2. The image signal according to claim 1, wherein a mode control corresponding to a setting condition of a use environment mode is performed by using said person information storage means and said background information storage means as needed in accordance with a use environment. Decryption device. The mode control means includes:
Fixed mode and movable mode can be set for the inside and outside of the vehicle as the use environment mode, respectively,
In the vehicle internal fixed mode, under the conditions using the person information storage means and the background information storage means, to execute the error correction function by the correction means,
In the vehicle internal movable mode, in addition to the function of the vehicle internal fixed mode, performs an update process of the background information storage means,
In the outdoor fixed mode, the error correction function is executed by the error correction unit under conditions using the background information storage unit as a center,
3. The image signal decoding device according to claim 2, wherein in the outdoor movable mode, the error correction function of the error correction unit is executed under the condition in which the person mode is set. The mode control means includes:
Person information storage means capable of storing and reading data corresponding to the person on the screen,
Background information storage means capable of storing and reading background information located in a fixed area excluding the target area in the screen,
2. The image signal decoding according to claim 1, wherein the mode control corresponding to a setting condition of a subject mode is performed by using the person information storage unit and the background information storage unit as needed in accordance with a subject. Device. The mode control means includes:
The subject mode is configured so that a person mode and a landscape mode can be set. In the person mode, a decoding process is performed based on data stored in the person information storage unit and a preset characteristic condition specific to the person. Do
5. The image signal decoding apparatus according to claim 4, wherein in the landscape mode, a decoding process is performed based on background information stored in the background information storage unit. The mode control means includes:
In the decoding process, the code of the encoding mode set in the encoding process of the image signal is detected, and the decoding process is executed under the decoding conditions corresponding to the encoding mode, and the error correction process is executed by the error correction unit. The image signal decoding apparatus according to any one of claims 1 to 5, wherein the image signal decoding apparatus is configured to perform the decoding. The mode control means executes a decoding process according to one of a moving image, a quasi-moving image, and a still image mode set as an encoding mode,
The error correction unit is configured to execute the error correction function in the moving image mode and the quasi-moving mode, and to perform a retransmission request of an image signal when an error occurs in the setting state of the still image mode. 7. The image signal decoding apparatus according to claim 6, wherein: The mode control means includes:
When the encoding mode is set to a use mode of a template corresponding to a previously prepared shape or a model base mode indicating motion information to a three-dimensional model, the information corresponding to the area information of a person is set,
7. The image signal decoding apparatus according to claim 6, wherein the decoding process in the person mode is configured to set the template use mode or the model base mode to execute the decoding process. The error correction means,
When an error occurs in the decoding process and the decoding is stopped, one of the following request signals (1) to (4) is selectively sent to the transmitting side of the coded bit stream. 9. The image signal decoding apparatus according to claim 6, wherein the apparatus is configured to be capable of transmitting.
(1) Retransmission request signal for moving image transmission starting from forced intra picture (2) New restart request signal for moving image transmission starting from forced intra picture (3) Transmission request signal for quasi-moving image using only forced intra picture (4) Model-based coding Request signal for motion parameter in   9. The personal information storage means is provided so that a personal image of a specific person can be registered as personal information as needed, and can be read out at an appropriate timing. Image signal decoding apparatus.   11. The image signal decoding apparatus according to claim 10, wherein said mode control means is configured to be able to use personal image data of personal information stored in said personal information storage means in said model-based mode. . An image signal decoding device that receives an image signal transmitted as an encoded bit stream compressed with a variable length code, decodes the image signal to reproduce the image signal,
Color information estimating means for estimating color information of each block of pixel data decoded in block units from past color information data of the target block or color information data of surrounding blocks;
Error evaluation means for evaluating the color information of the target block based on the error value between the color information data obtained by decoding the target block and the color information data estimated by the color information estimation means,
Error correction means for replacing the color information data of the target block with the color information data estimated by the color information estimation means when there is an error in the color information data based on the evaluation result of the error evaluation means. An image signal decoding device. Transform coefficient decoding means for detecting and decoding data described in a fixed length code using a variable length code or an ESC code of a specific bit pattern as orthogonal transform coefficient data in each block in an encoded bit stream;
In the process of detecting the orthogonal transform coefficient data by the transform coefficient decoding means, an ESC position storage means for calculating the similarity between the variable length code and the ESC code and storing the position when the bit coincidence ratio is equal to or higher than a predetermined value. When,
When an error in the color information data is detected by the error correction means, the data described in the ESC position stored in the ESC position storage means is replaced before the color information data estimated by the color information estimation means is replaced. 13. The image signal decoding apparatus according to claim 12, further comprising: re-evaluation means for recognizing the data as fixed-length code data and performing decoding processing again. The EOB code described at the end of each block in the coded bit stream is searched. If the EOB code is not detected by the EOB detection means within a predetermined condition range in the target block, the EOB code is assumed assuming a 1-bit error. EOB detection means for re-executing the detection, and specifying the EOB code when the predetermined condition range is satisfied,
When the error correction unit detects an error in the color information data and stops decoding, the process returns to the next block based on the EOB code position detected by the EOB detection unit to execute decoding processing. 14. The image signal decoding apparatus according to claim 12, further comprising means. An image signal decoding device that receives an image signal transmitted as an encoded bit stream compressed with a variable length code, decodes the image signal to reproduce the image signal,
Error correction means for estimating the error from the past color information data or the color information data of surrounding blocks and correcting the error when the color information of the pixel data obtained by decoding processing in block units contains an error; ,
Error propagation that requests the transmitting side to periodically transmit forced intra-block data to a block side image signal of a portion that becomes an important area of a decoded image in accordance with the frequency of error correction performed by the error correction means. A decoding device for an image signal, comprising a prevention unit. An image signal decoding device that receives an image signal transmitted as an encoded bit stream compressed with a variable length code, decodes the image signal to reproduce the image signal,
Area specifying means for specifying a target area based on the area information data described in the user data area in the encoded bit stream data,
An image signal decoding apparatus comprising: an error correction unit that corrects an error included in an image signal in a decoding process in a target region specified by the region specifying unit based on the region information. The area specifying means,
Equipped with a plurality of basic templates for specifying the shape of the target area,
Based on the information of the basic template specified by the area information described in the encoded bit stream data from the transmission side, select and set one specified from the plurality of basic templates, and set the basic template to a target area. 17. The image signal decoding apparatus according to claim 16, wherein the portion is specified by correspondingly deforming.   The plurality of basic templates are provided in a hierarchy from a template indicating the entirety to a template indicating a detailed portion of the target object, and are configured to express the specified portion as two-dimensional coordinates in the basic template. 18. The image signal decoding apparatus according to claim 17, wherein:   When a region including a block currently being decoded is specified by the region specifying unit, a motion compensation unit that performs motion compensation of the block based on a motion vector of a block in the already decoded region is provided. 19. The image signal decoding device according to claim 18, wherein:   When the area including the block currently being decoded is specified by the area specifying means, the color information data of the block based on the color information data such as the luminance, color, and texture of the block in the already decoded area is determined. 20. The image signal decoding apparatus according to claim 18, further comprising a color information compensator for performing correction. A background information storage unit that stores image information of a background region other than the target region specified by the region specification unit as background information,
17. The image processing apparatus according to claim 16, wherein the error correction unit is configured to perform correction based on background information of the background information storage unit when an error occurs during decoding of the image data of the background area. 20. The image signal decoding device according to any one of 20. An image signal decoding device that receives an image signal transmitted as an encoded bit stream compressed with a variable length code, decodes the image signal to reproduce the image signal,
If the target object cannot be specified as a result of decoding the image signal, a pseudo region specifying unit that specifies a pseudo target region by performing an evaluation calculation centering on the image centroid based on the pattern information data of the past frame An image signal decoding device, comprising: An image signal transmitted as an encoded bit stream compressed with a variable length code is received, and at least one frame of bit stream data is stored in a buffer capable of being stored and then decoded to reproduce the image signal. In the decoding device for the decoded image signal,
Code checking means for searching for a start code of a specific bit pattern located at the beginning of the encoded bit stream data for one frame by globally checking the bit stream data stored in the buffer. An apparatus for decoding an image signal, comprising: The code checking means includes:
A GOB (Group of Blocks) start of a specific bit pattern attached at the beginning of the first block group included in the image signal, which is represented by dividing pixel data for one frame into a plurality of block groups. 24. The image signal decoding apparatus according to claim 23, wherein a code is searched for by globally checking bit stream data stored in said buffer.   The code checking means, when performing a global check of bit stream data, specifies a bit error rate of a specific bit pattern as a start code when the bit error rate is equal to or lower than a predetermined level. Item 25. An image signal decoding device according to Item 23 or 24. The code checking means includes:
Data indicating the required number of bits described at a position following the start code or the GB start code in the bit stream data is detected, and based on the detected data, an actual start code between the specified start code and the next start code is detected. 26. The image signal decoding apparatus according to claim 23, wherein localization is performed by determining a start code by comparing the number of bits with the number of bits. The code checking means includes:
It is characterized in that 8-bit data indicating the required number of bits of the block group described at a position following the GB start code is obtained as data obtained by inverting a bit pattern between the upper side and the lower side. The image signal decoding apparatus according to any one of claims 24 to 26, wherein:

Images