KR20220027433A - Method for generating multimedia bitstream including deep learning network model and apparatus thereof - Google Patents

Abstract

The present invention relates to a method for generating a multimedia bitstream comprising a deep learning network model and a decoding method thereof. The present invention can transmit, to an additional information area of a multimedia bitstream, a bitstream with a deep learning network model so that deep learning model data can be easily managed, deep learning performance can also be improved since the deep learning network model can be applied in units of a frame, a scene, and a picture, and the deep learning network model can be accurately applied. The method for generating the multimedia bitstream comprises: a step of generating a payload type field; a step of generating a use case field; a step of generating a location index field; a step of generating a parameter index field; a step of generating an attribute field; and a step of generating a payload field.

Description

Method for generating a multimedia bitstream including a deep learning network model and an apparatus therefor

The present invention relates to a method for representing and storing a deep learning network model, and more particularly, to a technique for including deep learning network model data in a bitstream on a multimedia.

Deep learning technology is showing encouraging performance that surpasses existing traditional methods in various application fields such as image recognition, voice signal processing, and natural language processing. Deep learning technology improves performance by training a network model composed of weights and biases to suit a given application field, and then re-applying the learned network model to the application field.

For this, it is necessary to transmit not only the multimedia bitstream but also the corresponding deep learning network model. However, since the transmission of the multimedia bitstream and the transmission of the deep learning network model are performed separately, there is a problem in that it is difficult to manage which deep learning network model is applied to which bitstream.

The inventors of the present invention have been researching and trying to solve the difficulties of managing such a deep learning network model of the prior art. In the prior art, the present invention was completed after much effort to provide a multimedia bitstream generation method in which a multimedia bitstream and a deep learning network model applied thereto can be managed together.

It is an object of the present invention to provide a multimedia bitstream generating method that facilitates management of a multimedia bitstream and a deep learning network model by including deep learning network model information in the multimedia bitstream and generating it as a single file.

On the other hand, other objects not specified in the present invention will be additionally considered within the range that can be easily inferred from the following detailed description and effects thereof.

A method for generating a bitstream according to the present invention comprises:

generating a payload type field that identifies a deep learning network model; generating a use case field that identifies an algorithm to be applied to multimedia data; generating a location index field that specifies a location in the multimedia data to which the deep learning network model is applied; generating a parameter index field that designates a parameter to be applied to the multimedia data; generating an attribute field indicating an attribute of the multimedia data after the deep learning network model is applied; and generating a payload field including deep learning network model data according to the payload type.

The bitstream is characterized in that it is included in the additional information area of the multimedia bitstream.

The bitstream is characterized in that it is included in the multimedia data area of the multimedia bitstream.

The position in the multimedia data to which the deep learning network model is applied is characterized in that it is divided into frames, regions, or scenes of the multimedia data.

The parameter index to be applied to the multimedia data is a video parameter set (VPS) index, a sequence parameter set (SPS) index, or a picture parameter set (PPS) index.

In the step of generating the use case field, the algorithm to be applied to the multimedia data is characterized in that it is super resolution or noise reduction.

The properties of the multimedia data after the deep learning network model is applied include image resolution, frame rate, or color space.

An encoding apparatus according to another embodiment of the present invention includes: a memory; and a bitstream generator including one or more processors to generate a bitstream including multimedia data and a deep learning network model to be applied to the multimedia data and store it in the memory; including, wherein the bitstream generator comprises: a deep learning network Generates a payload type field for classifying a model, a use case field for classifying an algorithm to be applied to multimedia data, an index field for designating a location in the multimedia data to which the deep learning network model is applied, and the It is characterized in that a field indicating the properties of the multimedia data after the deep learning network model is applied is generated, and a payload field including the deep learning network model data according to the payload type is generated.

A bitstream parsing method according to another embodiment of the present invention,

classifying the deep learning network model type by the payload type field; classifying an algorithm to be applied to multimedia data according to a use case field; Separating the location in the multimedia data to which the deep learning network model is applied by the location index field; classifying the properties of the multimedia data after the deep learning network model is applied by the property field; and classifying the deep learning network model data according to the payload type by the payload field.

A decoding apparatus according to another embodiment of the present invention,

Memory; and a bitstream parsing unit that parses and stores a bitstream including a deep learning network model to be applied to multimedia data by including one or more processors in the memory; including, wherein the bitstream parsing unit performs the deep learning by the payload type field Classify the type of learning network model, classify an algorithm to be applied to multimedia data by a use case field, classify a location in the multimedia data to which the deep learning network model is applied by a location index field, and classify the deep learning network model by an attribute field It is characterized in that the attribute of the multimedia data after the learning network model is applied is distinguished, and the deep learning network model data according to the payload type is distinguished by the payload field.

According to the present invention, there is an advantage in that it is easy to manage which deep learning network model is applied to which multimedia bitstream by including the deep learning network model data in the multimedia bitstream.

In addition, since the deep learning network model can be applied to each more granular unit such as a frame unit included in the multimedia bitstream rather than the multimedia bitstream unit, there is an effect of improving the performance of the deep learning model.

On the other hand, even if it is an effect not explicitly mentioned herein, it is added that the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as described in the specification of the present invention.

1 shows an example in which a multimedia bitstream and a deep learning network model according to the prior art are applied.
2 is an example for applying a deep learning network model to a multimedia bitstream according to a preferred embodiment of the present invention.
3 shows the structure of a multimedia bitstream to which the multimedia bitstream generating method according to a preferred embodiment of the present invention is applied.
4 is a flowchart of a method for generating a multimedia bitstream according to another preferred embodiment of the present invention.
5 is a flowchart of a multimedia bitstream decoding method according to another preferred embodiment of the present invention.
6 is a structural diagram of an encoding apparatus according to a preferred embodiment of the present invention.
7 is a structural diagram of a decoding apparatus according to a preferred embodiment of the present invention.
※ It is revealed that the accompanying drawings are exemplified as a reference for understanding the technical idea of the present invention, and the scope of the present invention is not limited thereby

The above object and means of the present invention and its effects will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily understand the technical idea of the present invention. will be able to carry out In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. In this specification, the singular form also includes the plural form as the case may be, unless otherwise specified in the phrase. In this specification, terms such as "include", "provide", "provide" or "have" do not exclude the presence or addition of one or more other elements other than the mentioned elements.

In this specification, terms such as “or” and “at least one” may indicate one of the words listed together, or a combination of two or more. For example, “A or B” and “at least one of A and B” may include only one of A or B, or both A and B.

In the present specification, descriptions according to “for example” and the like may not exactly match the information presented, such as recited properties, variables, or values, tolerances, measurement errors, limits of measurement accuracy, and other commonly known factors The embodiments of the invention according to various embodiments of the present invention should not be limited by effects such as modifications, including

In this specification, when it is described that a component is 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in between. It should be understood that there may be On the other hand, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that there is no other element in the middle.

In this specification, when it is described that a certain element is 'on' or 'adjacent' to another element, it may be directly in contact with or connected to the other element, but another element may exist in the middle. It should be understood that On the other hand, when it is described that a certain element is 'directly above' or 'directly' of another element, it may be understood that another element does not exist in the middle. Other expressions describing the relationship between elements, for example, 'between' and 'directly between', etc. may be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various components, but the components should not be limited by the above terms. In addition, the above term should not be construed as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first component' may be termed a 'second component', and similarly, a 'second component' may also be termed a 'first component'.

Unless otherwise defined, all terms used herein may be used with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 shows an example in which a multimedia bitstream and a deep learning network model according to the prior art are applied.

The multimedia bitstream decoding apparatus 1 includes a decoder 2 and a deep learning processing unit 3 .

The decoder 2 may decode the multimedia bitstream 4 to produce video/audio.

The deep learning processing unit 3 applies the deep learning network model 5 to the decoded video/audio content to create video/audio content with improved picture quality or sound quality.

As the fields to which deep learning is applied increase, the need for a standard model for exchanging deep learning network models is increasing. Accordingly, the Open Neural Network Exchange (ONNX) format was established by a coalition of companies such as Microsoft and Facebook, and the Neural Network Exchange Format (NNEF) was established by the Kronos Group, a hardware and software industry consortium that establishes advanced acceleration standards. In addition, MPEG (Moving Picture Experts Group), an international standard organization, is establishing a standard for Neural Network Representation (NNR) to efficiently compress the weights and biases of a deep learning network model. VCM (Video Coding for Machine) standard establishment for compressing and transmitting feature information is also in progress.

However, this deep learning network model standardization only describes a method for defining the deep learning network model and effectively exchanging weights, and it is independently composed without any relation to the multimedia stream to which these deep learning network models are applied. .

Therefore, in Fig. 1, the decoder 2 processes the bitstream 4, the deep learning processing unit 3 applies the deep learning network model 5, and the bitstream 4 and the deep learning network model 5 are It is a reality that is managed independently without any connection.

If the bitstream (4) and the deep learning network model (5) are managed independently in this way, it is not easy to manage which deep learning network model (5) is applied to which bitstream (4). In addition, if the deep learning network model 5 is applied in units of scenes or frames even within the same bitstream 4, the difficulty of management is bound to increase.

2 is a structure of a multimedia bitstream including a deep learning network model according to a preferred embodiment of the present invention for solving the problems of the prior art.

In general, multimedia data composed of video or audio consists of a very large amount of data. Therefore, in order to efficiently store and transmit multimedia data, compression is mainly performed.

Various international standards exist for compressing multimedia data, and the data compressed for transmission or storage is called a multimedia stream. For example, international standards for image data include H.264 and HEVC (Hi Efficiency Video Coding), and international standards for image data include Joint Picture Experts Group (JPEG) and JPEG2000. Also, there are MPEG-3 (MP3) and AAC (Advanced Audio Coding) as standards for voice data.

Most of these multimedia bitstream standards are divided into a payload area 10, which is an area for storing multimedia data itself, and an additional information area 20 including other information. In some cases, the additional information area 20 is transmitted empty. An unspecified area exists in the additional information area 20, and this area is an area that is vacated for a use that is not determined at the time the standard is defined. Therefore, it is an object of the present invention to transmit a deep learning network model using some of these areas. By inserting the deep learning network model in the additional information area 20, it is possible to simultaneously include the multimedia payload and the deep learning network model in one stream.

This additional information area 20 may include not only the deep learning network model defined in each multimedia standard, but also the deep learning network model standard defined previously. For example, the above-mentioned ONNX, NNEF, NNR format, etc. may be included in the additional information area 20, and a flag indicating them may also be added.

In the additional information region 20, a use case 21 indicating which deep learning model was used, a header 22 that can distinguish the start of each payload, and payloads 23, 24, 25 which are deep learning network model data ) may be included.

In addition, in video standards having NAL (Network Abstraction Layer) units such as H.264, HEVC, and VVC, a deep learning network model can be added by adding a new type of NAL unit to an unspecified area among NAL units. . As in the example of the additional information area, the NAL unit may include not only the deep learning network model format defined in each multimedia standard but also the previously defined deep learning network model standard. A bitstream definition for a deep learning network model may include a flag indicating which frame, which scene, etc., the deep learning network is applied to.

Table 1 below shows an example of applying the above method to HEVC.

The HEVC standard defines a Supplemental Enhancement Information (SEI) area for transmitting additional information. The present invention proposes a method of adding a deep learning network model to an unassigned value among payloadTypes of the SEI area. By assigning aaa, bbb, ccc, and ddd values to payloadType, deep learning models such as ONNX, NNEF, NNR, and DNM (Deeplearning Network Model) can be assigned, respectively. DNM stands for a newly defined deep learning network model.

Table 2 below shows the syntax structure when payloadType is ONNX.

ONNX(payloadSize) explanation use_case Super resolution, Noise Reduction, etc. frame_id Frame index to be applied vps_id vps index to be applied sps_id sps index to be applied pps_id pps index to be applied frame_resolution Image resolution after deep learning model is applied frame_rate Frame rate after the deep learning model is applied color_space Color space after deep learning model is applied ONNX_payload Payload in ONNX format. All data contained in ONNX format ONNX_model_data Data not included in ONNX format but required additionally

use_case shows the application to which the deep learning network model is applied. Applications such as super resolution to increase resolution or noise reduction to remove noise may be applicable.

frame_id, vps_id (video parameter set), sps_id (scene paraeter set), and pps_id (picture parameter set) indicate which frame, which scene, and which picture the deep learning network model is applied to.

frame_reolution, frame_rate, and color_space represent image properties after the deep learning network model is applied, that is, image properties such as frame resolution, frame rate, and color space.

ONNX_payload refers to the data that the ONNX standard has, and includes all data from the ONNX standard.

In addition to ONNX, use cases such as NNEF, NNR, and DNM may also have payload defined in the same format.

3 shows the structure of a multimedia bitstream to which the multimedia bitstream generating method according to a preferred embodiment of the present invention is applied.

Additional information data 35 of which payloadType is ONNX shows that use_case is SR (Super Resolution) and is applied to the immediately preceding INTRA frame 34 .

The additional information data 38 whose payloadType is NNEF indicates that use_case is NR (Noise Reduction) and is applied to the two INTER frames 36 and 37 .

In this way, various deep learning network models can be applied even within one stream, and it can be set so that the deep learning model can be applied only to a specific frame.

4 is a flowchart of a method for generating a multimedia bitstream including the additional information area as described above.

First, a payload type is set ( S10 ). Payload types are assigned to deep learning network models such as ONNX, NNEF, NNR, and DNM. It is possible to determine which deep learning network model is included in the additional information area by the payload type value.

Next, a use case field is generated (S20). The use case (use_case) indicates which application the deep learning network model is used for. For example, Super Resolution technology for increasing the resolution of SD (Standard Defenition) images into HD (High Definition) images or Noise_reduction for noise removal are examples of use cases.

When the use case field is generated, a location index field indicating a location to which the use case is to be applied is generated (S30). It may be applied to the entire video or various application ranges such as frame unit, scene unit, and picture unit may be configured.

A position to which the deep learning network is applied is created, and then an index of the parameter to be applied is generated (S40). The parameters include a video parameter set (vps), a scene parameter set (sps), and a picture parameter set (PPS).

Next, a field indicating the properties of the image after the deep learning network is applied is generated (S50). For example, frame resolution (frame_resolution), frame rate (frame_rate), color space (color space), etc. may indicate the properties of the image.

Finally, a payload field is generated (S60). For example, the ONNX payload includes the entire deep learning network model included in the ONNX standard. The ONNX standard includes all data such as structure and weights of deep learning models.

After creating the payload field, a new area for data that is not included in the ONNX model data but can be used later may be added.

The deep learning network model can be included in the SEI message, but can also be defined and configured as a separate NAL unit type.

The following Table 3 shows an example of newly allocating the deep learning network model fill to nal_uit_type in the additional information area of H..264.

In case of HEVC, nal_unit_type values are not defined from 48 to 63. Therefore, deep learning network models can be applied to this area. For example, 48 may indicate ONNX, 49 may indicate NNEF, 50 may indicate NNR, and 51 may indicate DNM. The configuration of each deep learning network model is as described above. For example, the configuration of onnx_rbsp() is similar to the configuration included in the SEI discussed above.

Table 4 below shows the case in which the deep learning network model is included in the h.264 sei message.

Table 6 below is an example of including the deep learning network model in a separate NAL unit type, not in the additional information area in H.264.

After all, the included format or content is not much different from the HEVC example. Accordingly, such a bitstream generation method is not limited to HEVC or H.264, and may be included in VVC or any video stream standard to be defined later in a similar form.

5 is a flowchart of a multimedia bitstream decoding method according to another preferred embodiment of the present invention.

A method of decoding a deep learning network model included in the additional information area or a separate NAL unit type is as follows.

First, the deep learning network model type is classified by payloadType (S110). The deep learning network model includes ONN, NNEF, NMR, and DNM, and is not limited to these models, and various types of deep learning models may be added.

After classifying the deep learning network model, which algorithm is applied using use_case? classify (S120). In addition to Super Resolution and Noise Reduction, various algorithms may be included.

Next, the location to which the deep learning network model is applied is divided (S130). In order for the deep learning network model to be applied not only to the entire image but also to the frame, picture, and scene units, it is possible to figure out which deep learning model is applied to which position through the location ID.

After the data is decoded, it can be found out by parsing the attribute field which attribute it will have ( S140 ). This field may include image resolution (frame_resolution), frame rate, color space, and the like.

Finally, deep learning association can be performed by classifying the data of the deep learning network model to be applied to the multimedia bitstream (S150)

6 is a structural diagram of an encoding apparatus according to a preferred embodiment of the present invention.

The encoding apparatus 100 according to the present invention includes an encoder 110 , a deep learning network model 120 , a memory 130 , and a bitstream generator 140 .

When a video/audio signal is input, the encoding apparatus 100 encodes it into compressed data through the encoder 110 . Then, the bitstream generator 140 includes a deep learning network model to be applied to the multimedia signal together with the encoded multimedia signal to generate a bitstream and store it in the memory 130 .

The bitstream generating unit 140 generates a payloadType field for classifying a deep learning network model. The payload type may be ONNX, NNEF, NMR, DNM, or the like.

Then, a use case field indicating the algorithm and application in which this deep learning network model is used is generated. Use cases may be applications such as Super Resolution, Noise Reduction, and the like.

Next, an index field indicating the location to which this deep learning network model is applied is created, and an attribute field indicating the properties of the multimedia file after the deep learning network model is applied is created. The index field indicating the position may be divided into units of frames, scenes, pictures, and videos. The property field corresponds to properties of a multimedia file such as frame resolution, frame rate, and color space.

Finally, the bitstream generation is completed by including the deep learning network model data according to the payload type in the payload. Deep learning network model data? New undefined data may be additionally included.

7 is a structural diagram of a decoding apparatus according to a preferred embodiment of the present invention.

The decoding apparatus 200 includes a bitstream parser 210 , a memory 220 , a decoder 230 , and a deep learning processing unit 240 .

The decoding device 200 parses the bitstream including the multimedia bitstream and the deep learning network model together through the bitstream parser 210, and then decodes it into a multimedia file through the decoder 230 and deep learning network model data. A video/audio file is finally generated after deep learning processing is performed in the deep learning processing unit 240 using

The bitstream parser 210 classifies the deep learning network model included in the bitstream in the following order.

First, the deep learning network model is classified by the payload type field. The deep learning network model may be ONNX, NNEF, NNR, DNM, or the like.

Next, we classify the use cases. A use case refers to an application to which a deep learning network model is applied. There are use cases such as Super Resolution to improve resolution and Noise Reduction to reduce noise.

After classifying the use cases, the index on the location in the video/audio file to which the deep learning network model is applied is distinguished. The location in the file may be divided into units such as frames, scenes, and pictures.

After the following deep learning network model is applied, the attribute field is parsed. Properties include frame resolution, frame rate, and color space.

Finally, it identifies the deep learning network model data itself loaded in the payload. This has the advantage of being able to apply an appropriate deep learning network model to the required location of the multimedia file.

According to the present invention as described above, the deep learning network model can be transmitted together with the multimedia stream to which the deep learning network model is applied, so it is easy to manage. It has the advantage of being able to apply an accurate deep learning network model.

The protection scope of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the protection scope of the present invention cannot be limited due to obvious changes or substitutions in the technical field to which the present invention pertains.

Claims (10)

generating a payload type field that identifies a deep learning network model;
generating a use case field that identifies an algorithm to be applied to multimedia data;
generating a location index field that specifies a location in the multimedia data to which the deep learning network model is applied;
generating a parameter index field that designates a parameter to be applied to the multimedia data;
generating an attribute field indicating an attribute of the multimedia data after the deep learning network model is applied; and
generating a payload field including deep learning network model data according to the payload type;
The method of claim 1, wherein the bitstream comprises:
A method of generating a bitstream, characterized in that it is included in the additional information area of the multimedia bitstream.
The method of claim 1, wherein the bitstream comprises:
A method of generating a bitstream, characterized in that it is included in the multimedia data area of the multimedia bitstream.
According to claim 1, wherein the location in the multimedia data to which the deep learning network model is applied,
A method for generating a bitstream, characterized in that the multimedia data is divided into frames, regions, or scenes.
According to claim 1, The parameter index to be applied to the multimedia data,
A method of generating a bitstream, characterized in that it is a video parameter set (VPS) index, a sequence parameter set (SPS) index, or a picture parameter set (PPS) index.
The method of claim 1 , wherein in the step of generating the use case field,
An algorithm to be applied to the multimedia data is a method for generating a bitstream, characterized in that Super Resolution or Noise Reduction.
According to claim 1, wherein the properties of the multimedia data after the deep learning network model is applied,
A method of generating a bitstream, comprising image resolution, frame rate or color space.
Memory; and
A bitstream generator including one or more processors to generate a bitstream including multimedia data and a deep learning network model to be applied to the multimedia data and store it in the memory;
The bitstream generator,
Creates a payload type field that distinguishes the deep learning network model, creates a use case field that identifies an algorithm to be applied to multimedia data, and creates an index field that specifies a location in the multimedia data to which the deep learning network model is applied and generating a field indicating the properties of the multimedia data after the deep learning network model is applied, and generating a payload field including deep learning network model data according to the payload type, the encoding device.
classifying the deep learning network model type by the payload type field;
classifying an algorithm to be applied to multimedia data according to a use case field;
Separating the location in the multimedia data to which the deep learning network model is applied by the location index field;
classifying the properties of the multimedia data after the deep learning network model is applied by the property field; and
A bitstream parsing method comprising a; classifying the deep learning network model data according to the payload type by the payload field.
Memory; and
A bitstream parsing unit for parsing a bitstream including a deep learning network model to be applied to multimedia data, including one or more processors, and storing it in the memory; including,
The bitstream parsing unit,
Classify the type of deep learning network model by the payload type field, classify the algorithm to be applied to multimedia data by the use case field, and classify the location in the multimedia data to which the deep learning network model is applied by the location index field , Decoding device, characterized in that the attribute of the multimedia data after the deep learning network model is applied by the attribute field, and the deep learning network model data according to the payload type is distinguished by the payload field.

Images