CN108629745B - Image processing method and device based on structured light and mobile terminal - Google Patents

Abstract

The application provides an image processing method and device based on structured light and a mobile terminal, wherein the method comprises the following steps: acquiring a visible light image of an imaging object; acquiring depth data indicated by a structured light image of an imaging subject; according to the depth data, identifying a first imaging area of an imaging object in the visible light image and identifying a second imaging area of an article worn by the imaging object; determining an operation area of image processing operation in the visible light image according to the relative positions of the first imaging area and the second imaging area; an image processing operation is performed on the operation area. The method can avoid blurring the object worn by the imaging object under the condition that the image processing operation is beautifying, thereby improving the display effect of the worn object. When the image processing operation is background blurring, the object worn by the false-blurring object can be avoided, and the visible light image blurring effect is improved. And meanwhile, the imaging effect of the imaging photo can be improved.

Description

Image processing method, device and mobile terminal based on structured light

technical field

The present application relates to the technical field of mobile terminals, and in particular, to an image processing method, device and mobile terminal based on structured light.

Background technique

With the continuous development of mobile terminal technology, more and more users choose to use mobile terminals to take pictures. And in order to achieve a better shooting effect, the image can also be processed by using a related image processing method.

However, in the actual image processing process, there is a situation in which the image effect is deteriorated after image processing. Taking background blurring processing as an example, when a user wears earrings, hairpins and other items, the user may want the items to be displayed clearly, but do not want the items to be blurred, but in actual operation, the accessories will be blurred and lose a lot of image details. . In addition, when the user turns on the camera to take a beauty photo, a similar situation will also occur.

Therefore, in the prior art, when image processing is performed, in some scenarios, the image effect is deteriorated after image processing, and the image processing effect is not good.

SUMMARY OF THE INVENTION

The present application aims to solve one of the technical problems in the related art at least to a certain extent.

To this end, the present application proposes an image processing method based on structured light, so as to avoid blurring the objects worn by the imaging object when the image processing operation is beauty, thereby improving the display effect of the worn objects. When the image processing operation is background blurring, it is possible to avoid falsely imagining objects worn by the imaging object, thereby improving the blurring effect of visible light images. At the same time, according to the depth data indicated by the structured light image, identify the first imaging area of the imaging object in the visible light image, and identify the second imaging area of the object worn by the imaging object, and then after determining the operation area and performing the image processing operation, on the one hand The imaging effect of the imaging photo is improved, and on the other hand, the accuracy of the depth data is improved, so that the image processing effect is better.

The present application proposes an image processing device based on structured light.

The present application proposes a mobile terminal.

The present application proposes a computer-readable storage medium.

In order to achieve the above purpose, an embodiment of the first aspect of the present application proposes an image processing method based on structured light, including:

obtaining a visible light image of the imaging object;

acquiring depth data indicated by the structured light image of the imaging object;

According to the depth data, identifying the first imaging area of the imaging object in the visible light image, and identifying the second imaging area of the article worn by the imaging object;

According to the relative positions of the first imaging area and the second imaging area, in the visible light image, determining an operation area of an image processing operation;

The image processing operation is performed on the operation area.

The structured light-based image processing method according to the embodiment of the present application obtains the visible light image of the imaging object; acquires depth data indicated by the structured light image of the imaging object; identifies the first imaging area of the imaging object in the visible light image according to the depth data, and Identifying the second imaging area of the article worn by the imaging object; determining the operation area of the image processing operation in the visible light image according to the relative positions of the first imaging area and the second imaging area; and performing the image processing operation on the operation area. In this way, when the image processing operation is to beautify the face, it is possible to avoid blurring the articles worn by the imaging object, so that the display effect of the worn articles can be improved. When the image processing operation is background blurring, it is possible to avoid falsely imagining objects worn by the imaging object, thereby improving the blurring effect of visible light images. At the same time, according to the depth data indicated by the structured light image, identify the first imaging area of the imaging object in the visible light image, and identify the second imaging area of the object worn by the imaging object, and then after determining the operation area and performing the image processing operation, on the one hand The imaging effect of the imaging photo is improved, and on the other hand, the accuracy of the depth data is improved, so that the image processing effect is better.

In order to achieve the above purpose, an embodiment of the second aspect of the present application proposes an image processing device based on structured light, including:

an acquisition module for acquiring a visible light image of an imaging object; acquiring depth data indicated by the structured light image of the imaging object;

an identification module, configured to identify, according to the depth data, a first imaging area of the imaging object in the visible light image, and a second imaging area of an article worn by the imaging object;

a determination module, configured to determine an operation area of an image processing operation in the visible light image according to the relative positions of the first imaging area and the second imaging area;

A processing module, configured to perform the image processing operation on the operation area.

The structured light-based image processing apparatus according to the embodiment of the present application obtains the visible light image of the imaging object; acquires depth data indicated by the structured light image of the imaging object; identifies the first imaging area of the imaging object in the visible light image according to the depth data, and Identifying the second imaging area of the article worn by the imaging object; determining the operation area of the image processing operation in the visible light image according to the relative positions of the first imaging area and the second imaging area; and performing the image processing operation on the operation area. In this way, when the image processing operation is to beautify the face, it is possible to avoid blurring the articles worn by the imaging object, so that the display effect of the worn articles can be improved. When the image processing operation is background blurring, it is possible to avoid falsely imagining objects worn by the imaging object, thereby improving the blurring effect of visible light images. At the same time, according to the depth data indicated by the structured light image, identify the first imaging area of the imaging object in the visible light image, and identify the second imaging area of the object worn by the imaging object, and then after determining the operation area and performing the image processing operation, on the one hand The imaging effect of the imaging photo is improved, and on the other hand, the accuracy of the depth data is improved, so that the image processing effect is better.

In order to achieve the above purpose, a third aspect of the present application provides a mobile terminal, including: an imaging sensor, a memory, a processor, and a computer program stored in the memory and running on the processor, the processor The visible light image or the structured light image acquired by the imaging sensor, when the program is executed, implements the structured light-based image processing method according to the embodiment of the first aspect of the present application.

In order to achieve the above purpose, a fourth aspect embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and is characterized in that, when the program is executed by a processor, the implementation of the first aspect of the present application is implemented. The described image processing method based on structured light.

Additional aspects and advantages of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of an image processing method based on structured light provided by Embodiment 1 of the present application;

FIG. 2 is a schematic structural diagram of an electronic device provided in Embodiment 2 of the present application;

3 is a schematic flowchart of an image processing method based on structured light provided by Embodiment 3 of the present application;

4 is a schematic structural diagram of an image processing apparatus based on structured light provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of another image processing device based on structured light provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a mobile terminal according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of another mobile terminal according to an embodiment of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to be used to explain the present application, but should not be construed as a limitation to the present application.

The structured light-based image processing method, device, and mobile terminal according to the embodiments of the present application are described below with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of an image processing method based on structured light provided by Embodiment 1 of the present application.

As shown in Figure 1, the image processing method based on structured light includes the following steps:

Step 101, acquiring a visible light image of the imaging object.

In the embodiment of the present application, the electronic device may include a visible light image sensor, and the visible light image may be obtained by imaging the visible light reflected by the imaging object based on the visible light image sensor in the electronic device. Specifically, the visible light image sensor may include a visible light camera, and the visible light camera may capture the visible light reflected by the imaging object for imaging to obtain a visible light image.

Step 102: Acquire depth data indicated by the structured light image of the imaging object.

In this embodiment of the present application, the electronic device may further include a structured light image sensor, and a structured light image of the imaging object may be acquired based on the structured light image sensor in the electronic device. Specifically, the structured light image sensor may include a laser light and a laser camera. Pulse Width Modulation (PWM for short) can modulate the laser light to emit structured light, the structured light is irradiated to the imaging object, and the laser camera can capture the structured light reflected by the imaging object for imaging to obtain a structured light image. The depth engine can calculate and obtain the depth data corresponding to the imaging object according to the structured light image. Specifically, the depth engine demodulates the phase information corresponding to the deformed position pixels in the structured light image, converts the phase information into height information, and determines the object according to the height information. The depth data corresponding to the subject.

Step 103 , according to the depth data, identify the first imaging area of the imaging object in the visible light image, and identify the second imaging area of the article worn by the imaging object.

In the embodiment of the present application, in order to avoid that the objects worn by the imaging object, such as earrings, earrings, hairpins, etc., are falsely false when the background blurring operation is performed on the visible light image, resulting in poor visible light image blurring effect, or, in order to Avoid blurring the objects worn by the imaging object, such as necklaces, forehead ornaments, nose rings, etc., worn by the blurred imaging object when performing the beautification operation on the visible light image. In the embodiment of the present application, the imaging object in the visible light image can be identified according to the depth data. the first imaging area, and the second imaging area for identifying the article worn by the imaging subject.

As a possible implementation manner, after acquiring the depth data indicated by the structured light image of the imaging object, it can be determined whether the object is the foreground or the background according to the depth data of each object in the structured light image. Generally speaking, the depth data indicates that the object is closer to the plane where the camera is located, and when the depth value is small, it can be determined that the object is the foreground, otherwise, the object is the background. Further, the foreground part and the background part of the visible light image can be determined according to each object. After determining the foreground part and the background part of the visible light image, whether the imaging object is a human body can be identified in the foreground part based on the human body detection algorithm, and if so, the part surrounded by the outline of the human body is used as the first imaging area. Specifically, the edge pixels of the image and the pixels whose pixel value difference is less than a preset threshold value, that is, the pixel points with similar pixel values, can be extracted from the visible light image to obtain the outline of the human body. part as the first imaging area.

It should be noted that the objects worn by the imaging subject may be inside the first imaging area, such as necklaces, ear studs, nose rings, forehead ornaments, etc., or, the distance between the objects worn by the imaging subject and the first imaging area may be less than a preset distance , such as earrings, earrings, hairpins, etc. Therefore, in the embodiment of the present application, the second imaging region can be identified within the first imaging region and the foreground portion whose distance from the first imaging region is less than the preset distance, thereby saving computation and improving processing efficiency. As a possible implementation method, the depth data of various accessories can be collected as sample data, and the sample data can be used to train a recognition model for recognizing the accessories. thing. After the object worn by the imaging object is identified, the part surrounded by the outline of the object worn by the imaging object may be used as the second imaging region.

It can be understood that there is a color difference between the second imaging area and the skin color, and the depth corresponding to the first pixel point closest to the second imaging area in the first imaging area is the same as the depth corresponding to the second imaging area closest to the first imaging area in the second imaging area. The difference between the depths corresponding to the pixels is within the preset depth range. If the number of first pixels is multiple, the depth corresponding to the first pixel can be the average of the depths corresponding to multiple first pixels. Similarly, if the number of second pixels is multiple, then The depth corresponding to the second pixel point may be an average value of the depth corresponding to a plurality of second pixel points.

Step 104 , according to the relative positions of the first imaging area and the second imaging area, in the visible light image, determine the operation area of the image processing operation.

In this embodiment of the present application, the user can perform image processing operations on the visible light image according to their own needs. Among them, the image processing operations may be processing operations such as background blur, beauty (acne removal, face reduction, face brightening, skin resurfacing, etc.). It can be understood that in the case of different image processing operations, the operation areas may be the same or different.

For example, when the image processing operation is beauty, and the relative position is that the first imaging area includes the second imaging area, or the first imaging area and the second imaging area overlap, for example, the object worn by the imaging object is a necklace, Nose rings, forehead ornaments, etc. At this time, in order to avoid blurring the worn objects when the imaged object is beautified, the operation area may be the part of the first imaging area except the second imaging area.

Or, when the image processing operation is beauty, and the relative position is that the first imaging area is adjacent to the second imaging area, for example, the objects worn by the imaging subject are hairpins, earrings, etc. Beauty does not affect the display effect of the object worn by the imaging object, therefore, the operation area may be the first imaging area in the visible light image.

For another example, when the image processing operation is background blur, and the relative position is that the first imaging area includes the second imaging area, for example, the objects worn by the imaging object are necklaces, nose rings, forehead ornaments, etc., at this time, the imaging The background blurring of the part other than the first imaging area of the object does not affect the display effect of the object worn by the imaging object. Therefore, the operation area may be the part other than the first imaging area in the visible light image.

Or, when the image processing operation is background blur, and the relative position is that the first imaging area and the second imaging area are adjacent or overlapping, for example, the objects worn by the imaging object are hairpins, earrings, etc. When the part outside the first imaging area of the imaging object is blurred, the objects worn by the imaging object will be falsely false, which will seriously affect the display effect of the objects worn. A portion other than the first imaging area and the second imaging area.

Step 105, perform image processing operation on the operation area.

In this embodiment of the present application, after the operation area is determined, an image processing operation may be performed on the operation area.

For example, when the user wears a nose ring, at this time, if the user wants to beautify the visible light image, he can beautify the area of the body except the nose ring. If the user wants to blur the visible light image , the area outside the user's body can be blurred.

The image processing method based on structured light in the embodiments of the present application can be configured in an image processing apparatus based on structured light, and the image processing apparatus based on structured light can be applied to electronic equipment.

As a possible implementation manner, reference may be made to FIG. 2 for the structure of the electronic device, which is a schematic structural diagram of the electronic device provided in Embodiment 2 of the present application.

As shown in FIG. 2 , the electronic device includes: a laser camera, a flood light, a visible light camera, a laser light, and a microprocessor (Microcontroller Unit, MCU for short). Among them, MCU includes PWM, depth engine, bus interface and random access memory RAM. In addition, the electronic device further includes a processor, the processor has a trusted execution environment, and the MCU is dedicated hardware for the trusted execution environment, and executes the trusted application program to run in the trusted execution environment; the processor may also have a common execution environment, The normal execution environment and the trusted execution environment are isolated from each other.

It should be noted that those skilled in the art can know that the method corresponding to FIG. 1 is not only applicable to the electronic device shown in FIG. 2 , the electronic device shown in FIG. 2 is only used as a schematic description, and the method corresponding to FIG. 1 can also be used for other An electronic device having a trusted execution environment and dedicated hardware for the trusted execution environment is not limited in this embodiment.

Among them, PWM is used to modulate floodlights to emit infrared light, and to modulate laser lights to emit structured light; laser cameras are used to collect structured light images or visible light images of imaging objects; depth engines are used to, based on structured light images, The depth data corresponding to the imaging object is obtained by calculation; the bus interface is used to send the depth data to the processor, and the trusted application program running on the processor uses the depth data to perform corresponding operations. The bus interfaces include: Mobile Industry Processor Interface (MIPI for short), I2C synchronous serial bus interface, and Serial Peripheral Interface (SPI for short).

In the embodiments of the present application, the trusted execution environment is a secure area on the main processor of an electronic device (including smart phones, tablet computers, etc.), which can ensure the security of code and data loaded into the environment compared to a common execution environment security, confidentiality and integrity. The trusted execution environment provides an isolated execution environment, and the provided security features include: isolated execution, integrity of trusted applications, confidentiality of trusted data, and secure storage. In conclusion, the execution space provided by TEE provides a higher level of security than common mobile operating systems such as ISO, Android, etc.

The structured light-based image processing method of this embodiment obtains the visible light image of the imaging object; acquires depth data indicated by the structured light image of the imaging object; identifies the first imaging area of the imaging object in the visible light image according to the depth data, and identifies imaging the second imaging area of the article worn by the object; determining the operation area of the image processing operation in the visible light image according to the relative positions of the first imaging area and the second imaging area; and performing the image processing operation on the operation area. In this way, when the image processing operation is to beautify the face, it is possible to avoid blurring the articles worn by the imaging object, so that the display effect of the worn articles can be improved. When the image processing operation is background blurring, it is possible to avoid falsely imagining objects worn by the imaging object, thereby improving the blurring effect of visible light images. At the same time, according to the depth data indicated by the structured light image, identify the first imaging area of the imaging object in the visible light image, and identify the second imaging area of the object worn by the imaging object, and then after determining the operation area and performing the image processing operation, on the one hand The imaging effect of the imaging photo is improved, and on the other hand, the accuracy of the depth data is improved, so that the image processing effect is better.

In this embodiment of the present application, the second imaging area may be adjacent to the target sub-area of the first imaging area, or the second imaging area may be inside the target sub-area, where the target sub-area is used for the neck, ear, Imaging of the nose, lips or forehead.

It should be noted that, in this embodiment of the present application, the target sub-region can not only be used to image the neck, ears, nose, lips or forehead, but also the target sub-region can also be used to image fingers, wrists, navels, ankles, etc. Partial imaging. For example, when the user wears a ring on his hand and a watch on his wrist, when the user takes a selfie, if the user makes a "yeah" action beside his face, and the image processing operation is beauty, the user also does not want the ring. And the watch is blurred out, therefore, during beauty, the operation area is the part of the first imaging area other than the second imaging area.

As a possible implementation manner, after step 105, the expressive power of the first imaging area may also be enhanced. Specifically, sharpening processing, adjustment of contrast and/or saturation, etc. may be performed in the second area, thereby enhancing the display effect of the worn item.

As a possible implementation, referring to FIG. 3 , on the basis of the embodiment shown in FIG. 1 , step 104 may specifically include the following sub-steps:

Step 201 , among various image processing operations, determine an image processing operation to be performed.

In the embodiment of the present application, the user can determine the image processing operation to be performed from various image processing operations according to his own needs, for example, background blur, beauty, and the like.

As a possible implementation manner, the electronic device may have controls for different image processing operations, and the user can determine the image processing operations to be performed by triggering the corresponding controls.

Step 202: Determine the corresponding operation area in the visible light image according to the image processing operation to be performed and the relative position.

As a possible implementation, the mapping relationship between image processing operations, relative positions and operation areas can be pre-configured, and a mapping table can be configured according to the mapping relationship. The mapping table is used to indicate the mapping between image processing operations, relative positions and operation areas. relation. After the image processing operation to be performed is determined, the mapping table can be queried according to the image processing operation to be performed and the relative position to determine the operation area, and the operation is simple and easy to implement.

The image processing method based on structured light in this embodiment determines the image processing operation to be performed in various image processing operations; and determines the corresponding operation area in the visible light image according to the image processing operation to be performed and the relative position , the operation is simple and easy to implement.

In order to realize the above embodiments, the present application also proposes an image processing device based on structured light.

FIG. 4 is a schematic structural diagram of an image processing apparatus based on structured light according to an embodiment of the present application.

As shown in FIG. 4 , the structured light-based image processing apparatus 100 includes: an acquisition module 110 , an identification module 120 , a determination module 130 , and a processing module 140 . in,

The acquiring module 110 is configured to acquire the visible light image of the imaging object; acquire the depth data indicated by the structured light image of the imaging object.

The identification module 120 is configured to identify, according to the depth data, the first imaging area of the imaging object in the visible light image, and the second imaging area of the article worn by the imaging object.

As a possible implementation manner, the identification module 120 is specifically configured to identify the foreground part and the background part in the visible light image according to the depth data, and the depth of the foreground part is smaller than the depth of the background part; in the foreground part, identify whether the imaging object is a human body ; If the imaging object in the foreground part is a human body, in the foreground part, the part surrounded by the outline of the human body is used as the first imaging area; inside the first imaging area, and the foreground part whose distance from the first imaging area is less than the preset distance, identify Obtain a second imaging area; wherein, the second imaging area has a color difference with the skin color, and the depth corresponding to the first pixel point closest to the second imaging area in the first imaging area is the closest to the first imaging area in the second imaging area The difference between the depths corresponding to the second pixel points is within the preset depth range.

In the embodiment of the present application, the second imaging area is adjacent to the target sub-area of the first imaging area, or is inside the target sub-area; the target sub-area is used to image the neck, ear, nose, lips or forehead.

The determining module 130 is configured to determine the operation area of the image processing operation in the visible light image according to the relative positions of the first imaging area and the second imaging area.

As a possible implementation manner, the determining module 130 is specifically configured to determine the image processing operations to be performed in various image processing operations; according to the image processing operations to be performed and the relative positions, determine the corresponding image processing operations in the visible light image operating area.

Optionally, the determining module 130 is further configured to obtain a pre-configured mapping table, where the mapping table is used to indicate the mapping relationship between image processing operations, relative positions and operation regions; image processing operations performed as required, and relative position query Mapping table to determine the operating area.

As another possible implementation manner, the determining module 130 is specifically configured to, if the relative position is that the first imaging area includes the second imaging area, or the first imaging area and the second imaging area overlap, and the image processing operation is cosmetic If the relative position is that the first imaging area includes the second imaging area, if the image processing operation is background blur, the operation area Including: the part of the visible light image other than the first imaging area; if the relative position is that the first imaging area is adjacent to the second imaging area, in the case that the image processing operation is beauty, the operation area includes: the first imaging area; If the relative position is that the first imaging area and the second imaging area are adjacent or overlap, and in the case that the image processing operation is background blurring, the operation area includes: the visible light image except the first imaging area and the second imaging area. part.

The processing module 140 is configured to perform image processing operations on the operation area.

Further, in a possible implementation manner of the embodiment of the present application, referring to FIG. 5 , on the basis of the embodiment shown in FIG. 4 , the image processing apparatus 100 based on structured light may further include: an enhancement module 150 .

The enhancement module 150 is configured to enhance the expressive power of the second imaging area.

As a possible implementation manner, the enhancement module 150 is specifically configured to perform sharpening, and adjust contrast and/or saturation in the second region.

It should be noted that the foregoing explanations on the embodiment of the image processing method based on structured light are also applicable to the image processing apparatus 100 based on structured light in this embodiment, and are not repeated here.

The structured light-based image processing device of this embodiment acquires the visible light image of the imaging object; acquires depth data indicated by the structured light image of the imaging object; identifies the first imaging area of the imaging object in the visible light image according to the depth data, and recognizes imaging the second imaging area of the article worn by the object; determining the operation area of the image processing operation in the visible light image according to the relative positions of the first imaging area and the second imaging area; and performing the image processing operation on the operation area. In this way, when the image processing operation is to beautify the face, it is possible to avoid blurring the articles worn by the imaging object, so that the display effect of the worn articles can be improved. When the image processing operation is background blurring, it is possible to avoid falsely imagining objects worn by the imaging object, thereby improving the blurring effect of visible light images. At the same time, according to the depth data indicated by the structured light image, identify the first imaging area of the imaging object in the visible light image, and identify the second imaging area of the object worn by the imaging object, and then after determining the operation area and performing the image processing operation, on the one hand The imaging effect of the imaging photo is improved, and on the other hand, the accuracy of the depth data is improved, so that the image processing effect is better.

In order to realize the above embodiments, the present application also proposes a mobile terminal.

FIG. 6 is a schematic structural diagram of a mobile terminal according to an embodiment of the present application.

In this embodiment, the mobile terminal includes but is not limited to devices such as a mobile phone and a tablet computer.

As shown in FIG. 6 , the mobile terminal includes: an imaging sensor 210 , a memory 220 , a processor 230 and a computer program (not shown in FIG. 6 ) stored on the memory 220 and executable on the processor 230 , the processor 230 According to the visible light image or the structured light image acquired from the imaging sensor 210, when the program is executed, the structured light-based image processing method as proposed in the foregoing embodiments of the present application is implemented.

In a possible implementation manner of the embodiment of the present application, referring to FIG. 7 , on the basis of the embodiment shown in FIG. 6 , the mobile terminal may further include: a microprocessor chip MCU2140 .

The processor 230 has a trusted execution environment, and the program runs in the trusted execution environment.

The MCU 240 is dedicated hardware of the trusted execution environment, connected with the imaging sensor 210 and the processor 230, and is used to control the imaging sensor 210 to perform imaging, send the visible light image obtained by imaging to the processor 230, and transmit the structured light image obtained by imaging. The indicated depth data is sent to processor 230 .

In a possible implementation manner of this embodiment, the imaging sensor 210 may include: an infrared sensor, a structured light image sensor, and a visible light image sensor.

The infrared sensor includes a laser camera and a floodlight; the structured light image sensor includes a laser light and a laser camera shared with the infrared sensor, and the visible light image sensor includes a visible light camera.

In a possible implementation manner of this embodiment, the MCU 240 includes a PWM, a depth engine, a bus interface, and a random access memory RAM.

Among them, PWM is used to modulate floodlights to emit infrared light, and to modulate laser lights to emit structured light;

Laser camera for collecting structured light images of imaging objects;

a depth engine, used to calculate and obtain the depth data corresponding to the imaging object according to the structured light image; and

The bus interface is used to send the depth data to the processor 230, and the trusted application program running on the processor 230 uses the depth data to perform corresponding operations.

For example, the first imaging area of the imaged object in the visible light image can be identified according to the depth data, and the second imaging area of the article worn by the imaging object can be identified, and according to the relative positions of the first imaging area and the second imaging area, in the visible light image , to determine the operation area of the image processing operation, so as to perform the image processing operation on the operation area, the specific process can refer to the above-mentioned embodiment, which will not be repeated here.

In order to realize the above-mentioned embodiments, the present application also proposes a computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the structured light-based image as proposed in the foregoing embodiments of the present application is realized. Approach.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless expressly and specifically defined otherwise.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing custom logical functions or steps of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of this application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one of the following techniques known in the art, or a combination thereof: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those skilled in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be stored in a computer-readable storage medium. When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (12)

1. A method for structured light based image processing, the method comprising the steps of:

acquiring a visible light image of an imaging object;

acquiring depth data indicative of a structured light image of the imaging subject;

identifying a first imaging region of the imaging subject in the visible light image and a second imaging region of an item worn by the imaging subject from the depth data;

determining an operation area of an image processing operation in the visible light image according to the relative position of the first imaging area and the second imaging area, wherein the operation area is related to the relative position and the image processing operation; if the image processing operation is background blurring, if the relative position is that the first imaging region includes the second imaging region, the operation region includes a portion of the visible light image other than the first imaging region; if the relative position is that the first imaging area is adjacent to or overlapped with the second imaging area, the operation area comprises a part of the visible light image except the first imaging area and the second imaging area;

and executing the image processing operation on the operation area.

2. The image processing method according to claim 1, wherein the image processing operation is plural, and the determining an operation region of the image processing operation in the visible light image according to the relative positions of the first imaging region and the second imaging region includes:

determining an image processing operation to be executed in a plurality of image processing operations;

and determining a corresponding operation area in the visible light image according to the image processing operation to be executed and the relative position.

3. The image processing method according to claim 2, wherein the determining a corresponding operation area in the visible light image according to the image processing operation to be performed and the relative position comprises:

acquiring a pre-configured mapping table, wherein the mapping table is used for indicating the mapping relation among the image processing operation, the relative position and the operation area;

and inquiring the mapping table according to the image processing operation to be executed and the relative position to determine the operation area.

4. The image processing method according to claim 1, wherein determining an operation region of an image processing operation in the visible light image according to the relative positions of the first imaging region and the second imaging region comprises:

if the relative position is that the first imaging region includes the second imaging region, or the first imaging region overlaps with the second imaging region, if the image processing operation is a color beautification operation, the operation region includes: a portion of the first imaging region other than the second imaging region;

if the relative position is that the first imaging area is adjacent to the second imaging area, the operation area includes, when the image processing operation is a beauty operation: the first imaging region.

5. The method of any of claims 1-4, wherein identifying a first imaging region of the imaging subject in the visible light image and identifying a second imaging region of an article worn by the imaging subject based on the depth data comprises:

according to the depth data, a foreground part and a background part are identified and obtained in the visible light image, and the depth of the foreground part is smaller than that of the background part;

identifying whether the imaging object is a human body in the foreground part;

if the foreground part imaging object is a human body, taking a part surrounded by the human body outline as the first imaging area in the foreground part;

identifying to obtain the second imaging area in the first imaging area and a foreground part with a distance from the first imaging area smaller than a preset distance; the second imaging area and the skin color have color difference, the depth corresponding to a first pixel point in the first imaging area closest to the second imaging area is within a preset depth range, and the difference between the depth corresponding to a second pixel point in the second imaging area closest to the first imaging area is within the preset depth range.

6. The image processing method according to claim 5,

the second imaging region is adjacent to or inside a target sub-region of the first imaging region; the target sub-region is used to image the neck, ears, nose, lips, or forehead.

7. The image processing method according to any one of claims 1 to 4, further comprising, after performing the image processing operation on the operation region:

enhancing the expressive power of the second imaging area.

8. The image processing method according to claim 7, wherein the enhancing of the expressive power of the second imaging region comprises:

sharpening, adjusting contrast and/or saturation within the second imaging region.

9. A structured light based image processing apparatus, comprising:

the acquisition module is used for acquiring a visible light image of an imaging object; acquiring depth data indicative of a structured light image of the imaging subject;

the identification module is used for identifying a first imaging area of the imaging object in the visible light image and identifying a second imaging area of an article worn by the imaging object according to the depth data;

a determining module, configured to determine an operation area for an image processing operation in the visible light image according to a relative position of the first imaging area and the second imaging area, where the operation area is related to the relative position and the image processing operation; if the image processing operation is background blurring, if the relative position is that the first imaging region includes the second imaging region, the operation region includes a portion of the visible light image other than the first imaging region; if the relative position is that the first imaging area is adjacent to or overlapped with the second imaging area, the operation area comprises a part of the visible light image except the first imaging area and the second imaging area;

and the processing module is used for executing the image processing operation on the operation area.

10. A mobile terminal, comprising: an imaging sensor, a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the structured light based image processing method according to any one of claims 1 to 8 when executing the program based on a visible light image or a structured light image acquired from the imaging sensor.

11. The mobile terminal according to claim 10, wherein the mobile terminal further comprises a micro processing chip MCU; the processor has a trusted execution environment, the program running in the trusted execution environment;

the MCU is special hardware of the trusted execution environment, is connected with the imaging sensor and the processor, and is used for controlling the imaging sensor to image, sending visible light images obtained by imaging to the processor, and sending depth data indicated by structured light images obtained by imaging to the processor.

12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the structured light based image processing method according to any one of claims 1 to 8.

Images