KR20220081018A - Hair styling service providing system and hair image extraction apparatus and method for extracting hair image using the same - Google Patents

Abstract

The present invention relates to a hair image extraction method of a hair styling service providing system. The hair image extraction method includes the steps of: receiving an image of a subject by a control unit; outputting an image of the subject by the control unit to a display panel; receiving an input, outputting, by the control unit, the image of the extraction region to the display panel, receiving a drag selection for designating a hair region in the image of the extraction region outputted by the input unit, by the control unit grabcut extracting the hair region by removing a background part other than the drag-selected hair region from the extraction region using a (GrabCut) algorithm, and the control unit performs alpha matting on the boundary of the extracted hair region It may include the step of expressing the hair in the hair region finely by applying it.

Description

HAIR STYLING SERVICE PROVIDING SYSTEM AND HAIR IMAGE EXTRACTION APPARATUS AND METHOD FOR EXTRACTING HAIR IMAGE USING THE SAME

The present invention relates to a hair styling service providing system, a hair image extraction apparatus, and a hair image extraction method using the same.

Virtual hair styling service is provided. The virtual hair styling service is a service for superimposing and outputting a virtual hair image object created on a customer's face photo. As the hair image object, only the hair part was extracted using a binary mask from the image of a person with a hairstyle to be extracted, or one designed by a person using computer graphics was used.

Korean Patent Publication No. 10-1664940

A virtual hair styling service using a virtual hair image has a problem in that it takes a lot of time and money to produce a virtual hair image, and there is a need to solve this problem.

In addition, since the virtual hair image extracted from the human head image is extracted based on a simple binary mask or binary mask, there is a problem in that the quality of the hair image is remarkably low.

In addition, there is a problem in that the virtual hair image does not naturally overlap on the customer's head or face, causing awkwardness, and the need to solve this problem is being raised.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The present specification provides a hair image extraction method of a hair styling service providing system. The hair image extraction method includes the steps of: receiving an image of a subject by a control unit; outputting an image of the subject by the control unit to a display panel; receiving an input, outputting, by the control unit, the image of the extraction region to the display panel, receiving a drag selection for designating a hair region in the image of the extraction region outputted by the input unit, by the control unit grabcut extracting the hair region by removing a background part other than the drag-selected hair region from the extraction region using a (GrabCut) algorithm, and the control unit performs alpha matting on the boundary of the extracted hair region It may include the step of expressing the hair in the hair region finely by applying it.

The hair image extraction method and other embodiments of the hair styling service providing system may include the following features.

In some embodiments, the applying of the alpha matting may be a step in which the controller synthesizes the hair of the hair region with a background image based on the alpha value of each pixel in the hair region. have.

According to an embodiment, the applying of the alpha matting may include generating, by the controller, a mask having a value of 0 to 1 using the transparency value of each pixel, and selecting the hair region based on the generated mask. It may include generating a composite image synthesized with the background image.

는

의 공식을 통해서 산출되되,

는 투명도 값,

는 전경 영상,

는 배경 영상,

는 합성할 배경 영상이고,

는

및

의 공식을 통해서 산출될 수 있다.According to an embodiment, each pixel value of the composite image

Is

It is calculated through the formula of

is the transparency value,

is the foreground image,

is the background image,

is the background image to be synthesized,

Is

and

It can be calculated through the formula of

According to an embodiment, the method for extracting the hair image includes receiving, by the input unit, a drag selection for designating a part to be deleted from the extracted image of the hair region, and the controller designated as the part to be deleted from the image of the hair region. It may further include the step of deleting the part.

According to an embodiment, the method for extracting the hair image includes: superimposing, by the controller, the extracted image of the hair region on the hair part of the face image of the subject or another subject to output an overlapping image to a display panel; receiving a touch manipulation for transforming the hair region in the superimposed image, transforming the superposed image according to the touch input manipulation for the hair region by the controller, and outputting the transformed superposed image by the display panel It may include further steps.

On the other hand, the present specification provides a hair image extraction device. The hair image extraction device includes a display panel for outputting an input image of a subject and an image of an extraction region, and an input unit for receiving a selection for designating an extraction region including hair in the output image of the subject and a drag selection for designating a hair region , and a GrabCut algorithm to extract the hair region by removing a background part other than the drag-selected hair region from the extraction region, and apply alpha matting to the boundary of the extracted hair region Thus, it may include a control unit for finely expressing the hair in the hair region.

The hair image extraction device and other embodiments may include the following features.

According to an embodiment, the control unit synthesizes the hair of the hair region with a background image based on the alpha value of each pixel in the hair region, and a value of 0 to 1 using the transparency value of each pixel may be generated, and a synthesized image obtained by synthesizing the hair region with the background image may be generated based on the generated mask.

는

의 공식을 통해서 산출되되,

는 투명도 값,

는 전경 영상,

는 배경 영상,

는 합성할 배경 영상이고,

는

및

의 공식을 통해서 산출될 수 있다.According to an embodiment, each pixel value of the composite image

Is

It is calculated through the formula of

is the transparency value,

is the foreground image,

is the background image,

is the background image to be synthesized,

Is

and

It can be calculated through the formula of

According to an embodiment, the input unit may receive a drag selection for designating a part to be deleted from the extracted image of the hair region, and the controller may delete a part designated as the part to be deleted from the image of the hair region.

The hair styling service providing system, the hair image extraction apparatus, and the hair image extraction method using the same according to the embodiments disclosed in the present specification can extract only high-quality hair images from a human head image, so it takes time to produce a virtual hair image. It has the effect of saving time and cost.

In addition, the hair styling service providing system, the hair image extraction apparatus, and the hair image extraction method using the same according to the embodiments disclosed in the present specification are effective in extracting natural and detailed hair images in a short time.

In addition, the hair styling service providing system, the hair image extraction apparatus, and the hair image extraction method using the same according to the embodiments disclosed in the present specification can extract a high-quality hair image from an image taken directly from the customer's head, so that the customer's hair Alternatively, there is an effect of providing a hair styling simulation with a hair image that is naturally superimposed on the face.

On the other hand, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be able

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and serve to further understand the technical spirit of the present invention together with specific details for carrying out the invention, so the present invention is described in such drawings It should not be construed as being limited only to the matters.
1 is a diagram showing a schematic configuration of a hair styling service providing system according to an embodiment of the present invention.
2 is a diagram illustrating a configuration of a mirror display device according to an embodiment of the present invention.
3 is a diagram illustrating the registration contents of the customer management table.
4 is a flowchart showing the registration processing of personal information by the personal information management server.
Fig. 5 is a flowchart showing the updating processing of personal information by the personal information management server.
6 and 7 are schematic diagrams for explaining the hair design adjustment function realized by the hair style AP installed in the mirror display device.
Fig. 8 is a schematic diagram for explaining the hair color adjustment function realized by the hair style AP provided in the mirror display device.
Fig. 9 is a flowchart showing a hair style adjustment process executed by a control unit of the mirror display device;
10 is a flowchart illustrating a method of providing a hair styling service in a hair styling service system including a mirror display device.
11 is a diagram showing a schematic configuration of an apparatus for extracting a hair image according to an embodiment of the present invention.
12 is a schematic diagram for explaining a hair image extraction method according to an embodiment of the present invention.
13 is a flowchart illustrating a hair image extraction method of the hair image extraction apparatus according to an embodiment of the present invention.

The technology disclosed in this specification may be applied to a technology for providing a hair image extraction technology and a service to which the technology is applied. However, the technology disclosed in the present specification is not limited thereto, and may be applied to all devices and methods to which the technical idea of the technology may be applied.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments, and are not intended to limit the spirit of the technology disclosed herein. In addition, the technical terms used in this specification should be interpreted in the meaning generally understood by those of ordinary skill in the art to which the technology disclosed in this specification belongs, unless otherwise defined in this specification. It should not be construed in a very comprehensive sense or in an excessively reduced meaning. In addition, when the technical term used in this specification is an erroneous technical term that does not accurately express the spirit of the technology disclosed in this specification, a technical term that can be correctly understood by those of ordinary skill in the art to which the technology disclosed in this specification belongs should be understood and replaced with In addition, general terms used in this specification should be interpreted according to the definition in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

As used herein, terms including an ordinal number such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted.

In addition, in describing the technology disclosed in the present specification, if it is determined that a detailed description of a related known technology may obscure the gist of the technology disclosed in this specification, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the technology disclosed in this specification, and should not be construed as limiting the spirit of the technology by the accompanying drawings.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing a schematic configuration of a system for providing a hair styling service according to an embodiment of the present invention.

Referring to FIG. 1 , a hair styling service providing system 1000 includes a mirror display device 100 installed in a target facility such as a beauty salon and a customer who uses the mirror display device 100 through a communication network N (hereinafter “customer”). The personal information management server 200 that manages personal information of), the advertisement providing server 300 that provides advertisement information judged to be beneficial to customers, etc., beauty such as shampoo and cosmetic material provided to customers A product server 400 that manages related products (hereinafter referred to as “products”) is provided. In addition, the communication network N includes a communication network capable of exchanging information between the mirror display device 100 , the personal information management server 200 , the advertisement providing server 300 , and the product server 400 . The networks disclosed herein include, for example, wireless networks, wired networks, public networks such as the Internet, private networks, Global System for Mobile communication network (GSM) networks, and general packet wireless networks (General). Packet Radio Network (GPRN), Local Area Network (LAN), Wide Area Network (WAN), Metropolitan Area Network (MAN), Cellular Network, Public Switched Telephone Network (Public Switched Telephone Network) ; PSTN), Personal Area Network, Bluetooth, Wi-Fi Direct, Near Field communication, Ultra-Wide band, a combination thereof, or any may be other networks of, but not limited to.

2 is a diagram illustrating a configuration of a mirror display apparatus 100 according to an embodiment of the present invention.

Referring to FIG. 2 , the mirror display apparatus 100 includes a half mirror 101 , a display panel 102 , a camera 110 , and a control computer 120 .

The rear surface of the half mirror 101 is disposed to face the display screen 103 of the display panel 102 . The half mirror 101 is formed as a reflective material on a flat transparent substrate such as plastic or glass by, for example, sputtering of a metal film such as silver and aluminum. When the reflective material is formed, the amount of deposition of the reflective material is adjusted to be smaller than that of a mirror that completely reflects light to form a transflective reflective surface 101S through which a part of the light can be transmitted. In addition, the semi-transmissive reflective surface 101S can be formed of either a multiphase film or a single-layer film, and various methods can be adopted for its manufacturing method.

Here, light at the front side of the transflective surface 101S is partially reflected, while light at the rear side of the transflective surface 101S is partially transmitted forward through the transflective surface 101S. Accordingly, the half-mirror 101 serves as a general mirror in a state where external light is strong from the front, while the light to be displayed on the display panel 102 disposed on the rear side of the corresponding half-mirror 101, for example. In this state, it can function as a display for projecting various images, which are examples of light to be displayed through the half mirror 101 .

Therefore, a subject (here, a customer) positioned in front of the half mirror 101 can use the half mirror 101 as a mirror to check his or her appearance (image), and at the same time use the half mirror 101 as a display. Thus, for example, it is possible to view an image of a customer, advertisement information, or various contents.

The display panel 102 is a conventional panel type display having a display screen 103 , and may include, for example, a self-luminous display panel such as an organic light emitting diode (OLED) display panel and a plasma display panel. In addition, a backlight type or reflective type display panel may be used instead of the self-luminous type display panel.

The camera 110 is, for example, a camera using a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and may be installed, for example, in one of the half mirrors 101 . The camera 110 takes a non-contact image (eg, face) of a customer located in front of the half mirror 101 . The photographed image of the customer is transmitted to the control computer 120 .

The control computer 120 includes a control unit 121 equipped with a CPU, etc., an optical disk device, a semiconductor memory, a storage device 122 including a hard disk device, an input unit 123 including a keyboard, a mouse, a touch panel, etc., an ISDN modem, The communication unit 124 is provided with an ADSL modem, a cable modem, an optical modem, a software modem, and the like. The control unit 121 centrally controls each part of the control computer 120 by executing various programs stored in the storage device 122 .

In addition, the storage device 122 includes various video contents (movies, etc.) such as a photographed image of each customer previously photographed by the camera 110 and a beauty-related application (hereinafter, referred to as “beauty-related AP”). Here, the beauty-related AP is a beauty-related application for the purpose of improving the appearance and health of customers, such as ideal hair styling, hair coloring (dyeing), and ideal makeup method using the mirror display device 100, for example, Hair style applications (hereinafter referred to as "hairstyle AP") or makeup applications (hereinafter referred to as "makeup AP"), health applications such as exercise (hereinafter referred to as "health AP"), etc. may be mentioned. is only an example to some extent. In addition, the storage device 122 also includes a customer management table in which the advertisement information transmitted from the advertisement providing server and the identification ID of each customer of the corresponding store and the customer's personal information are matched. Also, the storage device 122 may store information on a date the customer has visited in the past and information on a hair style performed on the date of the visit. Also, the storage device 122 may store an electronic catalog including a collection of various hairstyle images.

The personal information management server 200 is constituted by, for example, a computer with high computational processing capability, and executes a predetermined server program on the computer to realize a server function (see Fig. 1). Here, the computer constituting the personal information management server 200 does not necessarily have to be one, and may be composed of a plurality of computers distributed in the communication network (N). The personal information management server 200 is connected to the mirror display device 100 installed in each facility (a hairdresser is assumed in this embodiment) through a communication network N. The personal information management server 200 includes a personal information DB 210 that manages personal information of customers of each facility and a processor 220 composed of CPU, ROM, RAM, and the like. The personal information DB 210 includes a customer management table TA1. In addition, the personal information management server 200 may also store information on the date the customer has visited in the past and the hair style information performed on the visit date.

These components may be implemented as hardware or software, or may be implemented through a combination of hardware and software.

3 is a diagram exemplifying the registration contents of the customer management table TA1.

In the customer management table TA1, an identification ID for identifying each customer and the customer's personal information are registered in correspondence with each other. Personal information includes the customer's name, age, gender, date of birth, address, phone number, email address, SNS account information, etc. used for the hairdresser's store ID, history information indicating the use of the hairdresser, and furthermore, the customer's shooting screen ( In some cases, the face and upper body), etc. may be included. The processor 420 may process new registration of personal information in the customer management table TA1, update of registered personal information, and the like.

In addition, although the present embodiment exemplifies a state in which the mirror display device 100 is introduced to several beauty salons, the number of beauty salon stores may be one. Of course, the place where the mirror display device 100 is installed is not limited to a beauty salon, and it is applicable to various facilities (clothing-related stores, health clubs, entertainment facilities, etc.) that introduce the mirror display device 100 .

The advertisement providing server 300 may be configured to include an advertisement selection unit 310 , an advertisement DB 320 , an advertisement delivery unit 330 , and the like (see FIG. 1 ).

The advertisement selection unit 310 may select an advertisement suitable for the customer according to a preset advertisement selection algorithm based on the customer's personal information transmitted from the mirror display device 100 . For example, to explain, the advertisement selection unit 310 uses the gender, age, etc. included in the customer's personal information to select one or several other advertisement information about the most popular sports, cuisine, housing, etc., that are most popular in the corresponding generation. can be selected.

The advertisement selection unit 310 uses the status information such as the customer's job and income to advertise information about real estate or investment trusts (targeted for the wealthy) or advertisement information about insurance and household financial planning services (targeted for the main consumer group). etc. can be selected. In addition, the advertisement selection unit 310 uses, in addition to the customer's personal information (or instead of personal information), the hair styling service that the customer received at the hairdresser and hair styling services such as hair makeup (in other words, the customer's selected beauty information). It is also possible to select advertisement information about products such as recommended cosmetics or shampoo. In addition, if there are several advertisements belonging to the same genre, one or more may be selected arbitrarily, but one or more may be selected from the most popular advertisements.

In the advertisement DB 320, advertisement information including an identification ID for identifying each advertisement, an advertisement name, and advertisement content may be registered.

The advertisement transmission unit 330 may transmit one or more advertisement information selected by the advertisement selection unit 310 to the mirror display apparatus 100 of the corresponding store.

The product server 400 includes, for example, a server computer, etc., and includes a product DB 410 and a processor 420 including a CPU, ROM, RAM, and the like (refer to FIG. 1). In the product DB 410, the identification ID of each product, inventory status, order status, etc. are managed for products such as cosmetics such as lip and eye shadow, hair care products such as shampoo and conditioner, health care products such as protein products and training products. A product management table TA2 to be used is registered. Also, the electronic catalog stored in the storage device 122 may be stored in the product DB 410 of the product server 400 and transmitted to the mirror display device 100 .

When the processor 420 receives an instruction to purchase the customer's product (for example, the lipstick found in this makeup; including the quantity) through the mirror display device 100 , first, the identification ID of the product included in the purchase instruction The product that the customer wants to purchase is specified by searching the product management table TA2 using . And the processor 420 uses the customer's personal information (specifically, name, place of residence, phone number, etc.) included in the purchase instruction to deliver the product the customer wants to purchase to the customer's residence. processing can be directed.

In addition, when the processor 420 receives a request for transmission of the electronic catalog for customer consultation through the mirror display device 100 , it transmits the request to the mirror display device 100 through the network.

4 is a flowchart showing the registration processing of personal information by the personal information management server 200 .

A user (a hairdresser or hair stylist) appropriately manipulates the input unit 123 (eg, a touch panel) of the mirror display device 100 of the beauty salon to obtain necessary customer personal information (eg, customer name, age, gender, date of birth). , residence, phone number, e-mail address or SNS account name, status information, the name of the hairdresser you use, date of visit, or customer's photographed image) can be entered. In addition, the customer may directly manipulate the input unit 123 to input the customer's personal information. The inputted personal information of the customer is given a unique identification ID for identifying the customer, is matched with the unique identification ID, and is stored in the storage device 122 of the mirror display device 100 and, at the same time, the mirror display device 100 It is received from the personal information management server 200 through the communication network (N) (step S410). When the personal information management server 200 receives the customer's unique identification ID and personal information, it may be registered in the customer management table TA1 by matching the identification ID and personal information (step S420).

In addition, the customer's unique identification ID may be stored in connection with the face ID photographing the customer's face. Accordingly, when the customer only takes a face image through the camera 110 of the mirror display apparatus 100 , the customer can log in to the mirror display apparatus 100 through face recognition.

5 is a flowchart showing an update processing of personal information by the personal information management server 200 .

The personal information management server 200 receives the customer's personal identification ID from any one of the mirror display apparatuses 100 installed in a plurality of beauty salons (places), and receives the updated personal information ( When receiving (for example, visit history and the customer's latest hairstyle, etc.) (step S510), the customer's electronic record matching the identification ID registered in the customer management table TA1 based on the updated personal information (that is, personal information) is updated (step S520), and then the processing is ended.

6 and 7 are schematic diagrams for explaining the hair design adjustment function realized by the hair style AP installed in the mirror display device 100 .

Selectable hair patterns HP1 to HP5 are displayed on the top of the display panel 102 or on a specific area, and the customer's photographed image IP is displayed on the right side of the display panel 102 . Several selectable hair patterns HP are stored in the storage device 122 . The hairdresser selects a recommended hair pattern that is considered suitable for the customer, for example, by touch-manipulating the hair pattern HP4 in FIG. 6 . When the start operation is in progress, an image in which the hair pattern HP4 is superimposed on the photographed face of the customer is displayed on the right side of the display panel 102 . In addition, of course, it is okay if the customer himself (or another third party; hereinafter collectively referred to as "customer, etc.") who is not a hairdresser selects the hair pattern.

Meanwhile, the hair patterns HP1 to HP5 displayed on the upper end of the display panel 102 or on a specific area may be hair patterns that have been previously treated by the customer.

According to an embodiment, the hair pattern HP may be adjusted according to an input operation of a customer or a hairdresser (hair stylist). Specifically, as shown in FIG. 7 , the customer touches the displayed hair pattern HP4 superimposed on their face image to twist the hair image H in the left and right and up and down directions, or the hair image H If you zoom in or out, you can change the flow of hair (for example, insert a wave to change the strength of the wave or change the direction of the wave, etc.), or control the hair volume (volume up or down) according to the corresponding touch operation. there will be

A customer's face is different, and even if there is a desired hair pattern (for example, a dreaded hairstyle or a short bob style), simply by overlapping the customer's selected hair pattern (HP) on the customer's face ( HP) does not overlap the customer's face properly, so it is impossible to accurately determine whether the hair pattern (HP) will suit the customer. In the case of ordering hair styling using a fixed hair pattern (HP) as described above, the customer's hair is often not as imaged, and there is a problem in that there is a problem of dissatisfaction with the hair styling.

Accordingly, according to the hair pattern HP according to the embodiment, the selected hair pattern HP is not only displayed overlaid on the customer's face image, but also the hair of the hair pattern HP selected by the customer or the like through an intuitive operation such as a touch operation. Image (H) can be adjusted. Therefore, the hair image (H) of the hair pattern (HP) superimposed on the customer's face can be delicately adjusted based on the hair pattern (HP), and the customer's hair style can be reproduced without limitation. If you order hair styling using the customized hair pattern (HP) image, the customer can receive the hair style as it is.

FIG. 8 is a schematic diagram for explaining the hair color adjustment function realized by the hair style AP installed in the mirror display device 100 .

When the user selects the menu 810 for starting the dyeing simulation on the display panel 102 , the mirror display apparatus 100 displays a list of selectable hair colors HC1 to HC5 at the bottom of the display panel 102 or in a specific area. ), and the customer's photographed image IP is displayed on the right side of the display panel 102 . Several selectable hair colors HC1 to HC5 are stored in the storage device 122 . The hairdresser selects a recommended hair color that is considered suitable for the customer, for example, by touching the hair color HC1 in FIG. 8 . When the start operation is in progress, the mirror display apparatus 100 first changes the color of the overlapped and displayed hair pattern to a predetermined color, and then displays the image 820 that is changed to the hair color HC1 selected by the user again on the display panel ( 102) is displayed. For example, if the original color of the hair pattern after styling is black and the hair color selected by the user is reddish brown, the mirror display device 100 first removes the black color from the hair pattern before dyeing to display only the outline of the hair. After changing to white or light gray, you can obtain a better quality dyed hair pattern by changing the hair color by applying the selected color, reddish brown, to the image of the hair pattern in this state. In this procedure, if a new color is applied while the original color is in the hair pattern, the color desired by the user or customer is not output. it is to make Also, of course, it is okay if the customer himself (or another third party; hereinafter collectively referred to as "customer, etc."), not the hairdresser, selects the hair color.

On the other hand, the list of hair colors (HC1 to HC5) displayed on the lower part or specific area of the display panel 102 is a list of hair colors most frequently selected in a hairdressing salon using a hair styling service providing system, or a list of hair colors that a customer has received treatment in the past. It may be a list of hair colors.

The hair color simulation may be independently executed as a service for hair dyeing separately from the hair pattern (style) selection and adjustment simulation of FIGS. 6 to 7 , or may be performed before or after hair styling. The example of FIG. 8 describes an example of performing hair dyeing simulation in addition to the result image after the adjustment simulation of the hair pattern of FIG. 7 .

9 is a flowchart showing a hair style adjustment process executed by the control unit 121 of the mirror display apparatus 100 .

The control unit 121 determines whether a selection operation of one hair pattern HP displayed on the display panel 102 is performed (step S910 ). When not even one hair pattern HP is selected (step S910; No), the control unit 121 repeats step S910. After that, when detecting that a certain hair pattern HP has been selected by, for example, the customer (step S910; Yes), the control unit 121 sets the selected hair pattern HP to the customer's face (specifically, the customer is superimposed on the photographed image) and displayed on the right side of the display panel 102 (step S920).

Thereafter, the controller 121 detects whether an adjustment operation (specifically, a touch operation) of the hair image H has been performed with respect to the hair pattern HP superimposed on the customer's face (step S930 ). If it is determined that the adjustment operation of the hair image H is not performed (step S930; No), the control unit 121 determines whether an instruction to end the hair adjustment processing is input (step S940). When an instruction to end the hair adjustment process is input (step S940; YES), the control unit 121 skips step S950 and the like and ends the hair adjustment process.

On the other hand, if it is determined that the end instruction of the hair adjustment process has not been input (step S940; NO), the control unit 121 returns to step S930. After that, the control unit 121, for example, when detecting that the adjustment operation of the hair image (H) by the customer (eg, operation to enlarge the hair image (H), etc.) has been made (step S930; yes ), the process proceeds to step S950.

In step S950, the controller 121 processes (adjusts) the hair image H according to the adjustment operation of the hair image H, displays the processed hair image H on the display panel 102, and then proceeds to step S940. proceed For example, if an operation to enlarge the hair image H is performed, the control unit 121 performs an operation such as volume up on the hair image H superimposed on the customer's face according to the corresponding operation, and the processed hair image ( H) is displayed on the display panel 102 . If the control unit 121 determines that the instruction for ending the hair adjustment processing is not input (step S940; No), the control unit 121 returns to step S930, while detecting that the instruction for ending the hair adjustment processing is input (step S940; Yes), The hair adjustment process described above is finished.

Hereinafter, a hair styling service providing method of the hair styling service providing system according to an embodiment of the present invention will be described.

10 is a flowchart illustrating a method of providing a hair styling service in a hair styling service system including a mirror display device.

First, when the camera 110 captures a customer's face or reads a QR code displayed on a customer terminal (eg, a smartphone), the communication unit 124 receives the read customer information (S1010).

The control unit 121 authenticates the connected customer based on the received customer information (S1020). The control unit 121 analyzes the customer's face image data or personal information DB 210 and the received face image or QR code stored in the personal information management server 200 to confirm the identity of the connected customer to perform customer authentication can be done In addition, the control unit 121 may output a customer consultation screen based on the authenticated customer information to introduce a promotion or guide a price according to a hair style before full-scale hair styling.

Next, the controller 121 may retrieve a hair style image from the hair style history or electronic catalog matching the customer based on the user's selection and output it to the display panel 102 ( S1030 ). For example, when a customer wants to receive a previously treated hair style again, the user (hair stylist) stores it in the storage device 122 or the customer information management server 200 through the mirror display device 100 . The hair style history of the customer can be retrieved and output on the display panel 102 . The hair style history of the customer may include information on the customer's past visit date and hair style selected and treated on the past visit date. In addition, when the customer wants to receive a treatment with a new hair style other than the previously treated hair style, the user (hair stylist) is stored in the storage device 122 or the product server 400 through the mirror display device 100 An electronic catalog can be retrieved and output on the display panel 102 . A user (hair stylist) may output a hair style collection searched or collected through the Internet on the display panel 102 after accessing the network through the communication unit 124 of the mirror display device 100 . The hair style searched or collected through the Internet may be a hair style through a search service such as Google, a hair style searched through Pinterest, or the like.

The controller 121 receives a hair style selected by a hair stylist or a customer from among the hair style images output on the display panel 102 (S1040).

The mirror display device 100 acquires the customer's face image by photographing the customer's face through the camera 110 in order to simulate the selected hair style to the customer (S1050), and the controller 121 acquires the selected hair style. After being superimposed on the customer's face image, the superimposed image may be output on the display panel 102 (S1060).

Next, when the input unit 123 receives a touch manipulation for changing a hair style in the superimposed image output on the display panel 102 ( S1070 ), the controller 120 overlaps the hair style according to the touch input manipulation for the hair style The image may be transformed (S1080). After superimposing the selected hair style image on the customer's face image, the process of transforming the hair style by the hair stylist's touch input may be performed by the method of FIGS. 7 and 9 .

Next, the display panel 102 outputs a modified superimposed image ( S1090 ), and shows the modified hair style to the customer, and the hair stylist may additionally change the style according to the customer's request.

Finally, when the customer's satisfactory hair style is completed, the hair stylist stores the currently output customer's hair style image (modified overlapping image) in the storage device 122 or the personal information management server 200 on the network, The customer's hair style history is updated (S1100).

In addition, the hair styling service providing method of the hair styling service system according to an embodiment of the present invention may provide a dyeing simulation for changing the customer's hair color regardless of whether the customer's hair style is changed. In the dyeing simulation, the control unit 121 outputs the hair color list stored in the storage device 122 through the display panel 102, and the control unit 121 applies a hair color selected from the output hair color list to the customer's face image. It may include changing the color of the overlapping hairstyle.

On the other hand, the step (S1030) of calling a hairstyle image from the hair style history or electronic catalog matching the customer based on the user's selection and outputting it to the display panel 102 (S1030) is performed by the control unit 121 through the communication unit 124. It is replaced with the step of receiving and outputting the hair style that the customer selects in advance or the hair style that the customer has changed through simulation before visiting the hairdresser through the customer terminal (eg, smartphone) A selected hair style or a hair style that a customer directly transforms may be included and output.

In addition, the hair styling service providing system 1000 captures a hair style simulation image for a customer, an image after hair style treatment, and a video containing a hair style treatment process, and transmits it to the customer terminal, or stores it in the personal information management server 200 . This allows customers to view them again whenever they want. For example, a customer can receive an image of a person with a hair style that has been treated and share it with acquaintances through SNS, etc., and thus can enjoy the effect of promoting hair style treatment in a beauty salon. In addition, if a video clip of a hairdresser styling a hair product such as wax is stored, the customer can refer to the video later to directly style the wax. In addition, the video can be used as a material to educate other hairdressers.

In addition, the hair styling service providing system 1000 provides a messaging service between the customer terminal and the hair stylist to transmit a hair style or message designed by the hair stylist to the customer to promote it, or a feedback message from the customer A function to receive can be provided.

Also, the hair styling service providing system 1000 may update the electronic catalog by receiving the hair style input by the hair stylist and storing it in the storage device 122 or the product server 400 .

In this specification, a user may be referred to as a hairdresser, a hair stylist, or a hair designer. Also, in the present specification, a hair style may be referred to as a hair pattern.

In the above description, steps S1010 to S1100 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be switched.

In addition, in the above-described embodiment, the personal information management server 200, the advertisement providing server 300, and the product server 400 were configured as separate separate servers, but for example, the personal information management server 200, All or part of the functions of each of the advertisement providing server 300 and the product server 400 may be combined with any one server (eg, the advertisement providing server 300) to form a single server. In addition, one or more of the functions of each of these servers may be mounted on the mirror display apparatus 100 . Also, in the above-described embodiment, the mirror display apparatus 100 in which one or more cameras 110 is mounted is exemplified, and it is applicable to the mirror display apparatus 100 in which the camera 110 is not mounted.

In addition, in the above-described embodiment, although the case in which the customer operates the mirror display device 100 is exemplified, the present invention is not limited thereto, and a person related to the customer (eg, a hairdresser in a beauty salon, a trainer in a health club, etc.) operates of course it's fine

Hereinafter, an apparatus for extracting a hair image and a method for extracting a hair image according to an embodiment of the present invention will be described with reference to FIGS. 11 to 12 .

11 is a diagram showing a schematic configuration of an apparatus for extracting a hair image according to an embodiment of the present invention.

12 is a schematic diagram for explaining a hair image extraction method according to an embodiment of the present invention.

11 and 12 , the hair image extraction apparatus 500 includes a display panel 510, a touch input type input unit 520 configured on the display panel, a control unit 530, and a communication unit 540. can be configured. The illustrated components are not essential, so a hair image extraction apparatus having more or fewer components may be implemented.

The display panel 510 may output an input image of the subject. The image of the subject may be input to the device after being photographed through a separately provided camera (not shown), or may be an image of the subject received from a network and stored in a storage device (not shown).

The input unit 520 may receive an input for designating an extraction region including hair in the image of the subject output on the display panel 510 . To designate the extraction region, the region to be extracted from the output image may be selected (1210) as a box to designate the region (refer to FIG. 12(a) ).

When the extraction region 1210 is designated by the inputter 520 , the controller 530 outputs only the designated portion of the input image to the display panel 510 .

Also, the input unit 520 may receive a portion designating the hair region 1250 in the image 1220 outputting the extraction region 1210 as a drag-type selection 1230 . For example, in order to extract only a portion of the hair region 1220 from the image 1220 of the extraction region 1210, the user drags the portion with a pointing device such as a finger or a mouse (1230), and then a process for hair extraction is executed, the controller 530 can extract only the hair part by separating the hair region 1250 designated by dragging from the other background parts (refer to FIG. 12(b) ). In this case, the controller 530 may extract only the hair region 1250 by removing a background portion other than the hair region 1250 from the extraction region 1210 using the GrabCut algorithm.

Also, when the input unit 520 receives an input for designating a part to be deleted from the image of the extracted hair region 1250 as a drag-type selection ( 1240 ), the controller 530 deletes it from the image of the hair region 1250 . Delete the part designated as the part to be. For example, the user extracts the hair region 1250 with the grabcut algorithm, but finds that the subject's face part, that is, the customer's face part, is not completely deleted as the background in the output image, and drags the part to If designated ( 1240 ), the controller 530 deletes the corresponding part so that only the hair region 1250 can be clearly output (refer to FIG. 12(c) ).

)의 선형결합으로 표현할 수 있으므로,

로 표현되고, 이를 이용하여

를 산출할 수 있다. 여기에서, 대상 영상에 대한 바이너리 맵(binary map)을 확장하여 트리 맵(trimap)을 생성할 수 있고, 이를 이용하여 F와 B를 계산할 수 있으며, 각 픽셀의 RGB 값이 3차원 벡터임으로 전술한 픽셀 값(I)의 수식을 변경하여 각 픽셀의 투명도(

)를 계산할 수 있다.Meanwhile, when extracting the hair region 1250 using the GrabCut algorithm, the controller 530 applies alpha matting to the boundary of the extracted hair region 1250 to form hair in the hair region. can be expressed in a subtle and natural way. Since the hair area has various shapes and is always attached to the face area, it is difficult to accurately extract the hair area only by setting a bounding box, that is, by designating the extraction area. Since hair has very fine and complex boundaries, there is a limit to expressing it only with binary images. Accordingly, the controller 530 applies alpha matting for synthesizing the hair of the hair region 1250 with the background image based on the alpha value of each pixel in the hair region 1250 . When alpha matting is applied, it is possible to calculate the transparency expressed in a gray image of at least 8 bits instead of a binary image for the target area, and through this, it is possible to express fine details of the hair. In general, images have foreground (F) and background (B), and transparency (

) can be expressed as a linear combination of

is expressed as, and using

can be calculated. Here, a tree map can be generated by extending a binary map for the target image, F and B can be calculated using this, and the RGB values of each pixel are a three-dimensional vector. By changing the formula for the pixel value (I), the transparency of each pixel (

) can be calculated.

는

이므로 수학식

를 통해서 산출할 수 있다. 여기에서,

는 픽셀 값,

는 전경 영상,

는 배경 영상이다. 따라서, 제어부(530)는 합성 영상의 각 픽셀 값(

)을 수학식

을 통해서 산출할 수 있다. 여기에서,

는 합성할 배경 영상이다.The controller 530 generates a mask having a value of 0 to 1 by using the transparency value of each pixel of the target image, and then generates a synthesized image in which the hair region 1250 is synthesized with a new background image based on the generated mask. can create Transparency value

Is

so the formula

can be calculated through From here,

is the pixel value,

is the foreground image,

is the background image. Accordingly, the controller 530 controls each pixel value (

) to the formula

can be calculated through From here,

is the background image to be synthesized.

The communication unit 540 may receive an image of the subject from an external device, or transmit the generated hair image to an external device to share with the external device.

The hair image extraction apparatus according to an embodiment of the present invention may be used as a mirror display apparatus of a hair styling service providing system, and may also be applied to a portable terminal such as a smart phone.

In this section, a hair image extraction method will be described with reference to FIGS. 11 to 13 .

13 is a flowchart illustrating a hair image extraction method of the hair image extraction apparatus according to an embodiment of the present invention.

First, the controller 530 receives an image of the subject (S1210). As described in the detailed description of FIG. 11 , the image of the subject may be an image acquired from a separately provided photographing device or an image collected from a network or the like and stored in a storage device.

Next, the controller 530 outputs the received image of the subject to the display panel 510 (S1220).

Next, the input unit 520 receives a selection from the user to designate an extraction region including hair in the image of the subject ( S1230 ).

The controller 530 outputs the image of the extraction region to the display panel 510 (S1240).

Next, the input unit 520 receives an input for designating a hair region in the output image of the extraction region, and a region selected by dragging the region (S1250).

Next, the controller 530 extracts the hair region by removing a background part other than the selected hair region by dragging from the extraction region using the GrabCut algorithm (S1260).

The controller 530 applies alpha matting to the boundary of the extracted hair region to finely express the hair in the hair region (S1270). Alpha matting is the same as described in the detailed description of FIG. 11 .

Finally, when the input unit 520 receives a drag selection for designating an unnecessary part not to be deleted when the background is separated from the extracted hair region image ( S1280 ), the controller 530 selects the part to be deleted from the hair region image ( S1280 ). The part is deleted (S1290).

In the above description, steps S1210 to S1290 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be switched.

Meanwhile, the above-described image of the hair region may be provided as a hair pattern HP in the virtual hair styling of FIGS. 6 and 7 and used to simulate a customer's hair style.

First, the controller 530 superimposes the extracted hair region image on the hair part in the face image of the subject or another subject, and outputs the superimposed image to the display panel 510 .

Next, the input unit 520 receives a touch manipulation for deformation of the hair region in the superimposed image from the user or customer.

Finally, after the controller 530 deforms the overlapped image according to a touch input operation on the hair region, the controller 530 outputs the transformed overlapped image through the display panel 510 .

As used herein, the term “unit” (eg, a control unit, etc.) may mean, for example, a unit including one or a combination of two or more of hardware, software, or firmware. The term “unit” may be used interchangeably with terms such as, for example, unit, logic, logical block, component, or circuit. A “part” may be a minimum unit of an integrally formed part or a part thereof. A “unit” may be a minimum unit or a part of performing one or more functions. “Part” may be implemented mechanically or electronically. For example, a “unit” may be one of an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs) or programmable-logic device, known or to be developed, that performs certain operations. It may include at least one.

At least a portion of an apparatus (eg, modules or functions thereof) or a method (eg, operations) according to various embodiments is, for example, a computer-readable storage medium in the form of a program module It can be implemented as a command stored in . When the instruction is executed by a processor, the one or more processors may perform a function corresponding to the instruction. The computer-readable storage medium may be, for example, a memory.

Computer-readable storage media/computer-readable recording media include hard disks, floppy disks, magnetic media (eg, magnetic tape), and optical media (eg, compact CD-ROMs). disc read only memory), digital versatile disc (DVD), magneto-optical media (e.g., floptical disk), hardware devices (e.g., read only memory (ROM), random (RAM) access memory), or flash memory, etc.) In addition, program instructions may include high-level language code that can be executed by a computer using an interpreter, etc. as well as machine code such as generated by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the various embodiments, and vice versa.

A module or program module according to various embodiments may include at least one or more of the above-described components, some may be omitted, or may further include additional other components. Operations performed by a module, a program module, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or in a heuristic method. Also, some operations may be executed in a different order, omitted, or other operations may be added.

As used herein, the term “a” is defined as one or more than one. Also, the use of introductory phrases such as “at least one” and “one or more” in a claim includes introductory phrases such as “at least one” and “one or more” and an obscure phrase such as “a” in the same claim. Even if there is, the introduction of another claim element by the obfuscated phrase "a" shall be construed to mean to limit any particular claim containing the claim element so introduced, to an invention containing only one such element. shouldn't be

Unless otherwise indicated, terms such as “first” and “second” are used to arbitrarily distinguish the elements described by such terms. Accordingly, these terms are not necessarily intended to indicate a temporal or other priority of such elements, and the mere fact that certain measures are recited in different claims does not indicate that a combination of these measures cannot be used to advantage. . Accordingly, these terms are not necessarily intended to indicate the temporal or other priorities of such elements. The mere fact that certain measures are cited in different claims does not indicate that a combination of these measures cannot be useful.

The arrangement of components to achieve the same function is effectively “related” such that the desired function is achieved. Thus, any two components combined to achieve a particular functionality may be considered "related" to each other so that the desired function is achieved, regardless of structure or intervening component. Likewise, two components thus associated may be considered “operably connected” or “operably coupled” to each other to achieve a desired function.

Also, those skilled in the art will recognize that the boundaries between the functionality of the operations described above are illustrative only. A plurality of operations may be combined into a single operation, the single operation may be distributed to additional operations, and the operations may be executed at least partially overlapping in time. Also, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments. However, other modifications, variations and alternatives are also possible. Accordingly, the detailed description and drawings are to be regarded in an illustrative rather than a restrictive sense.

The phrase “may be X” indicates that condition X may be satisfied. This phrase also indicates that condition X may not be met. For example, a reference to a system that includes a particular component must also include a scenario in which the system does not contain the specific component. For example, a reference to a method that includes a particular action should also include a scenario in which the method does not include the particular component. As another example, however, references to a system configured to perform a specific action should also include scenarios in which the system is not configured to perform a specific action.

The terms “comprising,” “having,” “consisting of,” “consisting of,” and “consisting essentially of,” are used interchangeably. For example, any method may include at least the acts contained in the drawings and/or the specification, and may include only the acts contained in the drawings and/or the specification.

One of ordinary skill in the art would recognize that the boundaries between logical blocks are exemplary only, and that alternative embodiments may incorporate logical blocks or circuit elements or impose an alternative decomposition of functionality on the various logical blocks or circuit elements. will recognize that Accordingly, it should be understood that the architecture shown herein is exemplary only, and in fact, many other architectures may be implemented that achieve the same functionality.

Also, for example, in one embodiment, the illustrated examples may be implemented as circuitry located on a single integrated circuit or within the same device. Alternatively, the above examples may be implemented as any number of individual integrated circuits or individual devices interconnected to each other in any suitable manner, and other variations, modifications, variations and alternatives are also possible. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Also, for example, the examples or portions thereof may be implemented as a software or code representation of a physical circuit or a logical representation convertible into a physical circuit, such as any suitable type of hardware description language.

Further, although the present invention is not limited to physical devices or units implemented in non-programmable hardware, mainframes, mini computers, servers, workstations, personal computers, notepads, which are generally referred to herein as 'computer systems' , personal digital assistants (PDAs), electronic games, automobiles and other embedded systems, cell phones, and various other wireless devices, such as programmable devices capable of performing desired device functions by operating in accordance with appropriate program code. Alternatively, it may be applied to units as well.

A system, apparatus, or device referred to in this specification includes at least one hardware component.

Connections as described herein may be any type of connection suitable for transmitting a signal to or from each node, unit or apparatus, for example via an intermediate apparatus. Thus, unless implied or otherwise stated, a connection may be, for example, a direct connection or an indirect connection. A connection may be described or depicted with reference to being a single connection, multiple connections, one-way connection, or two-way connection. However, different embodiments may change the implementation of the connection. For example, a separate one-way connection can be used rather than a two-way connection, and vice versa. In addition, a plurality of connections may be replaced with a single connection that transmits a plurality of signals sequentially or in a time multiplexing manner. Likewise, a single connection carrying multiple signals may be split into various connections carrying subsets of these signals. Therefore, many options exist for transmitting the signal.

One of ordinary skill in the art would recognize that the boundaries between logical blocks are exemplary only, and that alternative embodiments may incorporate logical blocks or circuit elements or impose an alternative decomposition of functionality on the various logical blocks or circuit elements. will recognize that Accordingly, it should be understood that the architecture shown herein is exemplary only, and in fact, many other architectures may be implemented that achieve the same functionality.

In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or acts listed in a claim.

In the above, preferred embodiments of the technology of the present specification have been described with reference to the accompanying drawings. Here, the terms or words used in the present specification and claims should not be construed as being limited to conventional or dictionary meanings, but should be interpreted as meanings and concepts consistent with the technical spirit of the present invention. The scope of the present invention is not limited to the embodiments disclosed herein, and the present invention may be modified, changed, or improved in various forms within the scope of the spirit and claims of the present invention.

1000: hair styling service providing system
100: mirror display device 101: half mirror
101S: transflective reflective surface
102, 510: display panel 103: display screen
110: camera
120: control computer
121, 530: control unit
123, 520: input unit
124, 540: communication department
200: personal information management server 210: personal information DB
320: advertisement DB 410: product DB
220, 420: processor
300: advertisement providing server 310: advertisement selection unit
330: advertisement delivery unit 400: product server
TA1: Customer management table TA2: Product management table
N: communication network

Claims (10)

receiving, by the controller, an image of the subject;
outputting, by the controller, an image of the subject to a display panel;
receiving, by an input unit, a selection for designating an extraction region including hair in the outputted image of the subject;
outputting, by the control unit, the image of the extraction area to the display panel;
receiving, by the input unit, a drag selection for designating a hair region in the output image of the extraction region;
extracting the hair region by the controller removing a background part other than the drag-selected hair region from the extraction region using a GrabCut algorithm; and
applying, by the controller, alpha matting to the boundary of the extracted hair region to finely express hair in the hair region;
A hair image extraction method of a hair styling service providing system comprising a. According to claim 1,
The step of applying the alpha matting is,
a step in which the controller synthesizes the hair of the hair region with a background image based on the alpha value of each pixel in the hair region
A method of extracting a hair image in a hair styling service providing system. 3. The method of claim 2,
The step of applying the alpha matting is,
generating, by the controller, a mask having a value of 0 to 1 by using the transparency value of each pixel; and
generating a synthesized image in which the hair region is synthesized with the background image based on the generated mask;
A hair image extraction method of a hair styling service providing system comprising a. 4. The method of claim 3,
Each pixel value of the composite image

Is

It is calculated through the formula of

is the transparency value,

is the foreground image,

is the background image,

is the background image to be synthesized,

Is

and

calculated through the formula of
A method of extracting a hair image in a hair styling service providing system. 4. The method of claim 3,
receiving, by the input unit, a drag selection for designating a portion to be deleted from the extracted image of the hair region; and
deleting, by the controller, a portion designated as the portion to be deleted from the image of the hair region;
The hair image extraction method of the hair styling service providing system further comprising a. According to claim 1,
superimposing the extracted image of the hair region on the hair part of the face image of the subject or another subject, and outputting the superimposed image to the display panel;
receiving, by the input unit, a touch manipulation for transforming the hair region in the superimposed image;
transforming, by the controller, the overlapped image according to a touch input operation on the hair region; and
outputting, by the display panel, the deformed superimposed image;
The hair image extraction method of the hair styling service providing system further comprising a. a display panel for outputting the input image of the subject and the image of the extraction area;
an input unit receiving a selection for designating an extraction region including hair and a drag selection for designating a hair region in the outputted image of the subject; and
The hair region is extracted by removing a background part other than the drag-selected hair region from the extraction region using the GrabCut algorithm, and alpha matting is applied to the boundary of the extracted hair region to a control unit for finely expressing hair in the hair region;
A hair image extraction device comprising a. According to claim 7, wherein the control unit,
The hair of the hair region is synthesized with a background image based on the alpha value of each pixel in the hair region,
generating a mask having a value of 0 to 1 using the transparency value of each pixel;
generating a synthesized image in which the hair region is synthesized with the background image based on the generated mask
Hair image extraction device. 9. The method of claim 8,
Each pixel value of the composite image

Is

It is calculated through the formula of

is the transparency value,

is the foreground image,

is the background image,

is the background image to be synthesized,

Is

and

calculated through the formula of
Hair image extraction device. 7. The method of claim 6,
The input unit receives a drag selection for designating a part to be deleted from the extracted image of the hair region,
The controller is configured to delete a part designated as the part to be deleted from the image of the hair region.
Hair image extraction device.

Images