JP2024537064A - Augmented reality makeup design filters - Google Patents

Abstract

Systems, devices, and methods for generating, sharing, and presenting an augmented reality cosmetic design filter. One or more non-transitory computer memory devices can store computer-readable instructions that, when executed by a computing circuitry, cause the computing circuitry to perform operations for generating an augmented reality cosmetic design filter. The operations can include defining a baseline description of a biological surface using one or more radiation sensors, the radiation sensors being electronically coupled to the computing circuitry. The operations can include identifying an application of a cosmetic formulation to the biological surface using the one or more radiation sensors. The operations can include generating a trace describing the application with reference to the baseline description. The operations can also include outputting the trace.
[Selected Figure] Figure 1

Description

Cross-reference to related applications

This application claims the benefit of U.S. Patent Application No. 17/491,061, filed September 30, 2021, and French Patent Application No. 2113547, filed December 15, 2021, the contents of which are incorporated herein by reference in their entireties.

Systems, devices, and methods for generating, sharing, and presenting an augmented reality cosmetic design filter. One or more non-transitory computer memory devices can store computer-readable instructions that, when executed by the computing circuitry, cause the computing circuitry to perform operations for generating an augmented reality cosmetic design filter. The operations can include defining a baseline description of a biological surface using one or more radiation sensors. The radiation sensors are electronically coupled to the computing circuitry. The operations can include identifying an application of a cosmetic formulation to the biological surface using the one or more radiation sensors. The operations can include generating a trace describing the application with reference to the baseline description. The operations can also include outputting the trace.

In some embodiments, identifying the application of the cosmetic formulation includes detecting the cosmetic formulation relative to the biological surface using one or more of the radiation sensors and the baseline description. Generating the trace can include receiving information describing a plurality of shape types, attributing a first shape type of the shape types to at least a portion of the biological surface using the baseline description, generating a numerical representation of the application of the cosmetic formulation, the numerical representation describing location information relative to the baseline description, and converting the numerical representation from the first shape type to a second shape type of the shape types. The biological surface can be a face. The first shape type and the second shape type can describe facial features. Defining the baseline description of the biological surface can include projecting invisible electromagnetic radiation onto the biological surface using a radiation source electronically coupled to the computing circuitry. The one or more radiation sensors can include a camera configured to detect electromagnetic radiation from the ultraviolet, visible, or infrared spectral ranges, or a combination thereof.

In some embodiments, generating a trace includes tracking a movement of the application relative to the biological surface and generating a numerical representation of the movement relative to a baseline description. The operations may further include identifying a cosmetic formulation and defining a color of the cosmetic formulation using the identifier information of the cosmetic formulation. The operations may further include estimating a surface tone of the biological surface, estimating a formulation tone of the cosmetic formulation using the surface tone, and determining a color of the cosmetic formulation using the surface tone and the formulation tone.

In some embodiments, the one or more non-transitory computer memory devices are electronically coupled to the smartphone. Outputting the trace may include communicating the trace to a remote computing system.

A method for generating an augmented reality cosmetic design filter can include defining a baseline description of a biological surface using one or more radiation sources and one or more radiation sensors. The method can include identifying an application of a cosmetic formulation to the biological surface using the radiation sensors. The method can include generating a trace describing the application with reference to the baseline description. The method can also include outputting the trace.

In some embodiments, generating the trace includes receiving shape information describing a plurality of shape types, using the shape information to attribute a first one of the shape types to at least a portion of the biological surface, generating a numerical representation of the application of the cosmetic formulation, the numerical representation describing the position information relative to a baseline description, and converting the numerical representation from the first shape type to a second one of the shape types. In some embodiments, identifying the application of the cosmetic formulation includes detecting an applicator of the cosmetic formulation, estimating a position of the applicator of the cosmetic formulation relative to the biological surface, tracking a movement of the applicator relative to the biological surface, and generating a numerical representation of the movement relative to the baseline description. The method may further include estimating a surface tone of the biological surface, estimating a formulation tone of the cosmetic formulation, and using the surface tone and the formulation tone to determine a color of the cosmetic formulation.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

FIG. 1 is a schematic diagram illustrating one embodiment of a system for extrapolating a cosmetic design, according to various embodiments.

FIG. 2 is a schematic diagram illustrating an exemplary technique for preparing an augmented reality cosmetic design filter, according to various embodiments.

FIG. 3 is a schematic diagram illustrating an example technique for deploying an augmented reality cosmetic design filter, according to various embodiments.

FIG. 4 is a diagram illustrating an example technique for converting an augmented reality cosmetic design filter from one shape type to another, according to various embodiments.

FIG. 5 is a schematic diagram illustrating an example technique for generating an augmented reality cosmetic design filter with multiple layers, according to various embodiments.

FIG. 6 is a schematic diagram illustrating an exemplary technique for determining formulation color using surface tone and formulation tone, according to various embodiments.

FIG. 7 is a block diagram illustrating an example system including components of the system of FIG. 1, according to various embodiments.

FIG. 8 is a block diagram illustrating aspects of an exemplary computing device, according to various embodiments.

In the above drawings, unless otherwise noted, like reference numerals refer to like parts throughout the various views. To simplify the drawings as necessary, not all of the example elements have been labeled. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles being described. The above-described aspects and many of the attendant advantages of the present invention will be more readily appreciated as the same becomes better understood by reference to the following detailed description of the invention in conjunction with the accompanying drawings.

Applying cosmetics and makeup into patterns and shapes is difficult by hand. For example, intricate designs and theatrical makeup are typically applied by licensed makeup professionals. In addition, it is generally difficult for people with limited mobility or vision to apply it themselves. Even relatively simple makeup designs may involve layering, blending, and application of cosmetics specific to the shape type of face and facial features. For example, makeup and other cosmetics are typically applied in a different configuration to an oval-shaped face than to a rectangular or oval-shaped face. Thus, the aesthetic effect of a makeup design applied to a surface cannot be replicated by simply mapping a static texture object, such as a texture mask, onto the surface. Furthermore, the exact sequence and extent of the actions that make up a makeup routine cannot usually be determined from an image of the final design. For this reason, makeup tutorials are usually given as step-by-step guides or videos with images showing the design in multiple intermediate states that lead to the final effect. Nevertheless, face shape-specific and skin tone-specific tutorials are more effective than general tutorials in illustrating specific makeup designs. However, such an approach takes time to implement and is difficult to replicate without the right tools and experience with the specific techniques and cosmetic formulations used.

Techniques are described for generating an augmented reality makeup design filter from a makeup routine to be applied to a target body surface of a biological subject, such as the subject's face or other region of interest. The described embodiments employ an image sensor to define one or more 3D contour mappings of the target body surface. In the context of such applications, the described embodiments provide improved accuracy and greater ease of use compared to complex manual routines. Thus, the present technology improves the manual application of makeup through the distribution of dynamic augmented reality makeup design filters that can be adapted automatically (e.g., without human intervention) to shape type and skin tone. The filters can recreate a step-by-step makeup routine, including application traces, finish effects, and augmented reality renderings of specific tools, in a manner that is sensitive to the shape type and skin tone of the end user of the augmented reality filter.

In one illustrative example, a smartphone user creates a video of a makeup routine that creates a particular aesthetic effect. As part of the video creation, the smartphone captures images and surface mapping of the user's face. An initial image is used to generate a baseline against which subsequent applications of cosmetics such as foundation, highlighter, filler, eye shadow, blush, mascara, etc. can be detected. As the video is being created, the smartphone iteratively redefines the baseline as part of defining each new application as a new layer of the makeup design. Also, as the video is being created, the smartphone recognizes new applications by detecting the user's hand movements, identifying new tools or formulations, and/or detecting color shifts in the user's face relative to the updated baseline. Over the course of the routine, a multi-layer dynamic filter is generated that includes motion, texture, and/or transparency information. The dynamic filter references the face shape of the creator of the makeup video and is stored in a distributed storage network. The filter is available to various users for virtual projection via an augmented reality interface. This filter can be used to recreate layering effects, realistically map makeup designs to various face shapes, and guide the viewer through a makeup routine in a manner specific to the viewer's face shape and skin tone, as described in the following paragraphs.

FIG. 1 is a schematic diagram illustrating one embodiment of an exemplary system 100 for generating an augmented reality cosmetic design filter 110, according to various embodiments. The exemplary system 100 includes one or more client computing devices 130, a camera 140, one or more remote computer systems 160 (also referred to as server(s)), and one or more user devices 190. As part of the exemplary system 100, the components of the configuration system may be operatively coupled via wireless communication and/or via wired communication. In some embodiments, the components may communicate directly via wireless pairing (e.g., Bluetooth) and/or via a local wireless network (e.g., a wireless router). In some embodiments, the components may communicate via one or more networks 170. The one or more networks 170 may be or include a public network (e.g., the Internet) or a private network (e.g., an intranet). In this manner, the exemplary system 100 may include multiple separate devices configured to communicate electronic information via wireless connections. Additionally or alternatively, some of the components may be integrated into a single device.

For clarity of explanation, the augmented reality cosmetic design filter 110 is also referred to as a cosmetic design 110. The cosmetic design 110 is a numerical representation of a cosmetic routine, including a set of textures, mapping data, surface information, and meta information, stored in memory of the client computing device(s) 130, server(s) 160, and/or user device(s) 190. Rather than a single texture that reproduces the final result of the cosmetic routine on a particular face, which is projected onto a plane and then used as a texture mask, the cosmetic design 110 includes one or more traces 115 that reference a contour map of the associated facial features 120 and/or baseline features 125 of the target area. The cosmetic design 110 may also include data describing the dynamic elements 117 of the traces 115. In this manner, the cosmetic design 110 can be deployed on the user device 190 as an animated augmented reality filter that reproduces the layered application of the cosmetic design 110 step by step.

As an illustrative example, the client computing device 130 may incorporate a camera 140. Similarly, the exemplary system 100 may include multiple client computing devices 130. The first client computing device 130 is a mobile electronic device (e.g., tablet, smartphone, or laptop) configured to host user interface elements and connect to the server(s) 160 via the network(s) 170. The second client computing device 130 is integrated with the camera 140 and configured to coordinate the operation of the camera 140 with the first client computing device 130. The components of the exemplary system 100 may be provided with computer-executable instructions (e.g., software, firmware, etc.) for implementing and coordinating the operation of one or more features of the exemplary system 100. In this manner, a user interface may be accessed via one or more components, and the operation of the exemplary system 100 may be coordinated via the interface.

The client computing device(s) 130 may be or may include a purpose-built mobile computing device including a camera 140, one or more electromagnetic (EM) radiation sources (illustrated as part of the camera 140), and one or more user interface elements 131 for prompting the biological surface 180 with visual and/or audio prompts. For example, the interface element 131 may be or may include a display 133 for generating a visual representation of the cosmetic design 110. The interface element 131 may also include user input components including, but not limited to, a touch screen, keyboard, trackpad, or mouse. In this example, the components of the client computing device 130 may be integrated into a housing resembling a consumer cosmetic product, such as a bathroom or vanity mirror. In this example, the housing may hide a power source, a thermal management system, and other components.

Although the client computing device 130 and the camera 140 are illustrated in a particular configuration, additional and/or alternative form factors are contemplated. For example, the system 100 may include a smartphone or tablet computer in communication with the client computing device 130, such that one or more computer-executable operations are performed by the smartphone or tablet computer. In this manner, image generation operations may be performed by a smartphone incorporating the camera 140, and image processing operations may be performed, at least in part, on a separate device, such as the server(s) 160.

In some embodiments, the camera 140 may be or include multiple sensors and/or sources, including, but not limited to, a visible light image sensor 141, a depth sensor 143, and/or a source of invisible EM radiation 145, including, but not limited to, infrared or near infrared. Like the client computing device(s) 130, the camera 140 may include communication circuitry 147 to enable wireless communication and/or pairing with other components of the exemplary system 100. Although the camera 140 is illustrated as a separate component of the exemplary system 100, the camera 140 may also be integrated into one of the other components of the exemplary system 100. For example, the client computing device 130 may incorporate the camera 140 (e.g., as a smartphone or tablet).

The depth sensor 143 can capture one or more images of the biological surface 180, including but not limited to an image of a target body surface 181 of the biological surface 180. In the illustration provided in FIG. 1, the biological surface 180 is a human using the exemplary system 100 to create a cosmetic design 110, and the target body surface 181 is the human face in the area around the eyes and eyebrows. The depth sensor 143 can generate a surface mapping 183 of the target body surface 181. The contour and depth information of the target body surface 181 can change over time or between users. The camera can generate the surface mapping 183 as part of an operation for extrapolation, modification, and/or generation of the cosmetic design 110 by the client computing device(s) 130.

The depth sensor 143 may be an image sensor and may capture an image within a field of view 185 that includes the target body surface 181. The depth sensor 143 may be or may include, but is not limited to, a laser-based sensor (e.g., LiDAR), a time-of-flight (ToF) camera, a structured light generator, a visual simultaneous localization and mapping (vSLAM) sensor assembly including a motion sensor and a visible image camera, or an ultrasonic-based sensor assembly, such that the camera 140 can generate a surface mapping 183.

For example, if the depth sensor 143 is an infrared depth sensing camera, the invisible EM radiation source 145 can be or include an infrared source that exposes the biological surface 180, including the target body surface 181, to structured invisible infrared radiation. In another example, if the depth sensor is a LiDAR system or a ToF camera, the invisible EM radiation source 145 can be or include an infrared diode laser. In this manner, the EM radiation generated by the invisible EM radiation source 145 can be scanned or otherwise directed toward the target body surface 181 over an angular spread 187 such that the target body surface 181 is exposed. In some embodiments, detection of the target body surface 181 is facilitated and/or enabled by feature and/or edge detection applied to a visible spectrum (e.g., RGB) image captured by the visible light sensor 141 (e.g., by vSLAM technology).

The surface mapping 183 can provide contour and/or position information regarding features of the target body surface 181, such as the relative positions of the eyebrow ridge and the bridge of the nose, precise information regarding where the eyebrows begin and end in relation to the eyes, etc. In this manner, the surface mapping 183 can be used to generate and/or extrapolate the cosmetic design 110 for deployment on social media platforms or other services, and the cosmetic design 110 can be distributed to users via user device(s) 190. In this manner, the cosmetic design 110 can be represented as a data file communicated to the server(s) 160, for example, as part of an online platform and/or database of cosmetic designs.

The cosmetic design 110 can be modified by using surface mapping 183 to determine the shape type of features of the biological surface 180 and converting the cosmetic design 110 to another, different shape type. In some embodiments, multiple layers are defined as part of the process of creating or modifying the cosmetic design 110. The exemplary system 100 can implement the exemplary techniques described with reference to Figures 2-6 to detect the application of multiple cosmetic formulations to the target body surface 181 and generate one or more traces 115 that spatially and temporally describe the application, which together define the cosmetic design 110.

Advantageously, the cosmetic design 110 can be distributed to the user device 190(s) as an augmented reality filter, for example through integration with social media, with improved accuracy and precision by mapping both the target body surface 181 and the corresponding area of the user of the user device 190 and extrapolating the cosmetic design 110, or each constituent layer of the cosmetic design 110, to fit the corresponding area. Furthermore, the cosmetic design 110 can be dynamically presented in the form of a tutorial visualization, whereby both layers and traces are animated to show the layers, movement, colors, and other cosmetic aspects of the cosmetic design 110. In this way, the cosmetic design 110 represents an improved user experience by adapting the cosmetic design 110 from the creator to the user, and also represents an improved system performance, since the complete cosmetic design 110 can be transferred as mask data rather than as image data. This reduces the amount of data and computational resources involved in distributing the cosmetic design 110 over the network(s) 170 and storing it on the server(s) 160.

2 is a schematic diagram illustrating an exemplary process 200 for preparing an augmented reality cosmetic design filter 110, according to various embodiments. The exemplary process 200 may be implemented as a number of operations performed or otherwise performed by components of the exemplary system 100 of FIG. 1. As such, the operations may be or may include operations performed by one or more processors of a computer system, such as the client computing device(s) 130 of FIG. 1, in response to execution of computer-readable instructions stored on a non-transitory memory of the computer system. Although the operations are illustrated in a particular order, the exemplary process 200 may include more or fewer operations, and the order of operations may vary. In some embodiments, some operations are performed by different physical computers, as described in more detail with reference to FIG. 1. For example, some operations may be performed interchangeably by different components, or may be performed by the coordinated operation of two or more components.

In operation 201, the computer system defines a baseline description 210 of the biological surface 180. In some embodiments, the baseline description 210 is defined as part of the process of generating the surface mapping 183, for example, by referencing visible color information to surface contour mapping information describing the biological surface 180. In this manner, the baseline description 210 can include a numerical representation (e.g., digital data) of the biological surface 180 that includes both the surface mapping 183 and color information. Although the exemplary process 200 is illustrated in the context of the target body surface 181 of FIG. 1, it will be understood that the biological surface 180 can be or include a face, a hand, a portion of a face, or other surface to which a cosmetic formulation is applied.

In some embodiments, defining the baseline description 210 includes generating one or more images of the biological surface 180 using a camera 140 or other radiation sensor configured to detect visible light or invisible EM radiation. The invisible EM radiation can be projected onto the biological surface 180 using a radiation source of the camera 140, such as a structured light generator, a LiDAR light source, or a ToF camera light source. For example, the image can depict the facial features 120 and the baseline features 125 of the biological surface 180. The baseline features 125 can be or include materials applied to the biological surface 180 prior to the exemplary process 200 such that at least a portion of the biological surface 180 includes a pigment, cosmetic formulation, or other feature detectable by the camera 140. For example, the baseline features 125 are illustrated as eyelashes and eyeliner. Similarly, the baseline features 125 can include, but are not limited to, blemishes, wrinkles, blemishes, or other aesthetic appearances of the biological surface 180 other than the facial features 120. In contrast, facial features 120 refer to the organs, hair, and other topographical features of the biological surface 180 that define the surface mapping 183 and affect the baseline description 210 through physical effects including, but not limited to, shadows, perspective, etc.

In operation 203, the computer system identifies one or more applications 215 of cosmetic formulations 220(s) to the biological surface 180. The cosmetic formulations 220(s) may include, but are not limited to, makeup, mascara, lipstick, lip liner, eyeliner, glitter, beauty creams, ointments, or other materials that may be topically applied to the biological surface 180. The application 215(s) of the cosmetic formulations 220(s) may be detected by the camera 140 using a radiation sensor, such as a visible light sensor, an invisible EM radiation sensor, or the like. In this manner, the application 215(s) may be detected as a shift of the biological surface relative to a baseline description, for example, in terms of coloration, surface reflectance, or absorbance/reflectance of invisible EM radiation. In an illustrative example, a glossy lip coating may be detected through an increase in specular reflectance of the biological surface 180 relative to a baseline description that includes negligible specular reflectance. Similarly, a matte foundation may be detected through a decrease in specular reflectance and an increase in diffuse reflectance relative to the baseline description. In a non-limiting example, specular reflection can be detected by defining a luminance channel in the visible light image and thresholding the luminance channel. Specular reflection is defined, on a pixel-by-pixel basis, as a luminance value that exceeds a threshold.

In some embodiments, the computer system detects an applicator 225 (such as a brush, sponge, pencil, finger, or other applicator 225) in proximity to the biological surface 180. In some embodiments, the computer system recognizes the identifier of the applicator 225, for example, if the applicator 225 includes a registration mark (such as a barcode, QR code, RFID tag, or other representation of identifier information). In this manner, the computer system can spatially and temporally identify the application 215 and can identify the cosmetic formulation(s) 220. To this end, operation 203 can optionally include identifying the cosmetic formulation 220 and using the identifier information of the cosmetic formulation 220 to define the color of the cosmetic formulation 220. For example, the color of the unidentified cosmetic formulation 220 can be estimated using the camera 140, as described in more detail with reference to FIG. 6. The process of identifying the applicator 225 may allow the computer system to directly determine the color of the cosmetic formulation 220 that the particular cosmetic formulation 220 is loaded with, rather than the applicator 225 being a multi-purpose applicator 225. Examples of applicators 225 that may identify the cosmetic formulation 220 in this manner include, but are not limited to, pencils, lipsticks, gel applicators, or other applicators 225 available with the prepared cosmetic formulation 220. The application 215 of the cosmetic formulation 220 may include a movement 230 of the applicator 225 relative to the biological surface 180. The movement 230 may include meaningful information regarding the aesthetic effect of the overall cosmetic design 110, such as blending and smudge movements. In this manner, the movement 230 of the applicator 225 may be detected and used during the operation to generate a trace.

The application 215 may also include a modification or manipulation of the formulation 220 already placed on the biological surface 180. For example, the coverage, coloration, and reflectance of the cosmetic formulation 220 may be modified by smoothing, blurring, blending, spreading, or other techniques. Such applications 215 that modify an already placed formulation 220 rather than adding a new formulation 220 to the biological surface 180 may be identified by detecting the tool or finger and detecting changes to the biological surface 180 relative to the baseline description 210. As described with reference to FIG. 1, a baseline description 210 may be associated with each application 215, where the applications 215 are layered and/or blended. For example, the baseline description 210 for a particular area of the biological surface 180 where the formulation 220 has already been applied may be redefined to include the formulation 220. In this manner, subsequent applications 215 may be detected more accurately and precisely.

In operation 205, the computer system generates one or more traces 115 describing the applications 215 identified in operation 203. Generating the traces 115 may include determining the extent 217, layer order (see FIG. 5), and coloring (see FIG. 6) of the applications 215 such that the cosmetic design 110 can realistically reproduce the aesthetic effects of the applications 215 and also reproduce the order of the applications 215. The extent 217, coloring, and layer order information, as well as other information related to realistically reproducing the traces 115 on the user device(s) 190, may be encoded as a numerical representation, such as an intensity data set referenced to the baseline description 210. For example, the extent 217 of the applications 215, illustrated as dashed lines in FIG. 2, may be detected as a shift in coloring, absorbance and/or reflectance of EM radiation compared to the baseline description. In this manner, the traces 115 may reproduce the aesthetic effects of the individual applications 215.

In some embodiments, the computer system tracks the applicator movement 230 using image and/or contour data generated by the camera 140. Thus, generating the trace 115 may include generating a numerical representation of the various movements 230 relative to the biological surface 180, as described in the baseline description, and generating color information describing the cosmetic formulation 220 used in the application 215. Once generated, the trace 115 can be transformed to reproduce the aesthetic effect of the cosmetic design 110 on various shapes of biological surfaces 180 when viewed through user device(s) 190, as described in more detail with reference to FIG. 3.

The traces 115 may encode layer blending and layering information such that a combination of multiple traces 115 may reproduce interactions between the traces 115. For example, a set of applications 215 of multiple eyeshadow colors to a region of the biological surface 180 between the eye and eyebrows may be blended to impart a color gradient to the eyeshadow region. In this example, the exemplary process 200 includes multiple iterations of operations 201-205 such that the region between the eyebrow and the eye is described by multiple traces 115 and also by interaction terms that describe how the multiple traces relate. In some embodiments, the traces 115 include transparency information, such as alpha channel information, referenced to dynamic elements 117 that describe the local transparency of the coloration or other aesthetic effects of the applications 215. In this manner, the traces 115 may reproduce edits of the cosmetic formulation 220, such as blending, blurring, smoothing, or other techniques typical of cosmetic design. In one illustrative example, the trace 115 and/or the dynamic element 117 can include a vector of transparency values that reference the movement 230 and the formulation 220 such that the trace 115 replicates the smoothing or blending of the formulation 220 based on the movement 230.

In operation 207, the computer system outputs the trace(s) 115 detected as part of operations 201-205. Outputting the trace(s) 115 may include electronic operations including, but not limited to, storing the trace(s) 115 in a local data store of the client computing device 130, transferring the trace(s) 115 over the network 170 to the server(s) 160, and/or sharing the trace(s) 115 with the user device(s) 190 directly (e.g., via electronic pairing) or over the network 170.

While the description of the exemplary process 200 has focused on cosmetic design for the brow/eye area, the operations may be applied to other surfaces as well. For example, the cosmetic design 110 may describe the application 215 of a cosmetic formulation 220 to additional/alternative surfaces, including but not limited to the lips, nose, cheeks, forehead, or hands. Similarly, the cosmetic design 110 may modify the appearance of facial features 120, including but not limited to the brows, eyes, lips, cheekbones, jawline, or hands. In another illustrative example, the cosmetic design 110 may include a series of traces 115 to enhance the appearance of the cheekbones through the application of one or more cosmetic formulations 220.

3 is a schematic diagram illustrating an example process 300 for deploying an augmented reality cosmetic design filter 110, according to various embodiments. Similar to the example process 200 of FIG. 2, the example process 300 may be implemented as a number of operations performed or carried out by components of the example system 100 of FIG. 1. As such, the operations may be or may include operations performed by one or more processors of a computer system, such as the user device 190(s) of FIG. 1, in response to execution of computer-readable instructions stored on a non-transitory memory of the computer system. Although the operations are illustrated in a particular order, the example process 300 may include more or fewer operations, and the order of operations may vary. In some embodiments, some operations are performed by different physical computers, as described in more detail with reference to FIG. 1. For example, some operations may be performed interchangeably by different components, or may be performed by the coordinated operation of two or more components, such as the user device 190(s) and the server 160(s).

In operation 301, the user device 190 receives data 310 describing the cosmetic design 110. Receiving the data 310 may include electronic communication over the network 170 (e.g., the Internet, a cellular network, etc.), but may also include direct communication via pairing or other approaches. While FIG. 3 illustrates the user device 190 receiving the data 310 from the server(s) 160, it will be understood that the data 310 may be received from the client computing device(s) 130. The data 310 may include traces 115 generated during one or more iterations of the exemplary process 200, as described in more detail with reference to FIG. 2. The data 310 may also include meta-information describing the biological surface 180, such as a shape type of the biological surface 180, as described in more detail with reference to FIG. 4.

The operation for receiving the data 310 may include one or more data transfer techniques, including, but not limited to, wireless communication or wired communication. For example, the user device 190 may wirelessly communicate with the client computing device 130 to receive the data 310 as a wireless transmission.

In some embodiments, receiving the data 310 may include a browser environment, such as a recommendation engine, whereby a user of the user device 190 may view a set of cosmetic designs 110 generated by one or more creators, rather than directly communicating with the creators to request designs. Additionally or alternatively, the data 310 may be pushed to the user device 190, for example, as part of a social media service to which the user of the user device 190 may register, follow, and/or subscribe. In some embodiments, the data 310 is recommended based at least in part on identifier information describing the user of the user device 190. If the user of the user device 190 indicates aesthetic preferences through historical traffic data, cosmetic designs 110 that reflect these preferences may be preferentially recommended. Demographic, socioeconomic, or biometric data may also be used to recommend cosmetic designs 110. For example, if the user of the user device 190 resides in a certain geographic region, cosmetic designs 110 that reflect the trends of this geographic region may be recommended. Similarly, if a user of the user device 190 is within a given age range, employed in a given field or sector, or part of a given social network, cosmetic designs that reflect trends corresponding to the respective categories can be identified and recommended.

In operation 303, the user device 190 maps a user surface 315 corresponding to the biological surface 180 to which the cosmetic design 110 is virtually applied. The mapping of the user surface 315 may include operations similar to those described with respect to generating the surface mapping 183 of FIG. 1, including but not limited to structured light methods, LiDAR or ToF methods, vSLAM methods, ultrasound methods, etc. The surface 315 may be described by a mesh of polygons 317, each of which may define multiple properties for a corresponding region of the surface 315, including but not limited to tone, reflectance, temperature, etc.

If the user device 190 is a smartphone incorporating a depth sensor 319, operation 303 may include executing one or more applications stored in memory of the user device 190 to map the surface using the depth sensor. In this manner, the user surface 315 may be represented as a numerical representation in data describing the surface contours and facial features 320 of the user surface 315. In some embodiments, the exemplary process 300 includes integrating the cosmetic design 110 with the existing aesthetic appearance of the user surface 315. To this end, aesthetic features 321 such as makeup, mascara, eyelashes, etc. are included in the surface mapping.

Mapping the user surface 315 may also include estimating a surface tone, which may be a local surface tone, as described in more detail with reference to FIG. 6. In this manner, the exemplary process 300 may take into account differences in skin tones between the biological surface 180 and the user surface 315. For example, the metadata provided as part of the data 310, or alternatively stored in memory of the server(s) 160 and/or the user device 190, may include a reference or look-up table that cross-references colors to skin tones so that the cosmetic design 110 may be adapted to reproduce the aesthetic effect of the cosmetic design 110, rather than a literal transposition of the exact colors.

In act 305, the traces 115 received as part of the data 310 are used to generate filters 325 that reference the mapping of the user surface 315 generated in act 303. The filters 325 may be or include a numerical representation, such as a texture object, formatted to be combined with the mapping of the user surface 315. In an illustrative example, the user surface 315 is described by a mesh of polygons 317, and the filters 325 reference the polygons 317 such that each polygon 317 defines a vector of values that describes the appearance of the filter 325.

In some embodiments, the filter 325 includes dynamic elements 330, such as animation effects or inter-layer effects, that can be used to indicate the sequence of applications 215 used to create the cosmetic design 110. The dynamic elements 330 can replicate at least a portion of the dynamic elements 117 described with reference to the traces 115 of FIGS. 1-2.

In operation 307, the computer system presents the filter 325 integrated with the image 335 of the user surface 315. Presenting the filter 325 may include displaying the filter 325 through a screen 337. For example, the user device 190 may be a smartphone or tablet that includes a touch screen display. The filter 325 may be mapped to the user surface 315 in real time or near real time using feature tracking or other augmented reality techniques, where "near real time" represents a qualitative experience by a user of the user device 190. This makes the latency introduced by the operation of the exemplary process 300 substantially imperceptible.

The filters 325 can be presented as interactive graphical elements that a user of the user device 190 can manipulate through the screen 337, for example, by selecting or highlighting. In some embodiments, selecting the filter 325 can initiate animation of the dynamic elements 330 in an optional action 309. Additionally or alternatively, animating the filter 325 can include initiating a virtual tutorial, whereby the filters 325 are presented in order by layer order such that the order of the application 215 is reproduced on the user surface 315. The order can be paused, reversed, advanced, or otherwise manipulated via the user device 190. A particular dynamic element can be activated, for example, by selecting the filter 325 with a user action 340 on the screen 337.

In some embodiments, interaction with the screen 337 while the filters 325 are presented allows the user of the user device 190 to edit, add, or remove one or more filters 325. Different user actions 340 can initiate one or more alternative processes instead of activating a dynamic element. For example, a double finger tap can open a menu 341 where the user of the user device 190 can view the applicator 225, the cosmetic preparation 220, or other information describing the filter 325. Similarly, a long press on the filter 325 can allow the user device 190 to access an online marketplace where the user can purchase the cosmetic preparation 220, the applicator 225, or other consumer products. In some embodiments, a menu of social media controls 343 can be generated such that the user of the user device can use the filter 325 to share the image 335, communicate with other users, or provide feedback or otherwise communicate with the creator of the cosmetic design 110. It will be understood that the types of user actions described are exemplary. User interface elements, such as menu 341 and/or social media menu 343, can be invoked by alternative and/or additional interactions. In some embodiments, menu elements are presented by default on display 337.

4 is a schematic diagram illustrating an exemplary process 400 for converting an augmented reality cosmetic design filter 110 from one shape type to another shape type, according to various embodiments. As with the exemplary processes 200 and 300, the exemplary process 400 may be implemented as a number of operations performed or otherwise performed by components of the exemplary system 100 of FIG. 1. As such, the operations may be or include operations performed by one or more processors of a computer system, such as the client computing device(s) 130, server(s) 160, and/or user device(s) 190 of FIG. 1, in response to execution of computer-readable instructions stored on a non-transitory memory of the computer system. Although the operations are illustrated in a particular order, the exemplary process 400 may include more or fewer operations, and the order of the operations may vary. In some embodiments, some operations are performed by different physical computers, as described in more detail with reference to FIG. 1. For example, some operations may be performed interchangeably by different components, or may be performed by the coordinated operation of two or more components.

The trace 115 is generated with reference to the baseline description 210, as described in more detail with reference to FIG. 2, and is itself specific to the biological surface 180. Thus, the exemplary process 400 describes operations for extrapolating the filter 325 from a first facial shape to a second facial shape as part of generating the trace 115 and/or the filter 325. While the description with reference to FIG. 4 focuses on the shape of the face, it will be appreciated that operations can be similarly applied to the shapes of individual facial features, such as the shape of the eyes, mouth, or other features, to transform the trace 115 to realistically reproduce the aesthetic effect of the trace 115, rather than simply applying the trace 115 as a texture to the user surface 315.

In operation 401, the computer system generates mask data 410 using one or more traces 115. If the traces 115 are relative to topographical data describing the biological surface, generating the mask data 410 includes projecting data describing the application 215 onto the topographical data describing the biological surface 180. This allows the traces 115 to be modified from one shape type 420 to another shape type 420.

In operation 403, the computer system attributes a first shape type 420 to at least a portion of the biological surface 180. For example, the computer system may reference shape information describing the plurality of shape types 420. The shape information may be stored on, accessed by, or otherwise received from the server(s) 160 or the client computing device(s) 130. In this manner, the exemplary process 400 may include receiving information describing the plurality of shape types 420.

Assigning a shape type 420 to the biological surface 180 may include referencing the baseline description 210 to determine characteristic dimensions, spacing, or other aspects of the biological surface 180 that are indicative of the shape type 420. In the illustrative example of a face shape, the characteristic aspects may be defined by projecting a grid or other form of guide 421 onto the baseline description 210 such that the shape type 420 may be identified by defining a contour 423 of the baseline description 210, by identifying a characteristic curvature 425 at one or more points of the baseline description 210, or by identifying a characteristic spacing 427 between features. It will be appreciated that approaches including edge detection and feature detection using computer image processing techniques may be employed to assign the shape type 420.

In operation 405, the computer system generates one or more shape filters 430 for the different shape types 420. As part of generating the shape filters 430, the mask data 410 generated in operation 401 is transformed from the first shape type 420-1 to the second shape type 420-2, the third shape type 420-3, or an additional or alternative shape type 420. The transformation of the mask data 410 may include, but is not limited to, projecting the mask data 410 of the first shape type 420-1 onto topographical data of the second shape type 420-2 and applying a corresponding transformation to the trace 115. In this manner, the shape filter 430 reproduces the aesthetic shape and coloring of the trace 115 onto a shape type 420 different from the first shape type 420-1.

In some embodiments, generating shape filters 430 for different shape types 420 can include additional modifications to the mask data 410 beyond spatial projection to account for different physical shapes. For example, some shape types 420 are characterized by more pronounced or thinner facial features, such as brow ridges, forehead curvature, jaw shape, cheekbones, chin shape, etc. Thus, the shape filter 430 can rearrange, split, omit, duplicate, mirror, or otherwise transform the mask data 410 as part of generating the shape filter 430. In one illustrative example, generating a shape filter 430 for a first shape type 420-1 corresponding to a heart-shaped face shape to a second shape type 420-2 corresponding to a rectangular face shape includes rearranging the forehead trace upward toward the hairline, extending the forehead trace to both ends of the forehead, and splitting and mirroring the chin trace along the jaw line. A corresponding transformation can be applied to the mask data 410 for each trace 115 as part of the process of generating a shape filter 430 for each shape type 420.

In the context of facial features, where the cosmetic design 110 addresses a particular facial feature (e.g., eyeshadow design between the eyebrows and the eyes), the composition operations of the exemplary process 400 can be applied based on similar principles. For example, where the cosmetic design 110 is created with reference to a round eye shape type 420, generating a shape filter 430 in operation 405 can include converting the mask data 410 from round eyes to an almond eye shape type 420 or other eye specific shape type 420. It will be appreciated that corresponding operations can be applied to the shape of lips, hands, ears, or other biological surfaces.

5 is a schematic diagram illustrating an exemplary process 500 for generating an augmented reality cosmetic design filter 510 in multiple layers, according to various embodiments. As with the exemplary processes 200, 300, and 400, the exemplary process 500 may be implemented as a number of operations performed or otherwise performed by components of the exemplary system 100 of FIG. 1. As such, the operations may be or include operations performed by one or more processors of a computer system, such as the client computing device(s) 130, server(s) 160, and/or user device(s) 190 of FIG. 1, in response to execution of computer-readable instructions stored on a non-transitory memory of the computer system. Although the operations are illustrated in a particular order, the exemplary process 500 may include more or fewer operations, and the order of operations may vary. In some embodiments, some operations are performed by different physical computers, as described in more detail with reference to FIG. 1. For example, some operations may be performed interchangeably by different components, or may be performed by the coordinated operation of two or more components.

As described in more detail with reference to FIGS. 1-3, the trace 115 can be generated by recording the sequential application of multiple cosmetic formulations 220 to the biological surface 180 such that the augmented reality cosmetic design filter 510 can be partitioned into separate layers. As described in more detail with reference to FIG. 3, the presentation of the filter 510 can recreate the layered order of application. Similarly, the exemplary process 400 can also incorporate the layered aspects of the exemplary process 500, where the different shape types 420 refer to different layering processes, inter-layer blending, and more complex light and shadow effects resulting from the differences in the shape types 420.

In operations 501-507, a series of layers are defined, each layer including one or more traces 115. For example, in operation 501, a first layer is defined including a forehead trace 515 and a cheekbone trace 517. In operation 503, a second layer is defined including a forehead trace 520(s), a cheek trace 521, and a chin trace 523(s). In operation 505, a highlight layer(s) is defined that includes a highlight trace(s) 525 for highlighting or highlighting one or more facial features, overlaid on the first and second layers. In operation 507, an eye layer(s) is defined that includes an eyeshadow trace 530 and an under-eye trace 531, overlaid on the highlight layer, the first layer, and the second layer.

In this manner, the augmented reality cosmetic design filter 510 can dynamically add or remove layers during presentation on the user device 190. Similarly, the client computing device 130 can associate one or more traces 115 with each of the multiple layers. Defining the layers as described in the exemplary process 500 allows for blending rules to be defined between the layers. For example, a higher order layer can overlay a lower order layer. A higher order layer can completely obscure a lower order layer. Transparency or partial transparency, defined, for example, through the alpha channel of the mask data 410, allows for multiple layers to be overlaid in the image 335 to create a net aesthetic effect that includes co-localized contributions from multiple layers.

Figure 6 is a schematic diagram illustrating an exemplary process 600 for determining formulation color using surface tone and formulation tone, according to various embodiments.

As will be described in more detail with reference to FIG. 2, detecting the application 215 of the cosmetic formulation 220 may include determining the color or the cosmetic formulation 220 and/or identifying the cosmetic formulation 220. If the technique for determining the color includes image processing operations based on the image generated by the camera 140, the color of the various cosmetic formulations 220 may be affected by the tone of the biological surface 180 in proximity to the application 215. Thus, a direct reading of the image pixel values may produce inaccurate color rendition when used in the filter 325.

In operation 601, the computer system measures a number of regions 610 corresponding to the application 215 and estimates a surface tone 615 of the biological surface 180 for each of the regions 610. In some embodiments, the operations of the exemplary process 600 are performed in parallel with the exemplary process 200, but may also be performed at different times using the baseline description 210, images recorded during generation of the trace 115, or a calibration set of images generated prior to beginning the exemplary process 200. It is expected that the surface tone 615 may vary locally between regions 610, for example, where facial features cast shadows or where the surface defines one or more contours. Thus, operation 601 may include defining multiple surface tones 610 for a single application 215.

In operation 603, the computer system measures the regions 610 in the image containing the application 215 and estimates one or more local formulation tones 625 of the cosmetic formulation 220. Estimating the surface tone 615 and formulation tone 625 may include sampling one or more pixels in each region 610 and determining an average coloration of each region 610. It will be appreciated that the formulation tone 625 is influenced by the underlying surface tone 615 such that the surface tone 615 can serve as a reference baseline value for estimating the formulation tone in a skin-tone sensitive manner. Advantageously, by implementing a computer image processing approach that includes estimating the surface tone 615 in this manner, the cosmetic design 110 can be responsive to the skin tone of the user, rather than presenting a static augmented reality mask.

In operation 605, the computer system determines the color 630 of the cosmetic formulation using the surface tone 615 and the formulation tone 625. In contrast to the surface tone 615 and the formulation tone 625, it is contemplated that each cosmetic formulation 220 includes one or more pigments that are uniformly dispersed or distributed such that the formulation appears to be a single visible color, for example, through diffuse reflection of visible light. To this end, the formulation color 630 can be or include a vector of color components, such as an RGB triad or another color representation approach.

As part of operation 605, corresponding surface tones 615 and formulation tones 625 can be compared, and the overall impact of the surface tone 615 on the formulation tone 625 can be considered as part of the process of determining the color 630 of the cosmetic formulation 220. In an illustrative example, when different regions 610 are illuminated differently, comparing the surface tone 615 and the formulation tone 625 can control the impact of lighting on color rendition, for example, by using the surface tone 615 to compensate for luminance variations in the formulation tone 625.

It will be appreciated that the process of determining formulation color 630 can improve the reproducibility of the aesthetic effect of the cosmetic design 110 on various biological surfaces, under various ambient conditions, and with various formulations. As described in more detail with reference to FIG. 2, formulation color 630 can be modified to account for skin color differences between creators and users. Similarly, formulation color 630 can be modified to improve the fidelity of realistic colors based on ambient conditions. For example, under bright conditions, the intensity of the color can be increased, while under dim conditions, the intensity of the color can be decreased. In some cases, the color shade can also be dynamically modified to account for light and shape. For example, as part of the exemplary process 300, formulation color 630 can be modified to include a greater blue component to reflect that the corresponding trace 115 is in shade. Conversely, the blue component can be decreased when the corresponding trace 115 is moved into bright lighting.

As described throughout the foregoing description, it is contemplated that the operations for generating, modifying, and/or presenting the trace 115 and the filter 325 are performed by one or more electronic devices including computer circuitry, communication modules, and computer-readable instructions stored in memory. Through the coordinated operation of the components of the exemplary system 100 of FIG. 1, an augmented reality filter can be generated and presented in near real-time to a user of a user device 190 integrated with a social media platform. The augmented reality filter can realistically reproduce the aesthetic effects of a makeup design, taking into account differences in face shape, skin tone, and ambient conditions. The augmented reality filter can be dynamically presented to enhance sharing of makeup routine sequences beyond what is currently possible with augmented reality or traditional social media.

7 is a block diagram illustrating an exemplary system 700 including components of the system of FIG. 1, according to various embodiments. The exemplary system 700 includes a client computing device 130 in electronic communication (e.g., via a network 170) with a remote computer system 160. The exemplary system 700 illustrates an example of the system 100 of FIG. 1 in the context of related system elements, and as such illustrates electronics and software for performing operations as described with reference to FIGS. 2-6. FIG. 7 illustrates non-limiting examples of system elements, features, and configurations, and many other features and configurations are contemplated. In the example illustrated in FIG. 7, the client computing device 130 of FIG. 1 includes a computer system 710, a number of components 720 for interacting with a biological surface 180, a computer-readable medium 730, and a client application 740, which may be stored as computer-executable instructions on the computer-readable medium 730 and which, when executed by the computer system 710, may implement operations described with reference to the system 100 of FIG. 1 and the exemplary techniques of FIGS. 2-3.

The client computing device 130 may incorporate auxiliary components, including but not limited to a power source 711, a human machine interface 713, one or more processors 715, a network interface 717, and may include a computer-readable medium 730. The power source 711 may be a direct current power source, such as a rechargeable battery or a rectified power source configured to connect to a line power source (e.g., 110V AC, 220V AC, etc.). The human machine interface (HMI) 713 may include any type of device capable of receiving user input or generating output for presentation to a user. The any type of device may be a speaker for audio output, a microphone for receiving voice commands, a push button switch, a toggle switch, a capacitive switch, a rotary switch, a slide switch, a rocker switch, or a touch screen.

One or more processors 715 are configured to execute computer-executable instructions stored on computer-readable medium 730. In one embodiment, processor(s) 715 are configured to send signals to and/or receive signals from component 720 via a communications bus or other circuitry, for example as part of the execution of client application 740. Network interface 717 is configured to send signals to and receive signals from client computing device 130 (or other computing device) on behalf of processor 715. The network interface 717 may implement any suitable communication technology, including, but not limited to, short-range wireless technologies such as Bluetooth, infrared, near field communication, and Wi-Fi, long-range wireless technologies such as WiMAX, 2G, 3G, 4G, LTE, and 10G, and wired technologies such as USB, FireWire, Thunderbolt, and/or Ethernet. The computer-readable medium 730 is any type of computer-readable medium capable of storing computer-executable instructions, including, but not limited to, flash memory (SSD), ROM, EPROM, EEPROM, and FPGA. The computer-readable medium 730 and the processor(s) 715 may be combined into a single device, such as an ASIC. Alternatively, the computer-readable medium 730 may include cache memory, registers, or other parts of the processor 715.

In the illustrated embodiment, the computer-readable medium 730 stores computer-executable instructions that, in response to execution by one or more processors 715, cause the client computing device 130 to implement a control engine 731. The control engine 731 controls one or more aspects of the client computing device 130, as described above. In some embodiments, the computer-executable instructions are configured to cause the client computing device 130 to perform one or more operations, such as generating a surface mapping of a target surface, generating a trace, or outputting the trace to a server(s) 160. In some embodiments, the control engine 731 controls basic functions by facilitating interaction between the computer system 710 and the components 720 according to the client application 740. In some embodiments, the control engine 731 detects an input from the HMI 713 indicating that a makeup routine is to be initiated (e.g., in response to activation of a power switch or a "start" button, or in response to detection of a face in front of the camera 140 of FIG. 1), or receives a signal from the client computing device 130(s), the remote computer system 160, or the user device 190(s) (e.g., via a Bluetooth® paired connection).

The components of the client computing device 130 can be adapted to the application or can be application (e.g., ASIC) specific. For example, the components 720 can include one or more cameras 721, a display 723, one or more radiation sources 725, and/or one or more radiation sensors 727, as described in more detail with reference to FIG. 1. In some embodiments, the components 720 are integrated into a single device. In this manner, the client computing device 130 can be a special-purpose computing device configured to execute the client application 740 in conjunction with the components 720. In some embodiments, the client computing device 130 is a general-purpose mobile electronic device, such as a tablet or smartphone, that stores the client application 740.

In some embodiments, the client application 740 also includes an image capture/3D scanning engine 741. The image capture/3D scanning engine 741 is configured to capture and process digital images (e.g., color images, infrared images, depth images, etc.) obtained from one or more of the parts 720, including but not limited to stereoscopic images, LiDAR data, or other forms of surface/depth sensing information. In some embodiments, such data is used to obtain a clean and accurate 3D contour mapping of the target body surface (e.g., target surface 181 of FIG. 1). In some embodiments, the digital images or scans are processed by the client computing device 130 and/or transmitted to the remote computer system 160 for processing by the 3D model engine 781. In one embodiment, the captured image data is used by a position tracking engine 743 to determine the location of key points, edges, or features on the target body surface. In some embodiments, the position tracking engine 743 tracks the contours of the target body surface in 3D space, for example, by implementing v-SLAM technology. In some embodiments, the position information from the position tracking engine 743 is used to generate a signal that is sent to the control engine 731. This signal is used to control one or more parts 720 or elements of the computer system 710, including, for example, the source 725 or the HMI 713, in accordance with the techniques described herein.

In some embodiments, the digital 3D models described herein are generated based on sensor data acquired by the client computing device 130. As such, the digital 3D models are generated by the client computing device 130, other computing devices such as cloud computing systems, or a combination thereof. In some embodiments, the digital 3D models include 3D topology and texture information, which can be used to recreate accurate representations of body surfaces, such as facial structure and skin features, as described in more detail with reference to Figures 1-6.

In some embodiments, the client application 740 includes a user interface 745. In one embodiment, the user interface 745 includes interactive features including, but not limited to, graphical guides or prompts presented via a display to guide the user in selecting a cosmetic design, tutorial video, or animation. In some embodiments, the user interface 745 provides guidance (e.g., visual guides such as arrows and targets, progress indicators, audio/haptic feedback, synthesized speech, etc.) to direct the user under particular lighting conditions, angles, etc. to ensure sufficient data is collected for use by the mapping and projection engine.

The client application 740 may include a mapping module 747. The mapping module 747 may be or may include computer readable instructions (e.g., software, drivers, etc.) for converting a numerical representation of a cosmetic design into an augmented reality cosmetic design filter. As part of the operation of the mapping module 747, the client application 740 may receive real-time data from the camera(s) 721 and the sensor 727. This data may be processed by the 3D scanning engine 741, the position tracking engine 743. This data may be used to progressively update the mapping and cosmetic design developed during a series of applications to the target body surface. In this manner, the mapping module 747 may respond to movements of the target body surface, thereby increasing the tolerance of the client computing device 130 to movements on the part of the user without losing fidelity to the cosmetic design filter. In some embodiments, the computational resource demands for such real-time scanning/tracking may be distributed across multiple devices, such as the remote computer system 160, via parallelization or distribution routines.

The communication module 749 of the client application 740 can be used to prepare information for transmission to, or receive and interpret information from, other devices or systems, such as the remote computer system 160 or user device(s) 190, as described in more detail with reference to FIG. 1. Such information can include captured digital images, scans, or videos, personal care device settings, custom care routines, user preferences, user identifiers, device identifiers, and the like. In one embodiment, the client computing device 130 collects data describing the performance of a care routine, body surface image data, or other data. In one embodiment, such data is transmitted to the remote computer system 160 via the network interface 717 for further processing or storage (e.g., in the product data store 783 or the user profile data store 785). The client computing device 130 can be used by a consumer, a personal care professional, or some other entity to interact with other components of the system 700, such as the remote computer system 160 or the user device(s) 190. In one embodiment, the client computing device 130 is a mobile computing device, such as a smartphone or tablet computing device, that includes the components 720 and the client application 740, or that includes the components through electronic coupling with peripheral devices.

Next, exemplary components and functions of the remote computer system 160 are described. The remote computer system 160 includes one or more server computers implementing one or more of the illustrated components, for example in a cloud computing arrangement. The remote computer system 160 includes a projection engine 787, a 3D model engine 781, a product data store 783, and a user profile data store 785. In one embodiment, the 3D model engine 781 uses image data (e.g., color image data, infrared image data) and depth data to generate a 3D model of the target body surface. The image data is obtained from the client computing device 130, for example from a camera(s) 721 or a sensor(s) 727 integrated with or otherwise electronically coupled to the client computing device 130. In one embodiment, the image data and depth data associated with the user are stored in the user profile data store 785. In one embodiment, the user's consent is obtained before storing any information that is personal information about the user or that can be used to identify the user.

In one embodiment, the mapping/projection engine 787 performs processing of data related to the makeup routine, such as using image/sensor data to generate a mapping of the target surface and/or generate a projection of the makeup routine as an augmented reality filter. The data related to the makeup routine can then be transmitted to the user device(s) 190. In some embodiments, the projection engine 787 generates makeup design data using user information from the user profile data store 785, the product data store 783, the 3D model engine 781, or other sources or combinations of sources.

The 3D model engine 781 may employ machine learning or artificial intelligence techniques (e.g., template matching, feature extraction and matching, classification, artificial neural networks, deep learning architectures, genetic algorithms, etc.). For example, to generate a cosmetic design for a facial surface mapping, the 3D model engine 781 may analyze the facial mapping generated by the 3D model engine 781 to measure or map the contours, wrinkles, skin texture, etc. of the target body surface. The 3D model engine 781 may receive data describing the cosmetic design based on an identifier code provided by a user through the user device(s) 190. In such a scenario, the 3D model engine 781 may use such information to generate an augmented reality cosmetic design filter projection of the cosmetic design (e.g., cosmetic design 110 of FIG. 1) onto the image of the user's corresponding body surface.

The devices shown in FIG. 7 can communicate with each other via a network 170. The network 170 can include any suitable communication technology, including, but not limited to, wired technologies such as DSL, Ethernet, fiber optics, USB, Firewire, Thunderbolt, wireless technologies such as WiFi, WiMAX, 3G, 4G, LTE, 5G, 10G, Bluetooth, and private networks (e.g., intranets) or public networks (e.g., the Internet). In general, communication between the computing devices or components of FIG. 7, or other components or computing devices used in accordance with the described embodiments, occurs directly or through intermediate components or devices.

Alternatives to the arrangements disclosed and described with reference to Figures 1 and 7 are possible. For example, functionality described as being implemented in multiple components can instead be integrated into a single component, and functionality described as being implemented in a single component can instead be implemented in multiple illustrated components, or in other components not shown in Figures 1 or 7. As another example, devices in Figures 1 and 7 illustrated as including certain components can instead include more, fewer, or different components without departing from the scope of the described embodiments. As another example, functionality described as being performed by a particular device or subcomponent can instead be performed by one or more other devices in the system. As an example, the 3D model engine 714 can be implemented in the client computing device 130, or in other devices or combinations of devices.

In addition to the technical advantages of the embodiments described elsewhere herein, some embodiments achieve a number of other technical advantages. For example, the system 700 allows some aspects of the process to be performed independently by the personal care device or client computing device, while shifting other processing burdens to the remote computer system 160 (which may be a relatively powerful and reliable computing system). Thus, performance of the functionality provided by the personal care device or client computing device is improved and battery life is preserved.

In general, the term "engine" as used herein refers to logic embodied in hardware or software instructions written in a programming language such as C, C++, COBOL, JAVA, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft.NET, etc. An engine can be compiled into an executable program or written in an interpreted programming language. Software engines can be called by other engines or by themselves. In general, engines as described herein refer to logical modules that can be integrated with other engines or divided into sub-engines. Engines can be stored on any type of computer-readable medium or computer storage device and are stored and executed on one or more general-purpose computers. Thus, engines or special-purpose computers configured to provide this functionality are created.

As will be appreciated by those skilled in the art, a "data store" as described herein can be any suitable device configured to store data for access by a computing device. One example of a data store is a reliable, high-speed relational database management system (DBMS) running on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, any other suitable storage technology and/or device capable of quickly and reliably providing stored data in response to a query can be used. The computing devices can be accessible locally rather than over a network, or can be provided as a cloud-based service. A data store can also include data systematically stored on a computer-readable storage medium, as further described below. Those skilled in the art will recognize that the separate data stores described herein can be combined into a single data store and/or the single data store described herein can be separated into multiple data stores without departing from the scope of the present disclosure.

Figure 8 is a block diagram illustrating aspects of an exemplary computing device 800, according to various embodiments. Although several different types of computing devices are described with reference to various embodiments, the exemplary computing device 800 illustrates various elements common to many different types of computing devices. Although Figure 8 is described with reference to a computing device implemented as a device on a network, the following description is applicable to servers, personal computers, mobile phones, smartphones, tablet computers, embedded computing devices, and other devices that can be used to implement some of the embodiments of the present disclosure. Moreover, those skilled in the art and others will recognize that the computing device 800 can be any one of any number of devices currently available or later developed.

In its most basic configuration, the exemplary computing device 800 includes at least one processor 802 and a system memory 804 connected by a communications bus 806. Depending on the exact configuration and type of device, the system memory 804 can be volatile or non-volatile memory, such as read-only memory ("ROM"), random access memory ("RAM"), EEPROM, flash memory, or similar memory technologies. Those skilled in the art and others will recognize that the system memory 804 typically stores data and/or program modules that are immediately accessible to and/or currently being operated on by the processor 802. In this regard, the processor 802 can function as the computational center of the computing device 800 by supporting the execution of instructions.

As further shown in FIG. 8, the computing device 800 may include a network interface 810 that comprises one or more components for communicating with other devices over a network. An embodiment of the present disclosure may access basic services that utilize the network interface 810 to implement communications using a common network protocol. The network interface 810 may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, 2G, 3G, LTE, WiMAX, Bluetooth, Bluetooth low energy, etc. As will be appreciated by those skilled in the art, the network interface 810 illustrated in FIG. 8 may represent one or more wireless or physical communication interfaces described and illustrated above with respect to certain components of the system 100 of FIG. 1.

In the exemplary embodiment depicted in FIG. 8, the computing device 800 also includes a storage medium 808. However, the service may be accessed using a computing device that does not include a means for persisting data on a local storage medium. Thus, the storage medium 808 depicted in FIG. 8 is represented by a dashed line to indicate that the storage medium 808 is optional. In any case, the storage medium 808 may be volatile or non-volatile, removable or non-removable, and implemented using any technology capable of storing information, including, but not limited to, a hard disk drive, a solid state drive, a CD-ROM, a DVD, or other disk storage, a magnetic cassette, a magnetic tape, a magnetic disk storage, and/or the like.

As used herein, the term "computer-readable medium" includes any method or technology capable of storing information, such as computer-readable instructions, data structures, program modules, or other data, both volatile and non-volatile, as well as removable and non-implemented. In this regard, system memory 804 and storage media 808 depicted in FIG. 8 are merely examples of computer-readable media.

Suitable implementations of a computing device including a processor 802, a system memory 804, a communication bus 806, a storage medium 808, and a network interface 810 are known and commercially available. For ease of illustration and as not important to an understanding of the claimed subject matter, FIG. 8 does not show some of the typical components of many computing devices. In this regard, the exemplary computing device 800 may include input devices such as a keyboard, a keypad, a mouse, a microphone, a touch input device, a touch screen, and/or the like. Such input devices may be coupled to the exemplary computing device 800 by wired or wireless connections, including RF, infrared, serial, parallel, Bluetooth, Bluetooth low energy, USB, or other suitable connection protocols using wireless or physical connections. Similarly, the exemplary computing device 800 may also include output devices such as a display, a speaker, a printer, and the like. These devices are well known in the art and therefore will not be further illustrated or described herein.

While exemplary embodiments have been shown and described, it should be understood that various changes to the exemplary embodiments can be made without departing from the spirit and scope of the present invention. It should be understood that the methods and systems described herein are not limited to specific methods, components, or implementations. It should also be understood that the terminology used herein is for the purpose of describing the embodiments only and is not intended to be limiting.

As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. As used herein, ranges can be expressed as from "about" one particular value and/or to "about" another particular value. When such a range is expressed, another embodiment includes from the one particular value and/or to the other particular value. Similarly, when values are expressed as approximations, it will be understood that the particular value forms another embodiment by use of the antecedent "about." It will further be understood that the endpoints of each range are significant both in relation to the other endpoint, and independently of the other endpoint.

"Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur. The description is intended to include instances in which the event or circumstance occurs and instances in which the event or circumstance does not occur.

Throughout the description and claims of this specification, the word "comprise" and variations of words such as "comprising" and "comprises" mean "including, but not limited to." For example, other components, integers, or steps are not intended to be excluded. "Exemplary" means "one example of" and is not intended to convey an indication of a preferred or ideal embodiment. "Such as" is used in a descriptive, not limiting sense.

Claims (15)

one or more non-transitory computer memory devices storing computer-readable instructions that, when executed by computing circuitry, cause the computing circuitry to perform operations for generating an augmented reality cosmetic design filter;
The operation includes:
defining a baseline description of the biological surface;
Identifying an application for the cosmetic formulation;
generating a trace; and outputting the trace,
The step of defining a baseline description of the biological surface includes defining the baseline description of the biological surface using one or more radiation sensors;
the radiation sensor is electronically coupled to the computing circuitry;
The step of identifying the application of the cosmetic formulation includes identifying the application of the cosmetic formulation to the biological surface using one or more of the radiation sensors;
The step of generating the trace comprises generating the trace describing the application with reference to the baseline description in one or more non-transitory computer memory devices. 10. The one or more non-transitory computer memory devices of claim 1,
The step of identifying the application of the cosmetic formulation comprises:
one or more non-transitory computer memory devices, comprising detecting the cosmetic formulation relative to the biological surface using one or more of the radiation sensors and the baseline description. 10. The one or more non-transitory computer memory devices of claim 1,
The step of generating the trace comprises:
receiving information describing a plurality of shape types;
using the baseline description to attribute a first one of the shape types to at least a portion of the biological surface;
generating a numerical representation of the application of the cosmetic formulation; and converting the numerical representation from the first shape type to a second one of the shape types;
One or more non-transitory computer memory devices, wherein the numerical representation describes position information relative to the baseline description. 4. One or more non-transitory computer memory devices according to claim 3,
the biological surface is a face,
One or more non-transitory computer memory devices, wherein the first shape type and the second shape type describe facial features. 10. The one or more non-transitory computer memory devices of claim 1,
one or more non-transitory computer memory devices, wherein defining the baseline description of the biological surface comprises projecting invisible electromagnetic radiation onto the biological surface with a radiation source electronically coupled to the computing circuitry. 10. The one or more non-transitory computer memory devices of claim 1,
One or more non-transitory computer memory devices, wherein the one or more radiation sensors comprise a camera configured to detect electromagnetic radiation from the ultraviolet, visible or infrared spectral ranges, or a combination thereof. 10. The one or more non-transitory computer memory devices of claim 1,
The step of generating the trace comprises:
tracking a movement of the application relative to the biological surface; and generating a numerical representation of the movement relative to the baseline description. 10. The one or more non-transitory computer memory devices of claim 1,
The instructions, when executed by the computing circuitry, cause the computing circuitry to perform further operations, the operations including:
one or more non-transitory computer memory devices, comprising: identifying the cosmetic formulation; and defining a color of the cosmetic formulation using the cosmetic formulation identifier information. 10. The one or more non-transitory computer memory devices of claim 1,
The instructions, when executed by the computing circuitry, cause the computing circuitry to perform further operations, the operations including:
estimating a surface tone of the biological surface;
one or more non-transitory computer memory devices, comprising: estimating a formulation tone of the cosmetic formulation using the surface tone; and determining a color of the cosmetic formulation using the surface tone and the formulation tone. 10. The one or more non-transitory computer memory devices of claim 1,
The one or more non-transitory computer memory devices are electronically coupled to a smartphone. 10. The one or more non-transitory computer memory devices of claim 1,
One or more non-transitory computer memory devices, wherein outputting the trace comprises communicating the trace to a remote computing system. 1. A method for generating an augmented reality cosmetic design filter, comprising:
defining a baseline description of the biological surface using one or more radiation sources and one or more radiation sensors;
using the radiation sensor to identify application of a cosmetic formulation to the biological surface;
generating a trace describing the application with reference to the baseline description; and outputting the trace. 13. The method of claim 12,
The step of generating the trace comprises:
receiving shape information describing a plurality of shape types;
using the shape information to attribute a first shape type among the shape types to at least a portion of the biological surface;
generating a numerical representation of the application of the cosmetic formulation;
converting the numerical representation from the first shape type to a second shape type;
The method of claim 1, wherein the numerical representation describes position information relative to the baseline description. 13. The method of claim 12,
The step of identifying the application of the cosmetic formulation comprises:
detecting an applicator of the cosmetic preparation;
estimating the position of the applicator of the cosmetic formulation relative to the biological surface;
tracking a movement of the applicator relative to the biological surface; and generating a numerical representation of the movement relative to the baseline description. 13. The method of claim 12,
estimating a surface tone of the biological surface;
estimating a formulation tone of the cosmetic formulation; and determining a color of the cosmetic formulation using the surface tone and the formulation tone.