KR20200098316A - Apparatus and method for evaluating the emotional preference of portrait images using an expert viewpoint - Google Patents

Abstract

The present invention is a technology related to an automatic portrait photo evaluation device and method based on viewer preference by systematically combining objective and subjective factors when evaluating a portrait. A face recognition unit that recognizes a face in a portrait photo, a face recognition unit recognized A composition evaluation unit that evaluates whether the composition of the face corresponds to the preference composition, and an emotional color evaluation unit that evaluates the degree of matching between the expression and the color tone of the portrait photograph, recognizing expressions from the face recognized by the face recognition unit, and composition And a calculation unit that calculates a comprehensive evaluation using the evaluation results of the evaluation unit and the emotional color evaluation unit, and a display unit that outputs the comprehensive evaluation calculated by the calculation unit.

Description

A device and method for evaluating the emotional preference of a portrait using a professional perspective {APPARATUS AND METHOD FOR EVALUATING THE EMOTIONAL PREFERENCE OF PORTRAIT IMAGES USING AN EXPERT VIEWPOINT}

The present invention relates to an apparatus and method for evaluating the emotional preference of a portrait photograph using an expert perspective, and more particularly, an automatic portrait photograph based on the viewer preference by systematically combining an objective element and a subjective element when evaluating a portrait photograph. It is a description of an evaluation device and method.

The etymology of the camera originated from the camera obscura, which painters used for sketching. In the 19th century, experiments to fix light energy by chemical methods began, and in 1826, Joseph Niepce first printed photographs, and the base of photographs was enlarged by using silver dyeing emulsion. In the 20th century, Kodak's digital camera development expanded the scope of photography and developed into a technology that anyone can use easily. Since then, digital cameras have rapidly spread to consumers with high quality and low prices by various manufacturers. In recent years, mobile phones are equipped with cameras of comparable quality to digital cameras, making them an indispensable daily necessity for modern people.

As the position of image-based products such as digital cameras changed, consumers' product selection criteria also changed. In the past, consumers placed emphasis on the objective reproducibility perspective, such as price and objective figures of specifications, for the selection criteria for image-based products. Accordingly, manufacturers also conducted marketing objectively emphasizing high quality and product excellence through various media. However, recently, consumers are choosing products with more weight on subjective image quality such as aesthetic elements and emotional elements of photography. As such, according to consumer demand, manufacturers are evolving in the direction of developing products, prioritizing customer-oriented quality satisfaction.

It is difficult to say that the product's image reproduction performance has a direct relationship with the image quality that consumers prefer. This is because the perception of objective attributes of images and psychophysical cognitive processes work together in the process of consumers' image quality evaluation, and subjective reactions accordingly appear as final preference for image quality (Clark, 2006; Li, 2011). Therefore, it is unreasonable to judge image quality only by objective reproduction performance of image-based products.

In order to solve these problems, experts in the field of image tried to analyze the subjective preferences of viewers through the objective elements of the image. The International Organization for Standardization (International Organization for Standardization) defined objective image quality factors of photographs and created an index to evaluate images from an objective point of view. Based on the results of prior research, Hyungju Park (2013) defined the quality evaluation items as three highly important items in total, dynamic range, color, and contrast, specifying the range of reproducing performance of digital image quality preferred by ordinary viewers. Roh Yeon-suk (2012) analyzed the factors affecting the image quality satisfaction from the viewer's point of view by using language, so that the image can be evaluated from a subjective point of view. He said that if language is used to derive emotional meanings for certain concepts, relatively clear interpretive meanings can be given to aspects that are difficult to explain, thereby securing objectivity of research. Through the results of these preliminary studies, it is possible to analyze the preferences of images more accurately than before, and many methods have been proposed to allow developers to understand the quality of photos preferred by consumers and apply them to image-based products. However, these studies were limited to a few specific factors, and the experiments were conducted, and the results were of the degree to which the correlation between them was investigated. In addition, it was unreasonable to say that it was possible to fully analyze the emotions of the viewers because it did not take into account that the individual's judgment criterion could be shaken due to mental and external environmental factors.

Registered Patent Publication No. 10-0983007

Accordingly, the present invention has been proposed in order to solve the problems of the related art as described above, and an object of the present invention is an apparatus and method for automatic portrait photo evaluation based on viewer preference by systematically combining objective and subjective elements when evaluating a portrait. The task is to provide

In order to achieve the above object, an apparatus for evaluating the emotional preference of a portrait photo using an expert perspective according to the technical idea of the present invention includes: a face recognition unit for recognizing a face in a portrait photo; A composition evaluation unit that evaluates whether the composition of the face recognized by the face recognition unit corresponds to a preference composition; An emotional color evaluation unit for recognizing an expression on the face recognized by the face recognition unit and evaluating a degree of matching between the expression and a color tone of the portrait photograph; An operation unit that calculates a comprehensive evaluation using the evaluation results of the composition evaluation unit and the emotional color evaluation unit; And a display unit for outputting the comprehensive evaluation calculated by the operation unit.

The composition evaluation unit may set a plurality of feature points on the face using a CLM-Framework algorithm, and calculate values of rotation, left and right tilt, and front and rear tilt of the face using the feature points.

In addition, the emotional color evaluation unit may be characterized in that the expression is classified into positive, neutral, and negative.

In addition, when the expression is positive and the color tone is a warm tone, the emotional color evaluation unit may determine that the expression and the color tone of the portrait photo match.

In addition, when the expression is negative and the color tone is a cool tone, the emotional color evaluation unit may determine that the expression and the color tone of the portrait photo match.

In addition, a contrast evaluation unit for evaluating the contrast by analyzing the portrait photo by RSC (Retinal-like Subsampling Contrast) may be further included, and the operation unit may further use an evaluation result of the contrast evaluation unit during a comprehensive evaluation operation. .

In addition, it further comprises a dynamic region evaluation unit for evaluating the dynamic range (dynamic range) using the photoelectric conversion function of the portrait, the calculation unit further using the evaluation result of the dynamic region evaluation unit in the comprehensive evaluation calculation. I can.

In addition, an exposure evaluation unit for evaluating an exposure degree of the face recognized by the face recognition unit may be further included, and the operation unit may further use an evaluation result of the exposure evaluation unit during a comprehensive evaluation operation.

In addition, the exposure evaluation unit is characterized in that the exposure degree is evaluated by changing the portrait photo into a YCbCr coordinate system to obtain a Y value, and stepping the Y value using the principle of a zone system. I can.

In addition, a weight evaluation unit for evaluating the weight of the size of the face recognized by the face recognition unit in the portrait photograph may be further included, and the calculation unit may further use an evaluation result of the weight evaluation unit during a comprehensive evaluation operation.

In addition, the weight evaluation unit, after classifying the portrait photo into one of a long shot, a full shot, a medium shot, and a close-up shot, Correspondingly, it may be characterized in that the size specific gravity is evaluated.

In addition, a position evaluation unit for evaluating whether the position of the face recognized by the face recognition unit corresponds to a position of the third division law, and the operation unit may further use an evaluation result of the position evaluation unit during a comprehensive evaluation operation. .

In addition, it further comprises a focus evaluation unit for evaluating the sharpness of the face by recognizing the depth of field of the face recognized by the face recognition unit, wherein the calculation unit further uses the evaluation result of the focus evaluation unit when performing a comprehensive evaluation operation. I can.

On the other hand, in order to achieve the above object, a method for evaluating the emotional preference of a portrait photo using an expert perspective according to the technical idea of the present invention includes the steps of: recognizing a face in the portrait photo by a face recognition unit; Evaluating whether a composition of the face recognized by the face recognition unit corresponds to a preference composition, and evaluating a degree of matching between the facial expression and the color tone of the portrait photograph; Calculating a comprehensive evaluation using the evaluation result of the evaluation unit; And outputting, by the display unit, the comprehensive evaluation calculated by the calculation unit.

In addition, the evaluation unit sets a plurality of feature points on the face using the CLM-Framework algorithm, and calculates values of rotation, left and right tilt, and front and rear tilt of the face using the feature points to quantify the composition of the face. It can be characterized.

In addition, the evaluation unit may be characterized in that the expression is classified as positive, neutral, or negative.

In addition, the evaluation unit may determine that if the expression is positive and the color tone is a warm tone, it is determined that the expression and the color tone of the portrait photo match.

In addition, when the expression is negative and the color tone is a cool tone, the evaluation unit may determine that the expression and the color tone of the portrait photo match.

In addition, the evaluation unit may be characterized in that the contrast is evaluated by analyzing the portrait photo by RSC (Retinal-like Subsampling Contrast).

In addition, the evaluation unit may be characterized in that the dynamic range is evaluated using a photoelectric conversion function for the portrait picture.

In addition, the evaluation unit may be characterized in that it evaluates the degree of exposure of the face recognized by the face recognition unit.

In addition, the evaluation unit may be characterized in that the exposure degree is evaluated by changing the portrait photo to a YCbCr coordinate system to obtain a Y value, and stepping the Y value using a principle of a zone system. .

In addition, the evaluation unit may be characterized in that the weight of the size of the face recognized by the face recognition unit in the portrait picture is evaluated.

In addition, the evaluation unit classifies the portrait photo into one of a long shot, a full shot, a medium shot, and a close-up shot, and then corresponds to the type of shot of the portrait photo. It may be characterized by evaluating the size specific gravity.

In addition, the evaluation unit may be characterized in that it evaluates whether the position of the face recognized by the face recognition unit corresponds to the position of the third division law.

In addition, the evaluation unit may be characterized by evaluating the degree of sharpness of the face by recognizing the depth of field of the face recognized by the face recognition unit.

According to the apparatus and method for evaluating the emotional preference of a portrait photograph using an expert perspective according to the present invention,

First, by using the present invention, an image is evaluated by combining an objective element and a subjective element, and all elements are summed in an appropriate score combination method according to their importance, so that more accurate viewers' preference results can be expressed than the results of prior studies. .

Second, the professionalism and completeness of the program can be enhanced by adding new elements that photography experts consider when evaluating portrait photography.

Third, the present invention reflects the perspectives of professional photographers and is based on the process they go through when they actually work on photography. Therefore, it is better to supplement the images that consumers prefer and what results the viewers prefer. It is possible to give specific feedback on photos to users who use this program by presenting numerical values such as whether they can be reborn as photos.

1 is a configuration diagram of an apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective according to an embodiment of the present invention.
2 is an exemplary diagram showing a range of tones generated by photographing the same scene with cameras having different dynamic ranges.
3 is a diagram showing a portrait photograph divided into equal parts by the rule of third division.
FIG. 4 is a diagram illustrating that when a subject is focused using a camera, light is not focused to exactly one point on the sensor surface, thereby forming a depth region.
5 is a distribution diagram of a group model compared to whether or not focus is present.
6 is a diagram showing the probability of a specific value in a function capable of representing only a range using CDF and a probability distribution.
7 is a result of the CLM-Framework algorithm setting a plurality of feature points on a face of a portrait, and calculating values of the face rotation (Turn), left and right tilt (Tilt), and front and rear tilt (Up/Down) using the feature points. Figure showing.
8 is a diagram showing an HSV color model.
9 is a diagram showing classification of an AU (Action Unit) and classification of an expression by a combination of AUs.
10 is a diagram showing classification of color tones.
FIG. 11 is an exemplary diagram illustrating determining whether to be out of focus by comparing a level of focus between an area occupied by a person and a rest area in order to determine whether a subject is out of focus.
12 is a diagram showing a configuration of a display unit according to an embodiment of the present invention.
13 is a view showing a method of evaluating the emotional preference of a portrait photograph using an expert perspective according to an embodiment of the present invention.
14 is a diagram showing the results of comprehensive evaluation of 10 portrait photos that received a number of recommendations from Flickr.
15 is a view showing the overall evaluation results of 10 portrait photos that received poor recommendation from Flickr.
16 and 17 are diagrams showing face focus and focus values of 10 portrait photos that received multiple recommendations and 10 portrait photos that received poorly recommended recommendations.
18 and 19 are diagrams showing RSC and dynamic range (D/R) results of 10 portrait photos that received multiple recommendations and 10 portrait photos that received poorly recommended recommendations.
20 and 21 are views showing face exposure and exposure values of 10 portrait photos that received a number of recommendations and 10 portrait photos that received poorly recommended recommendations.
22 and 23 are diagrams showing whether or not the rule of third division of 10 portrait photos that received a number of recommendations and 10 portrait photos that received poor recommendations, and face weight and weight values.
24 and 25 are diagrams showing subject focus values and background focus values of 10 portrait photos that received a number of recommendations and 10 portrait photos that received poorly recommended recommendations.
26 and 27 are diagrams showing face composition values of 10 portrait photos that received a number of recommendations, and 10 portrait photos that received poor recommendations, and a fracture result of whether or not the optimal composition corresponds.
28 and 29 are diagrams showing a result of matching color tones and emotions of 10 portrait photos that received a number of recommendations and 10 portrait photos that received poor recommendations.

With reference to the accompanying drawings, an apparatus and method for evaluating the emotional preference of a portrait photo using an expert perspective according to embodiments of the present invention will be described in detail. Since the present invention can apply various changes and have various forms, specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific form of disclosure, and it should be understood that all changes, equivalents, and substitutes included in the spirit and scope of the present invention are included. In describing each drawing, similar reference numerals have been used for similar elements.

In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

The apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective according to an embodiment of the present invention is installed and operated in one computer device, or distributedly installed in a plurality of computer devices, and each computer device is connected to each other by wire or wirelessly. It can be implemented as being interlocked.

Referring to FIG. 1, in this embodiment, a face recognition unit 110 for recognizing a face in a portrait photograph, and a composition evaluation unit 135 for evaluating whether a composition of a face recognized by the face recognition unit 110 corresponds to a preference composition. , An emotional color evaluation unit 136, a composition evaluation unit 135, and an emotion that recognizes an expression on the face recognized by the face recognition unit 110 and evaluates the degree of matching between the expression and the color tone of a portrait photograph. The color evaluation unit 136 includes an operation unit 160 that calculates a general evaluation using the evaluation result of the color evaluation unit 136, and a display unit 180 that outputs the general evaluation calculated by the operation unit 160.

This embodiment provides a new automatic portrait photo evaluation based on viewer preference by systematically combining objective and subjective factors when evaluating a portrait.

Objective factors include RSC (Retinal-Like Subsampling Contrast) and dynamic range. RSC stands for the viewer's perceptual contrast, and allows the contrast to be measured in a way that reflects the viewer's overall impression of the image and the human perspective. The dynamic range can be defined as the range of the tones of an image, and a camera with a wide dynamic range can express the range from the brightest to the darkest parts close to human vision. It is possible to evaluate objective quality evaluation items derived from each image using a statistical method, Cumulative Distribution Function (CDF). The CDF is a form of integrating a probability density function, and is a method that can represent the probability of a specific value in a function that can only represent a probability range on a probability distribution.

The subjective element analyzes face exposure, weight, position, composition, out-of-focus, and color tone based on face recognition and the overall impression of the image. The exposure of the face is evaluated by utilizing the emotional response data of the viewers and applying the basic principles of the Digital Zone System and the automatic exposure setting function of the mobile phone camera. The position, weight, composition, and out of focus of the face in a portrait photo are also utilized by viewers' emotional response data. Color tone allows you to grasp the emotional response according to the color tone through the analysis of the HSV color model. This can combine a color tone that is highly correlated with the viewer's emotion, and can evaluate the emotional preference of a general image. [Table 1] shows the classification of objective evaluation items and subjective evaluation items.

Classification Evaluation item Objective evaluation items RSC Dynamic range Subjective evaluation items Face exposure Specific gravity of the face Face position Composition of the face Out of focus Color tone

Image influences viewers by combining various elements. And each element works organically with each other and influences the final preference of the viewers.

Evaluation items classified in the objective and subjective evaluation items are numerically measured by the calculation method set in this embodiment. The total score of the image can be calculated by applying the derived numerical values for each evaluation item to the evaluation item priority calculation method designed based on the preferences of viewers.

Image quality refers to the performance of equipment that reproduces images such as cameras, monitors, and printers, and refers to a composition that encompasses various characteristics related to marketing, technology manufacturing, and safety that meet consumer expectations when using products or services. do. This comprehensive configuration may cause many problems due to different technology differences between products, and may act as a factor of image quality deterioration.

Deterioration in image quality may occur during the image generation and acquisition (Acquisition) process. Problems such as defocusing, improper exposure level or motion blur are the causes of image quality deterioration that are well known to the public. In addition, image quality deterioration may occur due to optical and electronic problems such as light diffraction, distortion, chromatic aberration, and sensor noise. These causes can be improved to a certain level of quality through investment of time, effort, and cost in the process of image creation and acquisition.

In addition, image quality deterioration may occur during image processing. The moment it goes through the image processing process, the information becomes different from the original data, leading to a factor of quality degradation. As a representative example, image compression may be stored, and information loss that cannot be recovered during the digitization process may occur in the process of adjusting brightness and contrast.

Also, image quality deterioration may occur during image reproduction. When the viewer enjoys an image, the display of the monitor, the limitation of resolution, loss of brightness and color, etc. contribute to the deterioration of image quality.

Small and large causes may cause image quality degradation in the imaging system. These problems can be solved by putting the most advanced technology of the time. However, it is practically limited to always produce products with expensive technology and parts taking into account all the issues. Therefore, it is important that developers always keep in mind the balance between image quality and the cost of implementing an image reproduction system and optimize the system to provide the maximum image quality at a minimum cost. For example, the quality of DVD is generally superior to that of analog VHS. It is difficult to say that the quality of DVD is a flawless and perfect product, but it is because the distortion and deterioration of the image have been reduced to a minimum due to the advancement of advanced technology. As a result, the picture quality of DVDs improved, but the general public paid more than VHS.

Keelan (2002) defined that the quality of an image is merely that viewers accept or judge an image as a certain impression. In addition, Yendrikhovskij (2002) defined image quality as subjective impression of the viewer, and he argued that the change according to how perfectly the elements constituting the image are in harmony can influence the image quality.

ISO 20462 (2004) classifies the attributes that affect image quality into an artificial defect attribute and a preferential attribute. The artificial defect property means an objective defect seen in an image and causes the overall image quality to deteriorate. The preferential attribute refers to the attribute of clear image quality in a specific image, and refers to attributes that can have a preferential effect on the viewer, such as hue, saturation, and contrast.

Image quality attributes can be classified into artificial defect attributes, preferred attributes, aesthetic attributes, and personal attributes. [Table 2] is a score table for objectively evaluating the quality of images within each attribute.

Attribute group Property item Possibility of objective control Intergroup correlation System dependency total Artificial defect haziness +1 +1 +1 +3 Particle +1 +1 +1 +3 Red eye +1 +1 +1 +3 Digital glitch +1 +1 +1 +3 Preferred Color balance +1 +1 +1 +3 prepare +1 0 +1 +2 Color clarity +1 0 +1 +2 Memory color reproduction +1 0 +1 +2 Aesthetic Light quality of lighting 0 0 0 0 ninth 0 0 0 0 Personal Preservation of memories -One -One -One -3 information transfer -One -One -One -3

Artificial defect properties include blurriness, graininess, red eye, and digital defects. Preferred attributes include color balance, contrast, color clarity, and memory color reproduction. The aesthetic properties include the light quality and composition of the lighting. Personal attributes include preservation of memories and transmission of information.

In this way, there are various factors that affect image quality, and factors that can be measured from the viewpoint of image processing, such as objective attributes, and those that do not interact with each other represent the final image quality.

Solso (2000) defined visual information processing as having three stages.

The first is a step in which light reflected from a specific object passes through the lens of the eye and is refracted to focus on the retina. This is a basic perception process, and it is converted into nerve energy through various electrochemical actions and reactions.

The second step is to analyze detailed features such as points, lines, and planes among the visual stimuli, and then transmit them to the brain in the area responsible for the features through a neural network.

The third is the step in which the transmitted signal is transmitted to the brain in another area and is associated with the knowledge stored in the existing human brain. Solso argues that since existing knowledge information and new visual information are allied and reflected in such visual information, it is necessary to focus on understanding the interpretation method of human perception.

Among the three stages of visual information processing, the first and second stages are referred to as upward processing in cognitive psychology, and the third stage is referred to as downward processing (Yeon-suk Ro, 2010). After the step of recognizing visual information, the individual's final emotion is determined.

Sensitivity is the opposite of rationality, and refers to a human's sensuous inner response to a certain stimulus. Therefore, certain stimuli can elicit intuitive and reflexive thought responses. In addition, since emotion can always be influenced by all external stimuli, it is a factor that affects changes in individual thinking and overall thinking about images. As such, the meaning of emotion does not appear as a remarkable behavioral change due to the fusion of a person's outer and inner reactions, and it was difficult for individuals to grasp and identify the change in emotion, so not only an academic approach but also a user-centered In terms of product development, the area of sensibility tends to be inaccessible. That is, from the viewpoint of a photo, a positive or negative emotion may be induced to a user according to an emotion inducing factor included in the photo frame. Therefore, it is necessary to be aware of the overall human sensibility that can affect the shift of thinking by being influenced by all external stimuli at all times.

In perceptual psychology, research was conducted by dividing into physiological, physical, and cognitive psychological approaches based on the characteristics of human sensory organs and physiological response data to scientifically identify the difficulty and ambiguity of emotion measurement (Myers, 2002). These studies can be used to understand and classify human visual characteristics, and have been used as preliminary research results to increase the use rate and satisfaction of imaging products through image quality improvement, evaluation, and development. However, it is difficult to apply it to reality due to realistic problems such as the time required for accurate emotional measurement, cost and difficulty in measuring standards.

Sang-Hyeon Ji (2002) conducted a study on the emotional response that can be elicited from the viewer by using words such as'bright, dark' and'simple, complex' using the semantic discrimination scale. In addition, Shin Soo-jin (2004) quantitatively analyzed the visual effects and sensibility of photographic techniques of specific photos. He asked about 200 test subjects to evaluate a total of 180 pictures as adjective vocabulary. With this experiment, he argued that depending on the combination of changes such as the location, weight, hue, and saturation of the subject, it can induce various sensibilities such as intensity and quietness.

Woo-Hyun Jung (2006) studied how changes in brightness, contrast, and hue of photographs affect the viewer's emotional response. He said that the darker the picture's brightness, the more negative reactions it elicits, and in the opposite case, it elicits a positive reaction from the viewers. The above studies are meaningful in that they quantitatively analyze the relationship between adjective vocabulary and photography, and prove that changes and combinations of elements of photography influence the emotional response of the viewer.

Considering the results of previous studies, it is possible to see what kind of emotional reaction the many elements that make up the photo, such as photography techniques and the components of the photo, elicit the viewer's emotional response. In addition, it can be seen that a systematic emotional response system can be formed when several factors are combined. If the user's emotional response data for a specific element is used, it can be determined that it is possible to develop a product that satisfies general users.

The face recognition technology implemented by the face recognition unit 110 is a technology for extracting, classifying, and confirming feature information individually possessed by a person. In recent years, with the development of advanced devices, technological developments have been made with security and convenience as major factors. In addition, face recognition technology is in the spotlight in the image reproduction industry. Currently, face recognition is supported by most imaging systems from DSLR cameras to mobile phone cameras. Due to this, the emergence of new technologies allows the general public to operate the camera more easily than before, and as a result, it is possible to obtain fast and high quality images.

The facial recognition process can be broadly divided into detection of a face region, extraction of facial features, and recognition or confirmation of faces (Moon, 2013). First, after acquiring an image of an area determined as a face, it is necessary to separate only the face area within the area. In this process, a face recognition system with a high recognition rate can be constructed only when the face area is accurately extracted. After the face detection step, the image of the face region that has undergone a normalization process under conditions of illumination and various modifications can be finally determined whose face is through a recognition algorithm.

In order to use the face recognition function of the latest image reproduction products, a clear face region extraction process is first required. This is because technologies provided through facial recognition, such as exposure, focus, and out-of-focus, can be used smoothly only when an accurate face area is extracted with a camera. With the recent advancement of face recognition technology, it is possible to detect a face area even in a wide variety of environments, but many supplements are still required. The reason for this is that the extraction accuracy may be lowered due to internal factors such as changes in facial expressions, poses, and directions of humans and external factors such as defense of natural and artificial lighting when detecting facial areas (Moon, 2013).

The face template matching method is a method of finding a face area by comparing a standard template of a face with an input face image. This method is divided into two types. The first is to train a large amount of face sample model images in machine learning, and check how much the input face image matches the trained model. The second method is the opposite of the first method, and learns a large amount of images without faces in machine learning. After that, the face area is checked by comparing it with the input image.

In general, a person's face color has a characteristic color that can be distinguished from other objects or backgrounds. Accordingly, the face recognition unit 110 may extract a face area in real time by setting a typical face color of a person for quick extraction and recognition.

The face recognition unit 110 extracts features of the face area. In the facial feature extraction, facial feature elements such as eyes, eyebrows, nose, and mouth are preferentially extracted. After that, a face region is detected by calculating vector values between facial feature elements. Based on the detection result of the facial area, the facial features of the individual are found. These elements are used as clues that can be used in the recognition stage after detection of the face area.

The face recognition unit 110 uses Principal Component Analysis (PCA). PCA is a statistical method that lowers a high-dimensional signal to a low-dimensional signal. This method was used in the early development of face recognition technology, and it is learned by constructing a scattering matrix defined by a correlation coefficient. This method obtains an Eigen Space, and the vector corresponding to this space is called an EigenFace (Yang, 2016). This eigenface is calculated by solving the eigenvalue and eigenvector problem in linear algebra for the covariance matrix of the training images (Hyunjun Moon, 2013). Therefore, the PCA method is sometimes called EigenFace.

The face recognition unit 110 uses LDA (Linear Discriminant Analysis). LDA is a method that can distinguish between changes in identity and changes due to other causes. It attempts to recognize multiple lights for a registered face, and enhances recognition characteristics for changes in lighting, enabling identification of the same person. This method can be obtained through transformation that minimizes the variance between face images belonging to different identity groups (Hagen, 2018). Therefore, it requires a variety of images of one person. For example, you can speak with facial images and various expressions under various lighting. The algorithms of the PCA and LDA methods have the advantage of fast calculation speed.

After completing the facial feature extraction step, the face recognition unit 110 verifies face recognition using the extracted feature vectors using a classifier. NNC (Nearest Neighbor Classifiers) and SVM (Support Vector Machines) can be used, and overall system performance can be optimized through the use of the nearest classifier optimized for each module such as facial image acquisition, preprocessing, and feature extraction ( Kang Hyun-woo, 2017).

In addition, the face recognition unit 110 uses an OpenFace library and a CLM-Framework Algorithm with a wide range of applications to determine more accurate face exposure, location, specific gravity, composition, and presence or absence of defocus. This face detection algorithm and library is a deep learning-based face recognition algorithm and can extract a total of 68 feature points. Through this, it is possible to track and analyze Head Pose (Orientation and Motion), Facial Expressions, and Eye gaze. These modalities, individually or together, can play an important role in the analysis of human behavior. In the library of OpenFace, the models of FaceScrub and CASIA-WebFace, which include various faces that differ by race, age, and gender, as well as lighting and facial expressions, are trained.

This embodiment further includes a contrast evaluation unit 123 for evaluating the contrast by analyzing the portrait photo by RSC (Retinal-like Subsampling Contrast). In addition, the calculating unit 160 further uses the evaluation result of the contrast evaluation unit 123 in the comprehensive evaluation operation.

Contrast is the difference in visual characteristics that makes it possible to distinguish an object from a background. In real human vision, contrast is determined by the difference in color and brightness between objects (Michelson, 1927). In this way, Rizzi (2008) developed RSC (Retinal-Like Subsampling Contrast), which is the viewer's perceptual contrast, in order to measure the contrast in a way that reflects the overall impression of the image and the human perspective. As a result of performing contrast image quality evaluation on RSC for the general public and experts, RSC and contrast showed a high correlation coefficient of 0.84. According to this number, RSC can be judged as a very accurate method of measuring image contrast. RSC is based on the human visual perception process and takes into account the local information and the overall impression of the image. The RSC measurement algorithm considers the brightness and color characteristics of the image by reflecting the chromaticity coordinate values a and b of the CIE LAB color space (see Equation 1).

[Equation 1]

는 RGB의 각 채널,

는 Difference Of Gaussian,

은 레벨 값,

과

은 가로와 세로의 레벨 값,

는 주변의 너비, x,y는 픽셀의 좌표, L*a*b는 CIEL*a*b의 색공간 좌표의 값,

는 각 채널에 가중되는 상수이다(박형주, 2016).

Is each channel of RGB,

Is the Difference Of Gaussian,

Is the level value,

and

Is the horizontal and vertical level values,

Is the surrounding width, x,y are the coordinates of the pixels, L*a*b is the value of the color space coordinates of CIEL*a*b,

Is a constant weighted to each channel (Hyungju Park, 2016).

In addition, this embodiment further includes a dynamic region evaluation unit 124 for evaluating a dynamic range of a portrait photograph using a photoelectric conversion function. The operation unit 160 further uses the evaluation result of the dynamic region evaluation unit 124 during the comprehensive evaluation operation.

Dynamic range can be defined as the tonal range of an image (Seungyeon Han, 2016). In other words, the range of tones in an image refers to the width and darkness of how wide the range from the brightest part to the darkest part can be expressed. A digital camera with a sensor with a wide dynamic range can visually express naturally from very bright to very dark places. In addition, in digital photos, it can be defined as the weight of the minimum and maximum luminance that can identify the gradation of an image, and can be expressed as an EV (Exposure Value) value, which has the same meaning as exposure latitude in an analog photo. Oh Se-cheol, 2011). These contents were standardized in ISO 14524 using photoelectric conversion functions. Equation 2 is a formula for measuring a dynamic range, and FIG. 2 is an example showing a range of tones displayed by photographing the same scene with a camera having a different dynamic range.

[Equation 2]

In addition, this embodiment further includes an exposure evaluation unit 132 for evaluating the degree of exposure of the face recognized by the face recognition unit 110. The operation unit 160 further uses the evaluation result of the exposure evaluation unit 132 in the comprehensive evaluation operation.

Photography is a compound word of light and graph, meaning'drawn with light'. Light is a very important factor in photography, and you can determine the exposure level of a picture by controlling the light. Exposure determination using a camera can control the physical amount of light by manipulating the aperture and shutter speed. When the amount of light is properly adjusted and the shot is taken, it is expressed as appropriate exposure, when the amount of light is high, it is overexposed, and when the amount of light is small, it is expressed as underexposure. It has been revealed through preliminary research that this level of exposure affects the emotional preferences of viewers. Woo-Hyun Jung (2006) published a study that showed that an image with a lower exposure value exhibited a negative reaction, and an image with an appropriate exposure or a high level of exposure exhibited the opposite reaction to the lower exposure. Recently released cameras are equipped with an automatic exposure setting function that properly adjusts the amount of light so that users can shoot without the burden of exposure control. In portrait photography, the proper level of exposure of the face is the most important visual attribute for viewers. In addition, since the most important subject in a portrait photography is the model's face, even if the background exposure is adequate, it is difficult to expect a high evaluation of the viewer if the exposure of the person is not appropriate.

In addition, this embodiment further includes a weight evaluation unit 133 for evaluating the weight of the size of the face recognized by the face recognition unit 110 in the portrait photograph. The operation unit 160 further uses the evaluation result of the specific gravity evaluation unit 133 during the comprehensive evaluation operation.

The proportion of face size in portrait photography greatly influences the preference of viewers. Therefore, professional photographers always take into account the weight of the face. Jong-Moo Kim (2012) produced five types of experimental images with differences in weight according to the size of the face within the image frame. In addition, the emotional vocabulary related to the size and weight of the face was set, and the change in the viewer's preference was evaluated accordingly. In addition, Hyungjoo Park (2016) measured the change in the viewer's preference by dividing the proportion of portrait photos by 10%. Through these preliminary studies, it was possible to find out the proportion of people who prefer portraits by ordinary people.

The weight evaluation unit 133 classifies a portrait photo into one of a long shot, a full shot, a medium shot, and a close-up shot, and then responds to the type of shot of the portrait photo. Evaluate the size specific gravity.

In portrait photography, shots are divided based on how large the subject appears within the image frame. Since the weight of the face is different depending on the type of shot, the expression to be expressed can be expressed in various ways by using this. In general, there are long shots, full shots, medium shots, and close-ups for shooting people.

Long shots are shots that focus on the background rather than the person, and allow you to understand and appreciate the relationship between the person and the background in the image frame. In particular, the long shot provides a variety of information such as the relationship between people or the relationship between background and people.

A full shot is a shot that takes a person's whole body, and is mainly used when trying to directly express the relationship between people or their actions. Since the weight of the face is similar to the long shot, it is difficult to observe the facial expression of the person, and the situation that the producer wants to express can be presented through the movement of the person.

The medium shot is a shot of a person's upper body. It is mainly used to express the main subject, the person and the surrounding atmosphere. The medium shot is suitable for describing a person's motion or behavior.

Close-up shots mainly emphasize a person's face, and because it can better reveal the details and expressions of the face, it can focus the viewers more. Therefore, the emotion that the photographer wants to express can be clearly expressed. Close-up shots can effectively elicit the listener's empathy.

In addition, this embodiment further includes a position evaluation unit 134 that evaluates whether the position of the face recognized by the face recognition unit 110 corresponds to the position of the third division law. The operation unit 160 further uses the evaluation result of the position evaluation unit 134 during the comprehensive evaluation operation.

Portrait photography tends to show a difference in emotional preference depending on the position of the face. Therefore, professional photographers constantly consider and test the position of the face in the image frame when taking a portrait. The position of the face plays a role in accurately conveying the photographer's intention and subject, and when taking a portrait, it binds the lines and shapes that make up the photo to express the size and perspective, contrast, texture, motion, time, direction, and emotion of the subject. do. Professional photographers mainly use the Rule of Third, which can give the viewer the most stability.

The rule of third division is the ratio that ideally divides a given length. This law originally originated in art, and is widely used in various fields such as photography, video, and design. It can also be applied to all 2D planes and 3D spaces. As shown in FIG. 3, when the top, bottom, left, and right are divided into virtual lines in an image frame, and the subject is positioned at four points where the lines meet, the rule of third division is satisfied. And when the subject is located in the dot area, people feel visually stable and comfortable composition (Kim Jong-moo 2009).

The composition evaluation unit 135 evaluates whether the composition of the face recognized by the face recognition unit 110 corresponds to a preference composition.

Kim Seol-hee (2016) conducted a study on the angle of face that people prefer when viewing portraits. He found that the angle of inclination of the face between about 30 and 45 degrees was the most preferred. It can be seen that classical paintings or photographs also have a high proportion of works with such a tilt angle of the face. For example, in the Historical Portrait Portrait Exhibition, a collection of Korean portraits from about the 1st century to the 19th century, there are overwhelmingly more portraits of faces facing the side at around 30 degrees than portraits facing the front. There are 240 pictures facing the side and 24 pictures facing the front. In addition, REMBRANDT-The painter at work, a collection of Western portraits, out of 183 portraits, has 131 paintings facing the side and 33 paintings facing the front.

Even though the subject is the same person, the sensibility felt by people who appreciate portrait photography varies depending on the angle and composition of the face. Recently, with the spread and development of Internet technology, users' self-portraits are in vogue along with a new digital culture called social media. For this reason, recently, the method of how to take a picture of yourself has become an issue, and a face picture rotated by about 45 degrees has become known as the most attractive and preferred composition of the face. In this way, it can be seen that the composition of the face is an evaluation item that can affect the preference of a portrait to the viewer.

In addition, this embodiment further includes a focus evaluation unit 137 that evaluates the degree of sharpness of the face by recognizing the depth of field of the face recognized by the face recognition unit 110. The operation unit 160 further uses the evaluation result of the focus evaluation unit 137 during the comprehensive evaluation operation.

When focusing on a subject when taking a picture, you can adjust the area and area in focus, which is called the depth of field. In general, it is expressed as a shallow depth or a deep depth, and the range is determined according to changes in optical properties. FIG. 4 shows that when focusing on a subject using a camera, light is not accurately focused on a single area point on the sensor surface, but is focused on a front or rear area to form a depth area. Light from a subject outside the depth of field is condensed by the sensor in a circle shape, which is called a confusion circle. The smaller the radius of the confusion circle, the higher the image clarity, but the out-of-focus area appears blurry. When this is large enough to be distinguished by the human eye, it is called a permissible source of confusion and becomes an important factor in determining the depth of field. Such depth-of-field adjustment can show only the subject that the photographer wants to express, and can enhance the preference of viewers by realizing out-of-focus in portrait photography.

Experts mainly use the out-of-focus technique to focus more on the person by separating the person from the background when taking a portrait photo. In order to use out-of-focus, the depth of field should be made shallow, but it is usually possible to shoot with a low aperture value or a telephoto lens. In recent years, with the development of advanced technology, it can be implemented in mobile phone cameras, so anyone can easily try out the focus effect.

Infocus is a technique used to describe a special relationship between a subject and a background or to express a relationship with another subject. Infocus is a method in which the main subject is photographed so that the main subject is blurred by focusing on the background, and it is a concept opposite to defocus in that the subject in focus has changed.

A photo with a deep depth of field corresponds to the pan focus, and the focus on the central subject is less than that of a photo using out-of-focus and in-focus. However, since everything is clearly revealed, the sense of reality is raised. For pan focus, a single focus lens such as a wide-angle lens is used, and is mainly used in news reports and videos that must maximize realism and accuracy.

The emotional color evaluation unit 136 of this embodiment recognizes an expression on the face recognized by the face recognition unit 110, and evaluates the degree of matching between the expression and a color tone of a portrait photograph.

Color tone is composed of three elements that make up color: hue, brightness, and saturation. When the viewer judges the image, it can be found in several preliminary studies that color tone affects the overall emotional content (Kaya, 2004; Suwoong Seo, 2013). Eun-Bae Moon (2011) experimented with emotional response according to color tone change using emotional adjective vocabulary, and the overall emotional evaluation value decreased when colors were excluded. In addition, it was found that the color of the sculpture has a great influence on the sensibility of the viewer. As described above, since color tone is an influential viewer-based image evaluation item that can influence the emotional content of an image, it is necessary to recognize which emotional content a specific color causes. Noh Yeon-sook (2015) and Lee Yeon-ran (2015) collected data on which emotional content the image induces according to the color properties through a survey. Colors can be classified into three broad categories: warm, cold, and neutral colors according to the color properties (Moon-Ju Kim, 2006). In addition, Roh Yeon-suk (2010) established item factors that affect image quality in order to establish a subjective quality evaluation model for images. He determined the item elements that influence the image quality as the image detailing ability, tone reproduction ability, and color reproduction ability, and made an assumption that these item elements influence each other. After that, a subjective image quality evaluation model was constructed using factor analysis and structural equation model. As a result of analyzing the correlation between these three factors, the correlation coefficient between detail and tone reproduction power was 0.468, the correlation coefficient between detail and color reproduction power was 0.572, and the correlation coefficient between tone reproduction power and color reproduction power was 0.421. It turned out to be related In other words, it was found that the power of detail description, tone reproduction, and color reproduction influence the subjective preference for image quality.

Human emotions are so diverse that it is difficult to define and categorize them. Ekman (2002) defines facial expressions for six common emotions as happiness, sadness, anger, fear, surprise, and disgust, and uses them to analyze the facial muscles involved in facial expressions, using the FACS (Facial Action Coding System). Developed. In addition, facial components were classified into AU (Action Unit) by dividing the facial components into the forehead, eyebrows, eyes, nose, mouth, and chin-connected facial muscles, and the intensity of facial expressions was divided into 5 levels (Ekman, 2005). . However, there seems to be a difference in the intensity of expression of AU used for facial expressions even with the same emotion due to emotional differences according to cultures (Matsumoto, Kasri, & Kooken, 1999). In order to compensate for this difference, the Japanese and Caucasian Facial Expression of Emotion (JACFEE) was created with Matsumoto, and it is still used as basic research data in many fields (Matsumoto, 1999).

Roh Yeon-suk (2012) examined the emotional reactions of viewers by experimenting with changes in shooting distance, changes in facial expressions, colors, exposure, contrast, sharpness, shutter speed, saturation, and depth of field. In particular, the fact that sensibilities such as joy and depression act as the most important factors in the overall perception of photographs was a meaningful result showing clearly the emotional specificity of photographs.

Woo-Hyun Jung (2006) conducted an experiment to check whether the change in overall color tone affects the emotional response. In addition, as a follow-up study, a study was published that the change of each hue affects a specific sensibility, and the green hue and the blue hue gave a cold and dark feeling, respectively, and were found to affect negative sensibility.

This embodiment further includes a focus evaluation unit 122 for evaluating the focus of a face recognized by the face recognition unit 110 in a portrait photograph.

Since the focus of the face is the most important factor in portrait photography, it is the item that professional photographers consider the most important. Before evaluating other items, the focus on the face must be checked. The focus evaluation unit 122 used a focus measurement operator to calculate a focus level of an area detected by a face recognition algorithm. The purpose of the focus measurement operator is to evaluate the sharpness or degree of focus of an image pixel. Laplacian-based operators were used to measure the degree of focus of an image pixel in a specific area (see Equation 3). The Laplacian-based operator takes advantage of the fact that an image area in focus has a sharper edge than an image area that is not in focus. The Laplacian-based operator comprehensively evaluates the contrast, saturation, noise and adjacent pixels of an image.

[Equation 3]

In this example, a total of 250 sets of portrait photos were analyzed in order to set a reference numerical value for confirming whether or not focus is present. One set consists of a total of two portrait photos, and one set of portrait photos has the same model and background, and is composed of a photo with a focus on the face and a photo with an out of focus.

The minimum standard numerical value of the face focus was specified using the reference-oriented standard setting method. The reference-oriented standard setting method is mainly used when trying to give meaning by classifying different groups according to specific criteria through a certain test (Yeonseong Jeong, 2006). 5 is a distribution diagram of a comparison group model for whether or not focus is on. Using such a distribution map, an expected reference point can be derived. To verify the validity of the expected reference point, the classification accuracy and validity coefficients are calculated and verified.

Classification TM (True Master) Wansu Jin Outstanding group with more than the standard score FM(False Master) Wansu Oh Non-excellent group with a baseline score or higher True Nonmaster (TN) Jinmisu Non-excellent group with less than or equal to the baseline score False Nonmaster (FN) Ohmisu Outstanding group with less than or equal to the standard score P(TM) = TM/(M+N), P(FM) = FM/(M+N)
P(TN) = TN/(M+N), P(FN) = FN/(M+N)

는

과

의 합으로 나타낼 수 있다. 타당도 계수는

수식을 통해 산출하여 나타낼 수 있다. 또한 BR(base rate)은

으로 나타낼 수 있으며, 모집단 중, 진완수자와 오미수자의 비율을 의미한다. SR(selection rate)은

으로 나타낼 수 있으며, 모집단 중 진완수자와 오완수자의 비율을 의미한다.Classification accuracy

Is

and

It can be expressed as the sum of The validity factor is

It can be calculated and expressed through an equation. Also, the base rate (BR) is

It can be expressed as, and it means the ratio of true Wansui and Missiles among the population. SR (selection rate) is

It can be expressed as, and it refers to the ratio of true and wrong complete players among the population.

[Equation 4]

Reference value Classification accuracy Validity factor 0.03 0.85 0.63

는 형상모수(Shape Parameter)이고,

는 척도모수(Scale Parameter)이며, 이 실시예는 평균과 표준편차로 정의하였다.On the other hand, the CDF used by the contrast evaluating unit 123 is obtained by integrating a Probabiltity Density Function once. Referring to FIG. 6, a probability of a specific value can be expressed in a function capable of representing only a range through a probability distribution using CDF. In the formula of CDF

Is the shape parameter,

Is a scale parameter, and this example is defined as the mean and standard deviation.

[Equation 5]

The exposure evaluation unit 132 of this embodiment used an automatic exposure setting algorithm applied to a mobile phone camera and features of Ansel Adams' zone system in order to accurately evaluate the exposure of the face. The automatic exposure setting algorithm determines exposure using the luminance component of the image, and changes the RGB color space to the YCbCr coordinate system to obtain a luminance value, Y. Also, using the principle of the zone system, the luminance value Y was stepped and the face exposure score was calculated.

[Equation 6]

After extracting the face region using a face recognition algorithm, the weight evaluation unit 133 of this embodiment compares the number of pixels of the entire image with the number of pixels of the face region. The comparison process was used to accurately calculate the proportion of faces in the image.

Hyungju Park (2016) measured the preferences of viewers by dividing 1000 portrait photos into 10% units according to the proportion of their faces. As a result, when the weight of the face was 11-20%, the preference was the highest, and as the weight gradually increased, the preference could be seen a tendency to decrease rapidly. The results of such preliminary research and opinions of photography experts were referred to and applied to this example. The score evaluation range was divided into 11~30%, 31~50%, 51~70%, and 71~90%, and extreme proportions of 0~10% and 91~100% were judged to have a negative effect on emotional preference. Therefore, it was set to give the lowest score.

The position evaluation unit 134 of this embodiment extracts a face area using a face recognition algorithm, and then determines where the face is located in the image. After that, the rule of third division was used to reflect the emotional preference according to the position of the face. The image was divided into three sections in the horizontal and vertical directions, and four points where the lines divided in the horizontal and vertical directions were orthogonal were derived. According to a previous study (Hyungju Park, 2016), it was confirmed that the person's face feels the most stable when the face is placed on four points, so it can be used to evaluate the professional screen composition.

The composition of the face is one of the items that greatly affects the attractiveness of a subject when a person appreciates a portrait. Referring to FIG. 7, the composition evaluation unit 135 of this embodiment sets a plurality of feature points on the face using the CLM-Framework algorithm, and uses the feature points to rotate the face, tilt and tilt the face. Calculate the value of the back and forth tilt (Up/Down). The CLM-Framework algorithm was designed using a machine learning technology called Constrained Local Neural Fields (CLNF). First, for face detection, 68 feature points are detected on a face with a support vector machine (SVM). Using the extracted feature points, it is possible to recognize the jaw line, lips, nose, and eyebrows. A combination of feature points calculates numerical values in 3D space for Turn, Up/Down, and Tilt. In addition, a reliability value is calculated based on the combination of each feature point to reflect the suitability of the extracted face. Accordingly, the extracted face composition can obtain a numerical value in 3D space.

Through the studies on the change in the emotions of the viewers according to the color tone, it was possible to know the color tones that are highly related to the emotions such as'warm and bright' and'cold and dark'. Using this, the emotional color evaluation unit 136 of this embodiment gives an additional point when the emotions appearing on the face of the portrait and the emotional response evoked from the color tone match.

Referring to FIG. 8, the emotional color evaluation unit 136 of this embodiment converts RGB information into an HSV color model consisting of hue, brightness, and saturation in order to determine the overall color tone of a photo, and converts the color tone according to the color attribute. It is implemented so that it can be automatically measured and scored according to emotional content (see Equation 7).

[Equation 7]

Human facial expressions show quick and subtle changes according to emotions. In this embodiment, the emotion color evaluation unit 136 grasps and classifies characteristic points of facial expressions representing each emotion in order to classify emotions. Referring to FIG. 9, feature points are named AU (Action Unit), and facial expressions can be distinguished through a combination of them (Pertuz, Puig, &Garcia, 2013).

The emotional color evaluation unit 136 of this embodiment classified the emotions into positive, neutral, and negative classifications. Positive was indicated by the combination of AU6 and AU12. Neutral was indicated by the combination of AU1, AU2, AU4, AU5, and AU 25. Negation is expressed as the sum of three emotions and consists of sadness, fear, and anger. Sadness was expressed as a combination of AU1, AU4, and AU15. Fear was expressed as a combination of AU1, AU2, AU4, AU6, AU7, AU20, and AU26. Anger was expressed as a combination of AU4, AU5, AU7, and AU23.

In this example, sadness, fear, and anger were classified into one group of negative emotions, and Pearson's bivariate correlation analysis was performed to find out what kind of correlation exists between the variables. To proceed with this analysis, three emotions were classified through AM-FED and JAFFE DATABASE, image data sets extracted from emotions under wild environmental conditions. Referring to Table 5, as a result of analyzing the correlation for each sub-factor of emotion, it was confirmed that the correlation between all variables was very significant.

<N=450> variable Average Standard Deviation correlation sadness fear anger sadness 3.0903 1.86136 One fear 8.81 3.38738

One anger 4.2672 2.00538

One

If the facial expression is positive and the color tone is a warm tone, the emotional color evaluation unit 136 determines that the expression and the color tone of the portrait photo match and gives an additional point. In addition, if the expression was negative and the color tone was Cool tone, it was judged that the color tone of the expression and the portrait photograph matched and an additional point was given. Referring to FIG. 10, warm-tone (warm color) is a color corresponding to warm color, and cool-tone (cool color) is a color corresponding to cold color.

In addition, the emotional color evaluation unit 136 evaluates items such as detailed description ability, tone reproduction ability, and color reproduction ability of a portrait photo, and assigns weights when the evaluation result of each item is higher than a preset criterion.

Defocus is a photography technique mainly used by portrait photography professionals to separate the person from the background and focus on the person. When this technology was used, it was found that the preference of the listeners was high (Lee Chang-seop, 2018). Referring to FIG. 11, the focus evaluation unit 137 of this embodiment applies the focus measurement operator used to determine whether the face is in focus in order to determine whether the subject is out of focus. By using this operator, it is possible to determine whether or not to be out of focus by comparing the level of focus between an area occupied by a person and the rest of the area.

The operation unit 160 according to an embodiment of the present invention sets the evaluation elements as priority by synthesizing the preliminary research results and a basic process for photographing photographs by photographers. The operation unit 160 mainly sets the elements measured based on the face as a high priority. The reason is that the preferences of the viewers can be clearly divided according to the face, the most important factor in portrait photography.

The degree of facial exposure evaluated by the exposure evaluation unit 132 was selected as the highest preference. Face exposure is one of the most important visual attributes for viewers who appreciate portrait photography. Even for a portrait photo that has adequate exposure to the rest of the face except for the face, it is difficult to expect a good preference from the viewers if the face is under or overexposed.

Next, the position of the face evaluated by the position evaluation unit 134 was selected as the second priority. The position of the face is also one of the important visual properties for viewers. In most art fields that have frames, work is done using the rule of three divisions, and if the main subject is located in a position outside of this, it is difficult to expect good preference from the viewers.

Next, the contrast evaluated by the contrast evaluation unit 123 using RSC was selected as the third priority. The contrast evaluation unit 123 measured the entire portrait image using RSC in this embodiment to calculate an objective value. RSC measures the color and brightness between objects using real human vision, and because the preference can be sharply divided according to the strength and weakness of the RSC, the RSC of only the face is not measured, but it is set as the third priority.

Next, the weight of the face size evaluated by the weight evaluation unit 133 was set as the fourth priority. In the preceding study, the preference for photos with 11-20% of the total weight of the person and background was the highest, and when it exceeded 50%, the preference tended to decrease.

Next, the dynamic range evaluated by the dynamic region evaluation unit 124 was set as the fifth priority. The dynamic region evaluation unit 124 measured the entire image with a dynamic range in this embodiment to calculate an objective value. Also, considering that the standard deviation value was lower than other factors through prior studies, it was found that most of the portrait photos were distributed in a specific dynamic range numerical range (Hyungju Park, 2013; Changsub Lee, 2018). And recently, with the development of imaging technology, the dynamic range expression capability of images taken with a mobile phone has been improved, so that a very wide range of brightness information can be expressed. Accordingly, the lowest priority among the five factors is given.

In addition, the evaluation results of the composition evaluation unit 135, the emotional color evaluation unit 136, and the focus evaluation unit 137 were reflected during the comprehensive evaluation calculation.

를 수학식 8과 같이 계산할 수 있다(송방원, 2017).The operation unit 160 applies weights for each evaluation element to the set priority elements using a ranking order centroid method. The ranking-centered method is a method of calculating a reasonable weight using a set ranking, and the weight of each attribute for n attributes.

Can be calculated as in Equation 8 (Song Bang-won, 2017).

[Equation 8]

는

번째 순위의 가중치,

은 속성의 개수이다.At this time,

Is

Weight of the first rank,

Is the number of attributes.

The calculation unit 160 of this embodiment assigns weights to the elements to be evaluated as shown in Table 6 according to their priorities. Other evaluation items were not weighted.

property RSC Dynamic
range Face exposure Face position Face weight Priority 3 5 One 2 4

)weight(

) 0.16 0.03 0.46 0.27 0.09

The apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective according to an embodiment of the present invention is intended to present a model of a new portrait preference measurement program for the development of advanced image reproduction products by automating the entire process of the portrait evaluation. I set a goal. In this example, subjective evaluation items and objective evaluation items affecting preference were developed based on the results of emotional response data and opinions of photographic experts.

As a first step, the face recognition unit 110 first determines whether the focus is on the most important face in the portrait photograph. Face recognition algorithm was used to determine the focus level of the face. It is checked whether or not the detected face area is in focus through a focus measurement operator.

As an objective factor, the RSC for the contrast evaluation and the dynamic range of the dynamic region evaluation unit 124 were selected. In order to statistically evaluate the two objective factors, 500 portrait photos with more than 100 recommendations were selected from Flickr (www.flickr.com), an image upload social media. After that, the two items were expressed as objective values, and the image to be evaluated could be scored using the CDF function.

The subjective elements were set as face exposure, position, weight, composition, matching of expression and color tone, and out of focus.

The exposure item of the face of the exposure evaluation unit 132 used the principle of the digital zone system and the automatic exposure setting algorithm applied to the mobile phone camera. Using this, you can measure and specify the portrait photo you want to evaluate by exposure step.

The position item of the face of the position evaluation unit 134 used the rule of third division. The evaluation items were composed of a total of five areas, including the four contact areas and the middle area of the image after dividing the top, bottom, left, and right in the image frame into three parts.

The weight item of the face of the weight evaluation unit 133 could accurately represent the face area through a face recognition algorithm. In addition, the item can be calculated using the ratio of the total number of pixels in the person's photograph to be evaluated and the number of pixels in the face area. The above items have a difference in preference depending on the method of taking a portrait photo. By synthesizing these portrait photography methods and research results, the score class was divided into 4 stages.

As for the composition item of the face of the composition evaluation unit 135, values of Up/Down, Tilt, and Turn of the face were extracted using the CLM-Framework. By synthesizing the results of prior research and the opinions of photography experts, the numerical range of the composition of the face when the preference is highest was set. Additional points were given when the portrait photo to be evaluated satisfies the range conditions set in this study.

The out-of-focus item of the focus evaluation unit 137 is to determine the presence or absence of out-of-focus by comparing the focus level of the person's area and the focus level of the other areas excluding the person area in the portrait photo. Was given.

As for the color tone and emotion items of the emotional color evaluation unit 136, a color tone that is highly related to the emotion that a person feels when viewing an image is selected, and when the emotional response matches, an additional point is given.

As such, the program was designed by systematically gathering the items that people consider when evaluating portraits, such as subjective evaluation items and objective evaluation items.

Referring to FIG. 12, the apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective according to an embodiment of the present invention loads a photographed portrait photograph, performs evaluation of applied items, and calculates the result. It includes the ability to store data that can be analyzed. Therefore, the display unit 180 of this embodiment presents a portrait picture and an evaluation item to be analyzed on one screen so that the user can view the evaluation result at a glance, and an intuitive and simple GUI is designed. Based on the original portrait picture, the evaluated and analyzed items were visualized and displayed, and the numerical values and analysis results for each item were constructed accordingly.

In this embodiment, the display unit 180 includes an original portrait photo 181 at the upper left, a face composition 182 analyzed at the right side of the original portrait photo, and the analyzed face position 183 at the bottom of the original portrait photo and the face composition. , The results of the analyzed face focus 184 and the analyzed facial expression 185 are displayed.

In addition, a color histogram related to the color tone of a portrait photograph, an RGB color space 186, and a graph 187 of analyzed RSC values are displayed on the right side of the composition of the face.

In addition, the evaluation results for each evaluation item are listed on the right side (188), and the overall evaluation value (189) is displayed on the bottom.

Next, a method of evaluating the emotional preference of a portrait photograph using an expert perspective according to an embodiment of the present invention will be described.

Referring to FIG. 13, in this embodiment, the face recognition unit 110 recognizes a face in a portrait photograph (S120), and the evaluation unit evaluates whether the composition of the face recognized by the face recognition unit 110 corresponds to a preference composition. And, the step of evaluating the matching degree of the facial expression and the color tone of the portrait photograph (S140), the operation unit 160 calculating a comprehensive evaluation using the evaluation result of the evaluation unit (S160), and the display unit ( A step (S180) of outputting the comprehensive evaluation calculated by the operation unit 160 is performed by 180).

When evaluating the composition of the face, the evaluation unit sets a plurality of feature points on the face using the CLM-Framework algorithm, and calculates values of rotation, left and right tilt, and front and rear tilt of the face using the feature points to quantify the composition of the face. do.

In addition, the evaluation unit classifies the expression as positive, neutral, and negative when evaluating the matching of the expression and color tone.

If the expression is positive and the color tone is warm tone, the evaluation unit determines that the expression and the color tone of the portrait photo match and assigns a weight. In addition, if the expression is negative and the color tone is a cool tone, it is determined that the color tone of the expression and the portrait photo match and weight is given.

The evaluation unit also evaluates the contrast by analyzing the portrait photo by RSC (Retinal-like Subsampling Contrast).

The evaluation unit evaluates the dynamic range of the portrait by using a photoelectric conversion function.

The evaluation unit evaluates the degree of exposure of the face recognized by the face recognition unit 110. The evaluation unit evaluates the degree of exposure by changing a portrait photo into a YCbCr coordinate system to obtain a Y value, and stepping the Y value using the principle of the zone system.

The evaluation unit evaluates the proportion of the size of the face recognized by the face recognition unit 110 in the portrait picture. The evaluation department classifies a portrait photo into one of a long shot, a full shot, a medium shot, and a close-up shot, and then evaluates the proportion of the size according to the shot type of the portrait photo. .

The evaluation unit evaluates whether the position of the face recognized by the face recognition unit 110 corresponds to the position of the third division law.

The evaluation unit evaluates the degree of sharpness of the face by recognizing the depth of field of the face recognized by the face recognition unit 110.

Experiment.

As an experiment, portrait photographs preferred by ordinary viewers were evaluated using this example.

This example complies with the content of objective evaluation items of images defined by ISO, and integrates prior research on subjective evaluation items of images and opinions of photographic experts, so that a total of nine items can be evaluated.

In the experiment, a photo group with high preference and a photo group with relatively low preference were set to check whether there was a difference in score measurement between the two groups.

For the experiment, 10 portrait photos with more than 100 recommendations and 10 portrait photos with less than 10 recommendations were selected from Flickr, a social media that uploads images for the experiment, and divided them into groups of photos with high general preference and the opposite photo group. .

In addition, statistical analysis was conducted to confirm the validity of the evaluation item priority set during the development process.

In this example, when the number of independent variables is 3 or more, multi-way ANOVA is applied to test whether there is a difference in the mean of the dependent variable between groups according to the independent variable.

Independent variable Dependent variable Face exposure Face position RSC Face weight Dynamic range Composite score

Referring to FIGS. 14 to 29, the portrait photos selected for the experiment exceeded 0.03, which is the focal value of the face specified by the reference-oriented criteria setting method, and thus all 20 portrait photos were found to be in focus. When evaluating portrait photography, the subjective evaluation items and objective evaluation items needed were analyzed.

RSC and dynamic range, which are objective evaluation items, were calculated using CDF, a statistical numerical analysis method. The subjective evaluation item, face exposure, was designed to be evaluated in three stages.

The appropriate exposure range was set as A. The range of A was designed to be within the range of the appropriate exposure or 3 stops exceeding the prior research results and opinions of photography experts. In addition, the exposure level range that does not fall within this range was stepped into the lower exposure steps B and C. As a result of measuring through this step, the range A was found 16 times and the range B was 4 times (see FIGS. 20 and 21). The photos measured with the B grade were those of the group with low preference, and it was confirmed that the photos corresponded to the appropriate exposure range even when viewed with the naked eye, so that the accuracy was high.

The position of the face was designed to be evaluated by classifying that it is located in areas A, B, C, and D according to the rule of three divisions (see FIGS. 22 and 23). As for the measurement results, A, B, C were found 16 times, D was 4 times, and two groups were given D grade. In addition, it was found that photographs located in areas A, B, and C, which are the preferred positions of professional photographers and photo viewers, occupy a large proportion.

The specific gravity of the face could be calculated through the ratio of the total number of pixels in the image of the portrait and the number of pixels in the face area (see FIGS. 21 and 22). As a result of the measurement, the proportion of A was 13 times, the proportion of B was 2 times, and the proportion of C was 5 times, mainly 10 to 40% of faces accounted for the majority. As for the specific gravity of the face, the photos of the group with low preference showed a tendency to receive a low rating, and it was found that the pictures of the group with low preference had the same weight even when actually checked with the naked eye, so the accuracy of the evaluation items was high.

As for the out-of-focus, 14 portrait photos were determined as the photos using the out-of-focus technique, and among the photos of the group with low preference, 6 photos were analyzed as those without the out-of-focus technique (see FIGS. 24 and 25). . On the contrary, all photos of the group with high preference were judged to be photos using the out-of-focus technique, which professional photographers like to use, and the same results were shown when actually checked with the naked eye. Through this, it can be seen that this embodiment accurately determines whether or not to be out of focus by distinguishing a person as a subject and a background as another area.

When viewing a portrait, the composition of the face corresponds to the value of the angle range preferred by people in 8 cases (see FIGS. 26 and 27). It was found that 5 sheets in the group with high preference and 3 sheets in the low group corresponded to the preferred angle range, and it was confirmed that the composition of the face taken by the subject showed the same result even when judged with the naked eye.

As for the color tone, 13 warm-tones and 7 cool-tones appeared, and 17 neutral, 2 positive, and 1 negative emotions (see FIGS. 28 and 29). There were 3 cases with color tones that correlate with emotions in portrait photography, and it was confirmed that the same result was shown even when judged with the naked eye. The total score could be expressed by converting these results into a total score of 100 points through the ranking centering method, a method of applying weights for each evaluation item.

Referring to FIG. 14, portrait photos with high preference were shown as 94.50, 94.36, 93.92, 91.11, 90.79, 90.71, 89.59, 84.94, 84.82, and 83.82 in order from the top. Referring to FIG. 15, the groups with low preference were found to be 71.70, 77.66, 72.03, 66.54, 62.55, 61.55, 60.85, 59.13, 54.63, 54.51 in order from the top.

The value of the comprehensive evaluation evaluated using this example represents the emotional preference value of the viewers in which the objective item and the subjective item are combined.

In addition, statistical verification was conducted using 500 portrait photos of 100 or more recommended times on social media Flickr.

sauce Type III sum of squares Degrees of freedom Mean square F Significance probability Modified model 558922512.315a 34 16438897.421 114.095 .000 Intercept 164913588.793 One 164913588.793 1144.591 .000 Face exposure * Face position * RSC * Face specific gravity * Dynamic range 262682421.794 25 10507296.872 72.926 .000 Face exposure 23938250.543 One 11938250.543 651.000 .000 Face position 230111554.499 2 115055777.250 598.550 .000 RSC 33443087.510 2 16721543.755 116.057 .000 Specific gravity of the face 26701776.306 2 13350888.153 92.662 .000 Dynamic range 208099.035 2 104049.518 .722 .006 error 66997601.564 465 144080.864 all 652430454.253 500 Modified sum 625920113.879 499

Table 8 shows the results of multi-factor variance analysis.

First, through the exposure of the face * the position of the face * the RSC * the specific gravity of the face * the dynamic range, it was found that the evaluation item element had an interactive effect on the final score.

The F value is 72.926 and the significance probability is .000, so it is within the significance level.

In addition, in order to confirm the validity of the evaluation item priority set in this example, the F value and the significance probability of each item were referenced. Face exposure, face position, RSC, face weight, and dynamic range F value and significance probability are all within the significance level, and the lower the significance probability, the higher the F value, the greater the difference between groups. It can be seen that the position of the face, RSC, the specific gravity of the face, and the dynamic range match the priority set in this example.

Although the preferred embodiment of the present invention has been described above, the present invention can use various changes, modifications, and equivalents. It is clear that the present invention can be applied in the same manner by appropriately modifying the above embodiments. Therefore, the above description does not limit the scope of the present invention determined by the limits of the following claims.

100: portrait photo sensitivity preference evaluation device
110: face recognition unit 120: objective evaluation unit
122: focus evaluation unit 123: contrast evaluation unit
124: dynamic domain evaluation unit 130: supervised evaluation unit
132: exposure evaluation unit 133: weight evaluation unit
134: location evaluation unit 135: composition evaluation unit
136: emotional color evaluation unit 137: focus evaluation unit
160: operation unit 180: display unit

Claims (26)

A face recognition unit that recognizes a face in a portrait photo;
A composition evaluation unit that evaluates whether the composition of the face recognized by the face recognition unit corresponds to a preference composition;
An emotional color evaluation unit for recognizing an expression on the face recognized by the face recognition unit and evaluating a degree of matching between the expression and a color tone of the portrait photograph;
An operation unit that calculates a comprehensive evaluation using the evaluation results of the composition evaluation unit and the emotional color evaluation unit; And
And a display unit that outputs the comprehensive evaluation calculated by the calculation unit. The apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective. The method of claim 1, wherein the composition evaluation unit,
Using the CLM-Framework algorithm, a plurality of feature points are set on the face, and values of rotation, left and right tilt, and front and rear tilt of the face are calculated using the feature points. A device to evaluate emotional preferences. The method of claim 1, wherein the emotional color evaluation unit,
An apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective, characterized in that the expression is classified into positive, neutral, and negative. The method of claim 3, wherein the emotional color evaluation unit,
If the expression is positive and the color tone is a warm tone, it is determined that the expression and the color tone of the portrait picture match. Device. The method of claim 3, wherein the emotional color evaluation unit,
If the expression is negative and the color tone is a cool tone, it is determined that the expression and the color tone of the portrait photo match. Device. The method of claim 1,
Further comprising a contrast evaluation unit for evaluating the contrast by analyzing the portrait photo RSC (Retinal-like Subsampling Contrast),
The apparatus for evaluating the emotional preference of a portrait photograph using the expert viewpoint, wherein the calculation unit further uses an evaluation result of the contrast evaluation unit during a comprehensive evaluation operation. The method of claim 1,
Further comprising a dynamic region evaluation unit for evaluating a dynamic range (dynamic range) using the photoelectric conversion function of the portrait,
The calculating unit further uses the evaluation result of the dynamic region evaluation unit when calculating the comprehensive evaluation. The apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective. The method of claim 1,
Further comprising an exposure evaluation unit for evaluating the exposure degree of the face recognized by the face recognition unit,
The calculating unit is an apparatus for evaluating the emotional preference of a portrait photograph using an expert viewpoint, characterized in that the evaluation result of the exposure evaluation unit is further used during a comprehensive evaluation operation. The method of claim 8, wherein the exposure evaluation unit,
A person using a professional perspective, characterized in that the exposure degree is evaluated by changing the portrait photo by changing the RGB color space to the YCbCr coordinate system to obtain a Y value, and stepping the Y value using the principle of a zone system A device that evaluates the emotional preference of photos. The method of claim 1,
Further comprising a weight evaluation unit for evaluating the weight of the size of the face recognized by the face recognition unit in the portrait photo,
The calculating unit is an apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective, characterized in that the evaluation result of the weight evaluation unit is further used when calculating a comprehensive evaluation. The method of claim 10, wherein the specific gravity evaluation unit,
After classifying the portrait photo into one of a long shot, a full shot, a medium shot, and a close-up shot, the proportion of the size is evaluated according to the shot type of the portrait photo. A device for evaluating the emotional preference of a portrait photograph using a professional perspective. The method of claim 1,
Further comprising a position evaluation unit for evaluating whether the position of the face recognized by the face recognition unit corresponds to the position of the third division law,
The calculating unit is an apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective, characterized in that the calculation unit further uses the evaluation result of the position evaluation unit during a comprehensive evaluation operation. The method of claim 1,
Further comprising a focus evaluation unit for evaluating the degree of sharpness of the face by recognizing the depth of field of the face recognized by the face recognition unit,
The operation unit further uses the evaluation result of the focus evaluation unit during a comprehensive evaluation operation. The apparatus for evaluating the emotional preference of a portrait photograph using an expert perspective. Recognizing a face in a portrait photo by a face recognition unit;
Evaluating whether a composition of the face recognized by the face recognition unit corresponds to a preference composition, and evaluating a degree of matching between the facial expression and the color tone of the portrait photograph;
Calculating a comprehensive evaluation using the evaluation result of the evaluation unit; And
A method for evaluating the emotional preference of a portrait photograph using a professional perspective, comprising the step of: the display unit outputting the comprehensive evaluation calculated by the calculation unit. The method of claim 14, wherein the evaluation unit,
Professional characterized in that a plurality of feature points are set on the face using a CLM-Framework algorithm, and values of rotation, left and right tilt, and front and rear tilt of the face are calculated using the feature points to quantify the composition of the face. A method of evaluating the emotional preference of a portrait photo using a point of view. The method of claim 14, wherein the evaluation unit,
A method for evaluating the emotional preference of a portrait photo using a professional perspective, characterized in that the expression is classified into positive, neutral, and negative. The method of claim 16, wherein the evaluation unit,
If the expression is positive and the color tone is a warm tone, it is determined that the expression and the color tone of the portrait picture match. How to. The method of claim 16, wherein the evaluation unit,
If the expression is negative and the color tone is a cool tone, it is determined that the expression and the color tone of the portrait photo match. How to. The method of claim 14,
The evaluation unit evaluates the emotional preference of a portrait picture using a professional perspective, characterized in that the evaluation unit analyzes the portrait picture by RSC (Retinal-like Subsampling Contrast) to evaluate the contrast. The method of claim 14,
The evaluation unit evaluates a dynamic range of the portrait photograph using a photoelectric conversion function. A method of evaluating the emotional preference of a portrait photograph using an expert perspective. The method of claim 14,
The evaluation unit evaluates the degree of exposure of the face recognized by the face recognition unit. A method of evaluating the emotional preference of a portrait photograph using an expert perspective. The method of claim 21,
The evaluation unit obtains the Y value by changing the RGB color space of the portrait photo to the YCbCr coordinate system, and evaluates the degree of exposure by stepping the Y value using the principle of a zone system. How to evaluate the emotional preference of portrait photos The method of claim 14,
The evaluation unit is a method of evaluating the emotional preference of a portrait picture using a professional perspective, characterized in that the evaluation unit evaluates the proportion of the size of the face recognized by the face recognition unit in the portrait picture. The method of claim 23,
The evaluation unit classifies the portrait picture into one of a long shot, a full shot, a medium shot, and a close-up shot, and then the size corresponding to the shot type of the portrait picture. A method of evaluating the emotional preference of a portrait photo using an expert perspective, characterized by evaluating the specific gravity. The method of claim 14,
The evaluation unit evaluates whether the position of the face recognized by the face recognition unit corresponds to the position of the rule of third division. The method of claim 14,
The evaluation unit evaluates the sensitivity preference of a portrait photograph using an expert perspective, characterized in that the evaluation unit recognizes the depth of field of the face recognized by the face recognition unit and evaluates the degree of sharpness of the face.

Images