KR20170015089A - User terminal apparatus and control method thereof - Google Patents

Abstract

Provided are a user terminal device and a control method thereof. The user terminal device comprises: a display unit which includes a main display area arranged in the front of the user terminal device and a sub display area arranged in at least a part of the rear of the user terminal device by being extended from one side of the main display area; a camera which photographs an image; and a processor which controls the display unit to display a live view obtained through the camera on one of the main display area and the sub display area and to display the live view on the other when the orientation of the user terminal device is changed.

Description

[0001] USER TERMINAL APPARATUS AND CONTROL METHOD THEREOF [0002]

The present invention relates to a user terminal and a control method thereof, and more particularly, to a user terminal using a display arranged on the front and rear sides of a user terminal at the time of shooting and a control method thereof.

Various types of electronic devices have been used in various fields due to the development of electronic technology. Particularly, in recent years, an electronic device (Half Round Display) in which a display is extended to the back side of an electronic device through Out Bending is being developed.

In general, the camera is provided on the back side of the electronic device, and the display is provided on the back side of the electronic device. Also, even if only one camera is provided, it is possible to shoot while confirming the shooting state using the rear display.

Accordingly, there has been a need to secure various possibilities of utilizing the rear display when using the photographing function.

It is an object of the present invention to provide a user terminal device and a control method thereof for controlling a display disposed on the front and rear sides of a user terminal device based on a photographing situation.

According to an aspect of the present invention, there is provided a user terminal apparatus including a main display region disposed on a front side of the user terminal and a display region extending from one side of the main display region, A display configured of at least one sub display area disposed in one area, a camera capturing an image, and a live view acquired through the camera, in one of the main display area and the sub display area, and displaying the live view on the other of the main display region and the sub display region when the orientation of the live view is changed.

The processor displays the live view in the main display area and does not provide information in the sub display area. When the direction of the user terminal is changed, the processor displays the live view in the sub display area, Information may not be provided in the display area.

The processor displays the live view in the sub display area and does not provide information to the main display area. When a user touch is detected in the main display area, the processor captures and stores an image in response to the user touch have.

In addition, when the swipe interaction in a predetermined direction is sensed from any one point in the main display area, the processor may change the shooting setting value to correspond to the predetermined direction.

The processor displays the live view on the main display area and displays the face area included in the live view on the sub display area if it is recognized that the face of the person is included in the live view Can be controlled.

Also, the processor may control the display to crop one area of the live view so as to correspond to the size of the sub display area and display the cropped one in the sub display area.

The processor may display the live view in the main display area and may control the display to display an animation in the sub display area when the live view is recognized as including a person.

In addition, the processor may determine the distance to the subject, display the live view in the sub display area if the determined distance is less than the preset distance, and if the determined distance is greater than the preset distance, And to control the display to display in the main display area.

The processor may further include a plurality of sensors, and the processor may determine that the direction of the user terminal is changed based on at least one of a position, a motion, and a grip of the user detected by the plurality of sensors have.

If the face of the person is not recognized in a state where a face of a person having a predetermined size or larger is recognized in the live view or a face of a person having the predetermined size or more is recognized, It can be judged that it has been changed.

In order to achieve the above object, according to an embodiment of the present invention, there is provided a user terminal, comprising: a main display area disposed on a front face of a user terminal device; and a display area extending from one side of the main display area, The method comprising the steps of: displaying a live view acquired through the camera in one of the main display area and the sub display area; And displaying the live view on the other of the main display area and the sub display area when the orientation of the user terminal device is changed.

Also, the displaying in one of the main display region and the sub display region displays the live view in the main display region and does not provide information in the sub display region, And displaying the live view on the other one may not display the live view on the sub display area and provide information on the main display area when the direction of the user terminal is changed.

The displaying on the other of the main display region and the sub display region may include displaying the live view on the sub display region and not providing information on the main display region, And photographing and storing the image in response to the user touch.

The method may further include changing an imaging setting value to correspond to the predetermined direction when a swipe interaction in a predetermined direction is sensed from any one point in the main display area.

The displaying of the live view on the other of the main display region and the sub display region may display the live view on the main display region and may be included in the live view And displaying the face area in the sub display area.

In addition, the step of displaying in the sub display area may crop one area of the live view so as to correspond to the size of the sub display area and display the cropped image in the sub display area.

The step of displaying the live view on the other of the main display region and the sub display region displays the live view on the main display region and displays an animation on the sub display region The method comprising the steps of:

In addition, the displaying in one of the main display area and the sub display area may include determining a distance to a subject, displaying the live view in the sub display area if the determined distance is less than a preset distance, If the determined distance is greater than the preset distance, the live view can be displayed on the main display area.

The method of claim 1, further comprising detecting at least one of a position, a movement, and a grip of the user terminal, wherein displaying on the other of the main display region and the sub display region includes: The direction of the user terminal may be determined based on at least one of a movement and a grip of the user.

The displaying of the main display area and the sub display area may include displaying a face of a person having a predetermined size or more in the live view, If the face is not recognized, it can be determined that the direction of the user terminal is changed.

According to various embodiments of the present invention as described above, the user terminal device can easily perform the shooting by controlling the display disposed on the front and rear sides of the user terminal device based on various shooting situations.

1A and 1B are views for explaining an example of a display structure of a user terminal.
FIG. 2A is a block diagram illustrating a configuration of a user terminal according to an exemplary embodiment of the present invention. Referring to FIG.
2B is a block diagram showing an example of a detailed configuration of a user terminal.
2C is a diagram for explaining various modules stored in the storage.
3 is a view for explaining an example in which only one of the main display region, the sub display region and the round display region is used.
4 is a view for explaining an example in which at least two areas of the main display area, the sub display area and the round display area are used.
5 is a view for explaining an example of an operation according to each region and a touch input.
6 is a view for explaining an operation when the orientation of a user terminal device is changed according to an embodiment of the present invention.
7A and 7B are diagrams for explaining an example of a method of changing an image provided according to a display area.
8A and 8B are diagrams for explaining an example of a method of executing a specific function through the main display area.
9 is a view for explaining a method of displaying a live view in both the main display area and the sub display area according to the embodiment of the present invention.
10 is a view for explaining a method of displaying a live view in a main display area and a specific content in a sub display area according to an embodiment of the present invention.
11A and 11B are diagrams for explaining an example of a method for determining an area to display a live view. FIG. 12 illustrates a method of detecting a change in the direction of a user terminal according to an exemplary embodiment of the present invention. Fig.
13A and 13B are views for explaining an example of an operation according to a direction change of a user terminal.
14A and 14B are diagrams for explaining an example of a method for executing the photographing function.
15 is a view for explaining a method of using a round display area according to an embodiment of the present invention.
16 is a view for explaining a case where a camera is provided on the front side of a user terminal according to another embodiment of the present invention.
17A to 17C are diagrams for explaining an example in which other applications are used.
18A and 18B are views for explaining an example of the configuration of the display.
19 is a flowchart illustrating a method of controlling a user terminal according to an exemplary embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. It is to be understood that the disclosure herein is not intended to limit the scope of the invention to the specific embodiments but includes various modifications, equivalents, and / or alternatives of the embodiments. In the description of the drawings, like or similar reference numerals may be used for similar components.

Also, the terms " first, "" second," and the like used in the present specification are used to distinguish the various components from each other regardless of order or importance. Therefore, the order and significance of the components are not limited by these expressions. For example, without departing from the scope of the rights described in this document, the first component can be named as the second component, and similarly the second component can also be named as the first component.

It will also be appreciated that where an element (e.g., a first element) is operatively or communicatively coupled to or coupled to another element (e.g., a second element) it should be understood that referring to the fact that each component is directly connected or indirectly connected through another component (e.g., a third component). On the other hand, when it is mentioned that a component (e.g. a first component) is "directly connected" or "directly connected" to another component (e.g. a second component) It can be understood that there is no other component (e.g., a third component) between the elements.

The terms used in the present disclosure are used to describe any embodiment and may not be intended to limit the scope of other embodiments. In addition, although the singular representation may be used herein for convenience of description, it may be interpreted as including plural representations unless the context clearly indicates otherwise. In addition, terms used in this specification may have the same meaning as commonly understood by those skilled in the art. General predefined terms used herein may be interpreted in the same or similar sense as the contextual meanings of the related art and are to be understood as meaning ideal or overly formal meanings unless explicitly defined herein . In some cases, the terms defined herein can not be construed to exclude embodiments of the present disclosure.

In the following, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIGS. 1A and 1B are views for explaining an example of a display structure of the user terminal device 100. FIG.

1A is a front view of the user terminal device 100, and a right side view of FIG. 1A shows a rear surface of the user terminal device 100. As shown in FIG. A front display is disposed on a front surface of the user terminal device 100, and a rear display and a camera are disposed on a rear surface of the user terminal device 100. The front display and the rear display may be connected together, and the rear display may be smaller than the front display. However, the present invention is not limited thereto, and the front display and the rear display may be the same size. The camera may also be disposed on the front side of the user terminal device 100. Hereinafter, the front display will be referred to as a main display area 10 and the rear display will be referred to as a sub display area 20. [

1B shows a front view of the entire display in which the main display area 10, the sub display area 20 and the round display area 30 are connected, and the right side view of FIG. 1B shows a rear view of the entire display.

The sub display area 20 may extend from one side of the main display area 10 and be disposed in at least one area of the rear surface of the user terminal device 100. In particular, the sub display area 20 may extend from the upper side of the main display area 10 to be bent. The curved region may be curved, but is not limited thereto, and may be angled depending on the type of display.

The round display area 30 is an area connecting the main display area 10 and the sub display area 20. [ As described above, the round display area 30 may be curved or angled. The round display region 30 is separated from the main display region 10 and the sub display region 20 by the boundary lines 30-1 and 30-2.

The boundary lines 30-1 and 30-2 shown in Fig. 1B are merely one embodiment, and the boundary lines 30-1 and 30-2 can be changed as much as possible. Further, the boundary lines 30-1 and 30-2 may be determined by the manufacturer at the time of manufacture, but may be changed by the user. When the boundary lines 30-1 and 30-2 are changed, the area size of the main display area 10, the sub display area 20, and the round display area 30 is changed, and the size of the content displayed in each area Can be changed.

Although FIGS. 1A and 1B illustrate the upper side of the user terminal 100, the lower side, the left side, and the right side of the user terminal 100 may be enclosed. Further, it is also possible to arrange a plurality of side surfaces instead of one side surface.

Meanwhile, the display wraps the touch pad, and the display and the touch pad have a mutual layer structure, and can be implemented as a touch screen. In this case, the touch pad also bends similarly to the display, so that the touch input can be input as opposed to what the user thinks. For example, in the case of dragging from one point to the upper point of the main display area 10 and dragging from one point to the upper point of the sub display area 10, the user touches the same direction other than the difference of the touch areas . However, since the touch pad is bent, the direction of dragging from one point to the upper point of the main display area 10 and the case of dragging from one point to the upper point of the sub display area 10 are reversely input. Accordingly, the user terminal device 100 can be set to recognize the direction of the touch input of one area in the opposite direction. The details of the above are omitted for the sake of deviating from the scope of the present invention.

Since the main display area 10, the sub display area 20 and the round display area 30 are connected to each other, various sensors and receivers can be provided on the side or bottom of the user terminal 100, A directional re-server may be provided. Alternatively, the receiver may be provided in an integral cover that can be interlocked with the user terminal device 100. [ On the other hand, when the receiver is provided at the lower portion, the processor 130 may display the telephone conversation by inverting up and down.

Hereinafter, the structure in which the sub display area 20 extends from the main display area 10 through the round display area 30 of the curved shape and the camera is arranged on the rear surface of the user terminal 100 will be described. In addition, an embodiment will be described in addition to the case where the display and the camera having different structures are changed in position.

The direction from the main display area 10 and the sub display area 20 toward the round display area 30 will be described above the user terminal device 100 and the opposite side downward. The left and right sides are described with reference to the display area that the user is looking at. Therefore, the left and right sides of the case where the user looks at the main display area 10 and the case where the user looks at the sub display area 20 will be reversed.

FIG. 2A is a block diagram illustrating a configuration of a user terminal 100 according to an embodiment of the present invention.

2A, a user terminal device 100 includes a display 110, a camera 120, and a processor 130. [

FIG. 2A is a view illustrating various components in a case where the user terminal 100 is a device having various functions such as a display function and a control function. Therefore, depending on the embodiment, some of the components shown in Fig. 2A may be omitted or changed, and other components may be further added.

The display 110 includes a main display area 10 disposed on the front side of the user terminal device 100 and a sub display area 10 extending from one side of the main display area 10 and disposed in at least one area of the rear surface of the user terminal device 100. [ A display area 20 and a round display area 30 connecting the main display area 10 and the sub display area 20. [ However, the present invention is not limited thereto. For example, the sub display area 20 may be configured to cover the entire rear surface of the user terminal device 100.

Here, the front and rear surfaces are words used for convenience of explanation, and are not particularly limited to the meaning of words. For example, it may be one side and the other side for a particular electronic device. On the other hand, it has been described that it extends from one side of the main display region 10, but it is not limited thereto. For example, it may extend from all sides of the main display area 10 and cover the entire user terminal device 100 with a display.

The sub display area 20 of the display 110 may be arranged to extend from the upper side of the main display area 10 to be bent at the upper area of the rear side. For example, when the main display region 10 and the sub display region 20 are viewed from the side, a connecting portion can form a curve like a 'U' character. However, the present invention is not limited thereto, and when the main display region 10 and the sub display region 20 are viewed from the side, the connecting portion may have an angle of 90 degrees as shown by a "C" character. In addition, various connection parts may be formed based on the form of the user terminal device 100. [

The display 110 may display various UIs under the control of the processor 130. For example, the display 110 may display live views, gallery applications, animations, and the like.

The display 110 can display different contents in the main display area 10, the sub display area 20 and the round display area 30 under the control of the processor 130. [ For example, the display 110 may display a moving picture in the main display area 10, display an image in the sub display area 20, and display a UI for message transmission in the round display area 30 .

Alternatively, the display 110 may display at least two areas of the main display area 10, the sub display area 20, and the round display area 30 in association with each other. For example, the display 110 may display a moving picture in the main display area 10, and may display a UI in the sub display area 20 for controlling the moving picture. In addition, the display 110 may display a UI for providing a function irrelevant to moving images in the round display area 30.

Alternatively, the display 110 may display the same content in at least two areas of the main display area 10, the sub display area 20, and the round display area 30. For example, the display 110 may display the same contents in the main display area 10 and the sub display area 20, and may display different contents in the round display area 30. [

The display 110 may be implemented as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), or the like, but is not limited thereto. Also, the display 110 may be implemented as a transparent display, a flexible display, or the like as the case may be.

The camera 120 is a configuration for capturing a still image or a moving image under the control of the user. The camera 120 can photograph a still image at a specific point in time, but can continuously photograph a still image. The camera 120 provides the acquired image to the display 110 and a live view can be displayed on at least one of the main display area 10 and the sub display area 20. [ The camera 120 may photograph a user or photograph a background image in accordance with the orientation of the user terminal device 100. The camera 120 may be implemented as a plurality of cameras such as a front camera and a rear camera.

The camera 120 includes a lens, a shutter, a diaphragm, a solid-state image sensor, an analog front end (AFE), and a timing generator (TG). The shutter adjusts the amount of time the reflected light from the subject enters the user terminal device 100, and the aperture adjusts the amount of light incident on the lens by mechanically increasing or decreasing the size of the aperture through which the light enters. The solid-state image pickup device outputs an electric signal as an image due to the photoelectric charge when the light reflected from the object is accumulated in the photo-electric charge. The TG outputs a timing signal for reading out the pixel data of the solid-state image pickup device, and the AFE samples and digitizes the electric signal output from the solid-state image pickup device.

The processor 130 may control the operation of the user terminal 100 as a whole.

The processor 130 displays the live view acquired through the camera 120 in one of the main display area 10 and the sub display area 20 and displays the live view when the orientation of the user terminal device 100 is changed. The display 110 can be controlled to display the view on the other of the main display area 10 and the sub display area 20. [ The processor 130 may determine that the direction of the user terminal device 100 is changed using a gravity sensor, an acceleration sensor, a gyro sensor, or the like. Alternatively, the processor 130 may analyze the live view obtained by the camera 120 to determine that the direction of the user terminal device 100 is changed. However, the present invention is not limited to this, and the processor 130 may change the area in which the live view is displayed through movement, waving, etc. in the specific direction as well as the direction of the user terminal device.

The processor 130 displays the live view and can additionally display the guidelines. In addition to the GUI for photographing, the processor 130 may also display a GUI for executing a gallery app, a GUI for executing an image correction app, and the like.

The processor 130 displays a live view in the main display area 10 and does not provide information in the sub display area 20. When the direction of the user terminal device 100 is changed, And may not provide information to the main display area 10. There are various methods such as a method of not supplying power to the display 110, a method of displaying the display 110 in black, and the like.

The processor 130 may enable the touch function of the main display area 10 in a state in which the live view is displayed in the sub display area 20 and information is not provided in the main display area 10. [ When the user touches the main display area 10, the processor 130 may photograph and store the image in response to the user touch.

Here, the processor 130 may change the shooting setting value to correspond to a predetermined direction when a sweeping interaction in a predetermined direction is sensed from any one point in the main display area 10.

The processor 130 displays the live view on the main display area 10 and displays the face area included in the live view on the sub display area 20 when it is recognized that the face of the person is included in the live view The display 110 can be controlled.

Here, the processor 130 may control the display 110 to crop one area of the live view so as to correspond to the size of the sub display area 20 and display the cropped image in the sub display area 20. [

The processor 130 may also display the live view in the main display area 10 and may control the display 110 to display animations in the sub display area 20 when the live view is perceived as containing human .

If the determined distance is smaller than the preset distance, the processor 130 displays the live view in the sub display area 20. If the determined distance is greater than the preset distance, the processor 130 displays the live view The display 110 can be controlled to display in the main display area 10.

The user terminal device 100 may further include a plurality of sensors and the processor 130 may control the position of the user terminal 100 based on at least one of the position, It can be determined that the direction of the apparatus 100 has been changed.

Alternatively, when the face of a person of a predetermined size or larger is recognized in the live view or a face of a person of a predetermined size or more is recognized, the processor 130 determines the direction of the user terminal 100 Can be judged to have been changed.

2B is a block diagram showing an example of a detailed configuration of the user terminal device 100. As shown in FIG. 2B, the user terminal 100 includes a display 110, a camera 120, a processor 130, a GPS chip 135, a storage 140, a sensing unit 145, a communication unit 150, An interface unit 155, an audio processing unit 160, a video processing unit 170, a speaker 180, a button 181, and a microphone 182. Details of the components shown in FIG. 2B that overlap with those shown in FIG. 2A will not be described in detail.

The display 110 may be divided into a main display area 10, a sub display area 20, and a round display area 30 as described above. The display 110 may be implemented as various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, and a plasma display panel (PDP). The display 110 may include a driving circuit, a backlight unit, and the like, which may be implemented in the form of an a-si TFT, a low temperature poly silicon (LTPS) TFT, an OTFT (organic TFT) Meanwhile, the display 110 may be implemented as a touch screen by being combined with the touch sensing unit included in the sensing unit 145. [

At this time, the touch sensing unit may include at least one of a touch panel and a pen recognition panel. The touch panel senses a user's finger gesture input and outputs a touch event value corresponding to the sensed touch signal. The touch panel may be mounted under at least one area of the main display area 10, the sub display area 20 and the round display area 30 of the display 110. [

There are two ways in which the touch panel senses the user's finger gesture input: electrostatic and pressure sensitive. The electrostatic method is a method of calculating the touch coordinates by sensing the minute electricity generated by the user's body. The pressure reduction type includes two electrode plates built in the touch panel, and touch coordinates are calculated by detecting the current flowing through the upper and lower plates of the touch point.

The pen recognition panel senses the user's pen gesture input according to the operation of the user's touch pen (e.g., a stylus pen, a digitizer pen) and outputs a pen proximity event value or a pen touch event value can do. The pen recognition panel may be mounted under at least one of the main display area 10, the sub display area 20 and the round display area 30 of the display 110.

The pen recognition panel can be implemented, for example, by the EMR method, and can detect the touch or proximity input according to the proximity of the pen or the change in intensity of the electromagnetic field due to the touch. More specifically, the pen recognition panel includes an electromagnetic induction coil sensor (not shown) having a grid structure and an electronic signal processor (not shown) for sequentially providing an AC signal having a predetermined frequency to each loop coil of the electromagnetic induction coil sensor . When a pen incorporating a resonant circuit exists in the vicinity of the loop coil of the pen recognition panel, the magnetic field transmitted from the loop coil generates a current based on mutual electromagnetic induction in the resonant circuit in the pen. Based on this current, an induction magnetic field is generated from the coils constituting the resonance circuit in the pen. The pen recognition panel detects the induction magnetic field in the loop coil in the signal receiving state, so that the approach position or the touch position Can be detected.

The pen recognition panel can be configured differently for each display area. For example, both the touch panel and the pen recognition panel may be provided in the main display area 10, and only the touch panel may be provided in the sub display area 20 and the round display area 30.

Processor 130 may de-energize a particular panel by de-energizing the particular panel. Alternatively, the processor 130 may provide power to the touch panel and the pen recognition panel and receive a touch or pen input, but may also inactivate a particular panel by ignoring input to the particular panel in software.

In this case, at least one of the hand touch and the pen touch may be input depending on whether the touch panel corresponding to the main display area 10 and the pen recognition panel are activated or not. For example, by activating only the touch panel, only the hand touch input can be received without receiving the pen touch input, or only the pen recognition panel can be received, and only the pen touch input can be received without receiving the hand touch input.

Alternatively, the touch panel and the pen recognition panel may be provided in the entire display area, and the pen recognition panel in some areas may be inactivated. For example, the processor 130 may deactivate only the pen recognition panel corresponding to the sub display area 20 and the round display area 30.

Meanwhile, the touch panel and the pen recognition panel may be implemented as one panel. In this case, the processor 130 may sense the touch input and the pen input through the entire display area.

The processor 130 may perform an operation corresponding to the input based on at least one of an input type and an input area when the touch panel and the pen recognition panel are implemented as one panel. For example, processor 130 may ignore certain inputs, such as pen input, in software. Alternatively, the processor 130 may ignore the pen input to the sub display area 20 by software.

The processor 130 controls the operation of the user terminal 100 in general by using various programs stored in the storage 140.

Specifically, the processor 130 includes a RAM 131, a ROM 132, a main CPU 133, a graphics processing unit 134, first through n interfaces 135-1 through 135-n, .

The RAM 131, the ROM 132, the main CPU 133, the graphics processing unit 134, the first to n interfaces 135-1 to 135-n, etc. may be connected to each other via the bus 136. [

The first to n-th interfaces 135-1 to 135-n are connected to the various components described above. One of the interfaces may be a network interface connected to an external device via a network.

The main CPU 133 accesses the storage 140 and performs booting using the O / S stored in the storage 140. [ Then, various operations are performed using various programs stored in the storage 140. [

The ROM 132 stores a command set for booting the system and the like. The main CPU 133 copies the O / S stored in the storage 140 to the RAM 131 according to the instruction stored in the ROM 132, executes the O / S, . When the booting is completed, the main CPU 133 copies various application programs stored in the storage 140 to the RAM 131, executes the application program copied to the RAM 131, and performs various operations.

The graphic processing unit 134 generates a screen including various objects such as an icon, an image, and a text using an operation unit (not shown) and a rendering unit (not shown). The operation unit (not shown) calculates an attribute value such as a coordinate value, a shape, a size, and a color to be displayed by each object according to the layout of the screen based on the received control command. The rendering unit (not shown) creates screens of various layouts including the objects based on the attribute values calculated by the operation unit (not shown). The screen generated in the rendering unit (not shown) is displayed in the display area of the display 110.

Meanwhile, the operation of the processor 130 described above may be performed by a program stored in the storage 140. [

The storage 140 stores various data such as an operating system (O / S) software module for driving the user terminal device 100, a photographing module, and an application module.

In this case, the processor 130 may process and display the input image on the basis of the information stored in the storage 140.

The GPS chip 145 is a component for receiving the GPS signal from a GPS (Global Positioning System) satellite and calculating the current position of the user terminal device 100. The processor 130 may calculate the user location using the GPS chip 145 when using the navigation program or otherwise requiring the user's current location.

The communication unit 150 is configured to perform communication with various types of external devices according to various types of communication methods. The communication unit 150 includes a Wi-Fi chip 151, a Bluetooth chip 152, a wireless communication chip 153, an NFC chip 154, and the like. The processor 130 communicates with various external devices using the communication unit 150. [

The Wi-Fi chip 151 and the Bluetooth chip 152 perform communication using the WiFi method and the Bluetooth method, respectively. When the WiFi chip 151 or the Bluetooth chip 152 is used, various connection information such as an SSID and a session key may be transmitted and received first, and communication information may be used to transmit and receive various information. The wireless communication chip 153 refers to a chip that performs communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3rd Generation Partnership Project (3GPP), LTE (Long Term Evolution) The NFC chip 154 means a chip operating in the NFC (Near Field Communication) mode using the 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 to 960 MHz and 2.45 GHz.

Meanwhile, the communication unit 150 can perform unidirectional communication or bidirectional communication with the electronic device. When performing unidirectional communication, the communication unit 150 can receive a signal from the electronic device. When performing bidirectional communication, the communication unit 150 may receive a signal from the electronic device or transmit the signal to the electronic device.

The user interface unit 155 receives various user interactions. When the user terminal device 100 is implemented as a touch-based portable terminal, the user interface unit 155 may be implemented as a touch screen having a mutual layer structure with the touch pad. In this case, the user interface unit 155 can be used as the display 110 described above.

The sensing unit (not shown) includes a touch sensor, a geomagnetic sensor, a gyro sensor, an acceleration sensor, a proximity sensor, a grip sensor, and the like. In addition to the touch described above, the sensing unit can sense various operations such as rotation, tilt, pressure, approach, grip, and the like.

The touch sensor can be implemented as an electrostatic or a pressure sensitive type. The electrostatic touch sensor refers to a sensor that uses a dielectric coated on a surface of a display to calculate touch coordinates by sensing minute electricity that is excited by the user's body when a part of the user's body is touched on the display surface. The pressure-sensitive type touch sensor includes two electrode plates built in the user terminal device 100, and when the user touches the upper and lower plates of the touched point, touches a touch Sensor. In addition, infrared sensing, surface ultrasonic conduction, integral tension measurement, and piezo effects can be used to detect touch interactions.

In addition, the user terminal device 100 may judge whether or not a touch object such as a finger or a stylus pen is brought into contact with or close to each other by using a magnetic and magnetic field sensor, an optical sensor, or a proximity sensor instead of the touch sensor.

The geomagnetic sensor is a sensor for detecting the rotation state and the moving direction of the user terminal device 100. The gyro sensor is a sensor for sensing the rotation angle of the user terminal device 100. Both the geomagnetic sensor and the gyro sensor may be provided, but even if only one of them is provided, the user terminal device 100 can detect the rotation state.

The acceleration sensor is a sensor for detecting the degree of tilting of the user terminal device 100.

The proximity sensor is a sensor for sensing the approaching motion without touching the display surface directly. The proximity sensor is a high-frequency oscillation type that forms a high-frequency magnetic field and senses a current induced by a magnetic field characteristic that changes when approaching an object, a magnetic type using a magnet, a capacitive type And the like.

The grip sensor is disposed at the rear surface, the rim, and the grip portion separately from the touch sensor provided on the touch screen, and is a sensor that detects a user's grip. The grip sensor may be implemented as a pressure sensor in addition to the touch sensor.

The audio processing unit 160 is a component that performs processing on audio data. In the audio processing unit 160, various processes such as decoding and amplification of audio data, noise filtering, and the like may be performed.

The video processing unit 170 is a component that performs processing on video data. The video processing unit 170 may perform various image processing such as decoding, scaling, noise filtering, frame rate conversion, resolution conversion, and the like on the video data.

The speaker 180 is a component for outputting various kinds of audio data processed by the audio processing unit 160, as well as various kinds of notification sounds and voice messages.

The button 181 may be various types of buttons such as a mechanical button, a touch pad, a wheel, and the like formed in an arbitrary area such as a front surface portion, a side surface portion, and a back surface portion of the outer surface of the main body of the user terminal device 100.

The microphone 182 is a configuration for receiving a user voice or other sound and converting it into audio data.

2B, depending on the embodiment, the user terminal 100 may include a USB port to which a USB connector can be connected, various external inputs for connecting to various external terminals such as a headset, a mouse, and a LAN Port, a DMB chip for receiving and processing a DMB (Digital Multimedia Broadcasting) signal, various sensors, and the like.

FIG. 2C is a diagram for explaining various modules stored in the storage 140. FIG.

The software of FIG. 2C may be stored in the storage 140 but is not limited thereto and may be stored in various types of storage means used in the user terminal device 100. 2C, software including an OS 191, a kernel 192, a middleware 193, an application 194, and the like may be stored in the user terminal device 100.

An operating system (OS) 191 controls and manages the overall operation of the hardware. That is, the OS 191 is a layer responsible for basic functions such as hardware management, memory, and security.

The kernel 192 serves as a channel for transmitting various signals including the touch signal and the like sensed by the display 110 to the middleware 193.

The middleware 193 includes various software modules for controlling the operation of the user terminal device 100. 2C, the middleware 193 includes a main UI framework 193-1, a window manager 193-2, a sub UI framework 193-3, a security module 193-4, a system manager 193-3, -5, a connection manager 193-6, an X11 module 193-7, an APP manager 193-8, and a multimedia framework 193-9.

The main UI framework 193-1 is a module for providing various UIs to be displayed in the main display area 10 of the display 110. The sub UI framework 193-3 is a module for providing various UIs to be displayed in the sub display area 20 It is a module for providing various UIs to be displayed. The main UI framework 193-1 and the sub UI framework 193-3 include an image composer module constituting various objects, a coordinate synthesizer for calculating coordinates to be displayed on the object, A 2D / 3D UI toolkit that provides a tool for constructing a UI in 2D or 3D form, and the like.

The window manager 193-2 can detect touch events or other input events using the user's body or pen. When this event is detected, the window manager 193-2 transmits an event signal to the main UI framework 193-1 or the sub UI framework 193-3 to perform an operation corresponding to the event.

In addition, when a user touches and drags a screen, a writing module for drawing a line according to a drag trajectory or an angle calculation for calculating a pitch angle, a roll angle, and a yaw angle based on the sensor value sensed by the sensing portion Modules and the like may be stored.

The security module 193-4 is a module that supports hardware certification, permission permission, and secure storage.

The system manager 193-5 monitors the status of each component in the user terminal device 100 and provides the monitoring result to other modules. For example, when a remaining battery level is low, an error occurs, a communication connection state is lost, or the like, the system manager 193-5 transmits the monitoring result to the main UI framework 193-1 or the sub UI frame Work 193-3 to output a notification message or a notification sound.

The connection manager 193-6 is a module for supporting a wired or wireless network connection. The connection manager 193-6 may include various detailed modules such as a DNET module, a UPnP module, and the like.

The X11 module 193-7 is a module for receiving various event signals from various hardware provided in the user terminal device 100. [ Here, the event can be variously set, such as an event in which a user operation is detected, an event in which a system alarm occurs, an event in which a specific program is executed or terminated, and the like.

The APP manager 193-8 is a module for managing execution states of various applications installed in the storage 140. [ When the event that the application execution command is input from the X11 module 193-7 is detected, the APP manager 193-8 calls and executes the application corresponding to the event. That is, when at least one object selected event is detected, the APP manager 193-8 performs an operation to call and execute the application corresponding to the object.

The multimedia framework 193-9 is a module for reproducing the multimedia contents stored in the user terminal device 100 or provided from an external source. The multimedia framework 193-9 may include a player module, a camcorder module, a sound processing module, and the like. Accordingly, various multimedia contents can be reproduced to generate and reproduce a screen and sound.

The software structure shown in FIG. 2C is merely an example, and is not necessarily limited to this. Therefore, it goes without saying that parts may be omitted, modified, or added as necessary. For example, the storage 140 may include a sensing module for analyzing signals sensed by various sensors, a messaging module such as a messenger program, a Short Message Service (SMS) & Multimedia Message Service (MMS) program, an email program, (Call Info Aggregator) program module, a VoIP module, a web browser module, and the like.

Meanwhile, as described above, the user terminal device 100 may be implemented as various types of devices such as a mobile phone, a tablet PC, a laptop PC, a PDA, an MP3 player, an electronic frame device, a TV, a PC, a kiosk, 2B and 2C may be modified in various ways according to the type of the user terminal device 100. For example,

As described above, the user terminal device 100 can be implemented in various forms and configurations.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a basic configuration and various embodiments for facilitating understanding of the present invention will be described.

3 is a view for explaining an example in which only one of the main display region 10, the sub display region 20, and the round display region 30 is used.

As shown in FIG. 3, the processor 130 may determine an area to provide information based on the orientation of the user terminal device 100. For example, as shown in FIG. When the processor 130 provides information to one area, it may not provide information to the other two areas.

The processor 130 may perform an operation of determining an area to provide information when there is a user input for pressing the power button. Alternatively, the processor 130, when receiving a telephone or text, may perform an operation of determining an area to provide information.

In the above description, the area for providing information is determined based on the orientation of the user terminal device 100, but the present invention is not limited thereto. For example, the processor 130 may recognize a user when there is a user input to press the power button, and may provide information through an area close to the user's line of sight. At this time, the provided information may be different depending on the executed application.

Meanwhile, the processor 130 may change the display area to which information is to be provided through the touch input of the user with respect to the area not providing information. For example, when a user input that continuously touches one area of the sub display area 20 is received while providing information to the main display area 10, the processor 130 provides the main display area 10 with information Information may be displayed in the sub display area 20. [

On the other hand, the processor 130 may display different information according to an area to be displayed even though the same information is displayed. For example, as shown in FIG. 3, the processor 130 may change the layout of the home screen for each display area and display the changed home screen.

Meanwhile, when the direction of the user terminal device 100 is changed, the processor 130 may change the area for providing information. For example, if the user terminal device 100 is rotated by a predetermined angle or more while the home screen is displayed in the main display area 10, the processor 130 can display the home screen in the sub display area 20 have.

In particular, the processor 130 may change the layout of the information to be provided when the direction of the user terminal device 100 is changed to change the display area. The processor 130 may change not only the object but also the size, content, and layout of the object when the area to be provided with information is changed.

Meanwhile, when the direction of the user terminal device 100 is changed and the display area is changed, the processor 130 may change the operation state of the running application and provide the changed display area to another display area. Alternatively, the processor 130 may divide the displayed information and display it in other display areas when the direction of the user terminal device 100 is changed.

4 is a view for explaining an example in which at least two of the main display region 10, the sub display region 20, and the round display region 30 are used.

As shown in FIG. 4, the processor 130 may provide information related to the two areas of the main display area 10, the sub display area 20, and the round display area 30. For example, the processor 130 may display a moving picture in the main display area 10, and may display a UI for controlling the moving picture in the round display area 30.

However, the present invention is not limited thereto, and the processor 130 may provide information that is not related to each other in two areas of the main display area 10, the sub display area 20, and the round display area 30.

4, the processor 130 displays the telephone receiving UI in the round display area 30, and displays the moving image in the round display area 30, and when the telephone is displayed in the state in which the UI for controlling the moving picture is displayed in the round display area 30, The UI for control may be moved to the sub display area 20 and displayed.

The processor 130 may display the phone call UI in the round display area 30 and continuously reproduce the moving picture when the phone is connected by touching the phone receiving UI. At this time, the processor 130 may mute the moving picture. Alternatively, the processor 130 may pause the motion picture.

Meanwhile, the processor 130 may provide information related to each other using all of the main display area 10, the sub display area 20, and the round display area 30. For example, the processor 130 may display a moving picture in the main display area 10, display a UI for controlling the moving picture in the sub display area 20, and display a moving picture playback time in the round display area 30 UI can be displayed.

Alternatively, the processor 130 may provide information that is not related to each other by using all of the main display area 10, the sub display area 20, and the round display area 30. Alternatively, the processor 130 may provide information related to each other in two areas of the main display area 10, the sub display area 20, and the round display area 30, and may provide information that is not related to the other area You may.

On the other hand, when the direction of the user terminal device 100 is changed, the processor 130 can change the display area of information displayed in each area. In particular, the processor 130 may combine or divide the information displayed in each area to change the displayed area.

The processor 130 may display the UI displayed on the sub display area 20 in the main display area 10 when the touch input is received on the sub display area 20. [ Alternatively, when the touch input is received on the main display area 10, the processor 130 displays the UI displayed in the main display area 10 in at least one of the sub display area 20 and the round display area 30 You may.

Meanwhile, the processor 130 may adjust the degree of adjustment of the set value according to the touch region even if the same user touch input is sensed. For example, when the drag operation for the main display area 10 or the sub display area 20 is input while the moving image is displayed in the main display area 10, Time or volume can be adjusted. At this time, the processor 130 can finely control the reproduction time, volume, and the like in the case of the sub display area 20 as compared with the case where the drag operation is the main display area 10.

4 is only an example, and the present invention is not limited thereto. For example, the main display area 10, the sub display area 20, and the round display area 30 shown in Fig. 4 can be replaced with each other. In addition, the operation of FIG. 4 can be applied to any other application.

5 is a view for explaining an example of an operation according to each region and a touch input.

5, when the processor 130 receives a drag input from one area of the main display area 10 toward the round display area 30, the processor 130 converts the information provided in the main display area 10 into a sub- Can be displayed in the area (20).

The processor 130 may display information provided in the main display area 10 in the sub display area 20 only when there is a drag input of a predetermined size or larger.

The processor 130 may no longer provide the information provided in the sub display area 20 when there is information provided in the sub display area 20 before the drag input. Alternatively, the processor 130 may display the information provided in the sub display area 20 in the main display area 10 or the round display area 30. [

5, only the main display region 10 and the sub display region 20 are used. However, the present invention is not limited thereto. For example, the processor 130 may display information provided to one of the main display area 10 and the sub display area 20 in the round display area 30, depending on the type of user input. Alternatively, the processor 130 may display information provided in the round display area 30 in one of the main display area 10 and the sub display area 20. [

Meanwhile, the processor 130 may receive the touch input even if the processor 130 does not provide information to a specific area. For example, the processor 130 may change the channel or volume of broadcast content when a user input for dragging the round display area 30 is received while the broadcast content is being displayed in the main display area 10. At this time, the round display area 30 may not be provided with information.

6 is a view for explaining an operation when the orientation of the user terminal 100 according to the embodiment of the present invention is changed.

6, the user terminal 100 includes a main display area 10 disposed on the front surface of the user terminal 100, a display area 10 extending from one side of the main display area 10, And a display area 110 including a main display area 10 and a round display area 30 connecting the sub display area 20 with each other.

The user terminal 100 may include a camera 120 for capturing an image on the rear side. For convenience of description, a structure in which the camera 120 is provided on the rear surface of the user terminal 100 will be described. The structure in which the camera 120 is provided on the front surface of the user terminal device 100 will be described later.

The processor 130 may display the live view obtained through the camera 110 in one of the main display area 10 and the sub display area 20. [ 6 is a diagram in which the processor 130 displays a live view in the main display area 10. In FIG. Alternatively, the processor 130 may display the live view in the sub display area 20, and the display of the main display area 10 and the sub display area 20 will be described later.

The processor 130 may control the display 110 to display the live view on the other of the main display area 10 and the sub display area 20 when the orientation of the user terminal device 100 is changed . 6, when the processor 130 displays a live view on the main display area 10 and a rotational movement of a predetermined angle or more is detected in a specific direction, the live view is displayed on the sub display area 20 FIG. The image captured by the camera 120 may also be changed as the direction of the user terminal device 100 is changed. The processor 130 can determine that the direction of the user terminal device 100 has been changed through various methods, and the details will be described later.

When the direction of the user terminal device 100 is changed and the area in which the live view is displayed is changed, the processor 130 may display a soft key, a control GUI, or the like so as to correspond to the area. For example, when the live view is displayed in the main display area 10, the processor 130 can display a GUI for photographing on the right side of the main display area 10. [ When the direction of the user terminal device 100 is changed and the live view is displayed in the sub display area 20, the processor 130 does not display the GUI for photographing but touches the sub display area 20 You can also take a picture.

The processor 130 may display the shooting setting value in the main display area 10 when the direction of the user terminal device 100 is changed while the live view is being displayed in the sub display area 20. [

The processor 130 displays a live view in the main display area 10 and does not provide information in the sub display area 20. When the direction of the user terminal device 100 is changed, And may not provide information to the main display area 10. [ For example, in the case of the upper diagram of Fig. 6, the processor 130 may not provide information to the sub display area 20. [ Alternatively, in the case of the bottom view of FIG. 6, the processor 130 may not provide information to the main display area 10. However, the present invention is not limited to this, and the live view may be displayed in both the main display area 10 and the sub display area 20 of the processor 130. [

The processor 130 may display one area of the main display area 10 and the sub display area 20 in black so as not to provide information. Alternatively, the processor 130 may not supply power to the display 110 of one of the main display region 10 and the sub display region 20. [ Even when the processor 130 does not provide information to one of the main display region 10 and the sub display region 20, the processor 130 may activate the touch function to receive the touch input.

The processor 130 may change the shooting mode when the direction of the user terminal device 100 is changed. For example, the processor 130 may switch to the self-portrait mode if the direction of the user terminal device 100 is changed to recognize that the subject is close. The self-portrait mode is a mode optimized for self-portrait photography, and can automatically correct the user's skin.

Alternatively, the processor 130 may switch to the background shooting mode if the direction of the user terminal device 100 is changed to recognize that the subject is distant. Background shooting mode is optimized for landscape shooting, and can automatically adjust the contrast and white balance.

Meanwhile, the processor 130 may vary the view angle depending on the photographing mode. For example, the processor 130 may narrow the angle of view in the self-portrait mode and widen the angle of view in the background mode.

The processor 130 may change the UI provided in the display according to the photographing mode. For example, the processor 130 provides UI including a function of correcting the user's skin, a red-eye reduction function and the like in the case of the self-portrait photographing mode. In the case of the background photographing mode, the processor 130 provides a contrast adjustment function, And the like.

7A and 7B are diagrams for explaining an example of a method of changing an image provided according to a display area.

As shown in FIG. 7A, the processor 130 may display a live view in the main display area 10. When the user performs the shooting operation, the processor 130 can store the same image as the image displayed in the main display area 10. [

Alternatively, as shown in FIG. 7B, the processor 130 may display a live view in the sub display area 20. [ At this time, the processor 130 can display only a part of the image recognized through the camera 120. [

7A and 7B show that the same subject is photographed for convenience of explanation. That is, the processor 130 photographs the image shown in FIG. 7A, but can display only a part of the photographed image in the sub display area 20 as shown in FIG. 7B.

However, this is merely an embodiment, and the processor 130 may display the entire photographed image in the sub display area 20 by rotating 90 degrees.

Also, even if the photographing is performed while the live view is displayed in the sub display area 20, the processor 130 may be storing the entire image. For example, even if the user performs the self-portrait photographing as shown in FIG. 7B, the processor 130 can photograph the original image including not only the user's face but also the upper body, as shown in FIG. 7A.

In this case, the processor 130 may display the stored image differently according to the area where the gallery application or the like is executed and displayed. For example, when an image is displayed in the main display area 10, the processor 130 can display an original image including up to the user's upper body. Alternatively, when an image is displayed in the sub display area 20, the processor 130 may display a partial image including only the face of the user.

The processor 130 may store both the original image and the partial image. Alternatively, the processor 130 may store only the original image, crop the image so as to correspond to the sub display area 20 as needed, and display the image in the sub display area 20. [ 8A and 8B are views for explaining an example of a method of executing a specific function through the main display area 10. In this case, The processor 130 executes a corresponding function according to a user input that touches the main display area 10 in a state in which the live view is displayed in the sub display area 20 and information is not provided in the main display area 10 .

8A shows that the main display area 10 is used to photograph an image. For example, when the processor 130 displays a live view in the sub display area 20 and does not provide information to the main display area 10, if the user touch is detected in the main display area 10, The image can be captured and stored in response to the touch.

The processor 130 may perform photographing only when a user touch to the predetermined area 11 of the main display area 10 is detected. The predetermined area 11 may be set by a manufacturer or may be set by a user. In this case, the processor 130 may activate the touch function of the predetermined area 11 of the main display area 10 and deactivate the touch function of the other area of the main display area 10. [

8B is a view for changing the photographing setting value by using the main display area 10. Fig. For example, the processor 130 displays a live view in the sub display area 20 and displays the live view in a predetermined direction from any one point in the main display area 10 without providing information in the main display area 10 The sweep setting value can be changed so as to correspond to the predetermined direction.

The processor 130 may change the shooting setting value only when a sweeping interaction in a predetermined direction is sensed from any one point in the predetermined area 11 of the main display area 10. [ The preset area may be set by the manufacturer or may be set by the user. In this case, the processor 130 may activate the touch function of the predetermined area 10 of the main display area 10 and deactivate the touch function of the other area of the main display area 10.

The processor 130 can display the live view by changing the exposure value when the swipe interaction of the left and right directions 810-1 and 810-2 is detected. The processor 130 can display a live view by zooming in or out when the swipe interaction of the up and down directions 820-1 and 820-2 is detected. However, the present invention is not limited thereto, and the processor 130 may change the focus or change the white balance through swipe interaction.

8A and 8B, the predetermined area 11 of the main display area 10 is described, but the present invention is not limited thereto. For example, it is possible to activate the touch function of the entire main display area 10 of the processor 130 and to perform the photographing when a user touch to a point of the main display area 10 is detected. Alternatively, the processor 130 may divide the main display area 10 into a plurality of preset areas, and assign different functions to the respective areas.

In addition to touch and swipe interaction, the processor 130 can perform corresponding functions for various interactions such as drag-and-drop interaction, multi-touch interaction, and stretch interaction that opens two points. The interaction and the corresponding function may be set by the manufacturer or may be set by the user.

8A and 8B, the main display region 10 is touched to perform a specific function. However, the present invention is not limited thereto. For example, if the processor 130 displays a live view in the main display area 10 and does not provide information to the sub display area 20, if the user touch is detected in the sub display area 20, The image can be captured and stored in response to the touch. Alternatively, the processor 130 may change the imaging setting value to correspond to a predetermined direction when a sweeping interaction in a predetermined direction is sensed from any one point in the sub display area 20. [

Alternatively, the processor 130 may display a live view in both the main display area 10 and the sub display area 20, and use an area located on the opposite side of the area viewed by the user as an area for receiving the touch input .

8A and 8B, only the case of using the camera 120 is described. However, even when another application is used, the touch input may be received using an area located on the opposite side of the area viewed by the user. For example, the processor 130 may change the channel or the volume through the swipe interaction of the sub display area 20 while displaying the broadcast content in the main display area 10.

8A and 8B, only the main display region 10 and the sub display region 20 are used. However, any one of the main display region 10 and the sub display region 20 and the round display region 30 ) May be used. For example, the processor 130 may perform shooting according to a user input that touches the round display area 30 in a state where a live view is displayed in the sub display area 20. [

9 is a view for explaining a method of displaying a live view in both the main display area 10 and the sub display area 20 according to an embodiment of the present invention.

9, the processor 130 displays the live view in the main display area 10, and when it is recognized that a human face is included in the live view, the face area included in the live view is displayed in the sub display area The display 110 may be controlled to display on the display 20 of the display. For example, the processor 130 may display the live view in the main display area 10 when the distance from the subject is long. Details of this will be described later.

The processor 130 can recognize the face of a person in the live view displayed in the main display area 10. [ In a case where a human face is recognized using a live view, the processor 130 divides a specific still image of live view into a plurality of pixel block units, and calculates a representative pixel value for each pixel block. The representative pixel value may be calculated as an average value of all the pixels included in the pixel block, or may be calculated as a maximum distribution value, an intermediate value, a maximum value, and the like. The processor 130 compares the representative pixel values of each pixel block with each other to determine whether or not the pixel blocks having the representative pixel value of the similar range are successively arranged. When placed consecutively, the processor 130 determines that the pixel blocks constitute one object. The processor 130 determines whether there is an object having a pixel value range similar to human skin color among the pixel blocks judged as objects. When such an object exists, the processor 130 can recognize the object as a face region or other body region of a person, and determine the remaining portion as a surrounding background.

The processor 130 may control the display 110 to display one area 910 of the live view in the sub display area 20 so as to correspond to the size of the sub display area 20. [ Particularly, the processor 130 can crop one area 910 of the live view while maintaining the aspect ratio of the sub display area 20. [ The processor 130 may crop one area 910 of the live view so that the face area included in the live view is located at the center of the sub display area 20. [ Alternatively, the processor 130 may crop an area smaller than one area 910 of the live view shown in FIG. 9, enlarge the cropped area, and display it on the sub display area 20. FIG.

The processor 130 can display a face region of a person nearest to a plurality of people's faces in the sub display region 20 when the processor 130 recognizes that a plurality of people's faces are included in the live view. Alternatively, the processor 130 may display a face region of a person who maintains a line of sight of the face of the user in the direction of the user terminal 100 in the sub display region 20. Alternatively, the processor 130 may crop one region of the live view so that a plurality of people are all present. The processor 130 may display the live view in the sub display area 20 without cropping if there are many people. Thus, a person who performs shooting using the user terminal device 100 can confirm an image to be photographed through a live view displayed in the main display area 10, You can see the image to be shot through one area of the displayed cropped live view.

9, the operation of displaying the live view in both the main display region 10 and the sub display region 20 has been described, but the present invention is not limited thereto. For example, when the processor 130 operates in the self-portrait mode, the main display area 10 displays a live view in the sub display area 20, and the UI in the main display area 10 indicates that the self-photographing is in progress.

10 is a view for explaining a method of displaying a live view in the main display area 10 and a specific content in the sub display area 20 according to an embodiment of the present invention.

10, the processor 130 displays the live view in the main display area 10 and displays the animation in the sub display area 20 when it is recognized that the live view is included in the display 110 Can be controlled. For example, the processor 130 may display a child animation or an animal image in the sub display area 20 when a child is recognized in the live view. The processor 130 may output a sound corresponding to the content displayed in the sub display area 20. [ Thereby, it is possible to orient the line of sight of the person being photographed toward the user terminal device 100.

The processor 130 may determine the content to be displayed in the sub display area 20 based on at least one of the age and gender of the person recognized in the live view. For example, the processor 130 may display a precious metal or the like in the sub display area 20 when the person recognized in the live view is an exciter.

When a plurality of persons are recognized in the live view, the processor 130 can determine contents to be displayed in the sub display area 20 based on a person who is closest to the plurality of persons. Alternatively, the processor 130 may determine the content to be displayed in the sub display area 20 based on a plurality of people who maintain their gaze in a direction other than the user terminal device 100. Alternatively, the processor 130 may determine the content to be displayed in the sub display area 20 based on the person located closest to the central area of the live view.

In FIGS. 9 and 10, the case where a face of a person or a person is recognized in the live view has been described, but the present invention is not limited thereto. For example, the processor 130 may display one area of the live view, an animation, or the like in the sub display area 20 even if an animal or the like is recognized in the live view. Alternatively, the processor 130 may display one area of the live view, an animation, or the like in the sub display area 20 according to a user's operation.

11A and 11B are views for explaining an example of a method for determining an area for displaying a live view. The processor 130 determines the distance to the subject and displays the live view in the sub display area 20 if the determined distance is less than the preset distance. If the determined distance is greater than the preset distance, And to control the display 110 to display in the area 10.

11A shows a case where the distance to the subject is smaller than a predetermined distance, and FIG. 11B shows a case where the distance to the subject is larger than a preset distance. The processor 130 may determine the distance to the subject using autofocusing. Alternatively, the processor 130 may analyze the still image at a specific point in time of the live view to determine the distance to the subject based on the outline of the subject. Thus, when the user views the back side of the user terminal device 100 and photographs the user himself / herself, the live view is displayed in the sub display area 20, so that the user can view and photograph the image to be photographed. Alternatively, when the user views the front side of the user terminal device 100 and photographs another person or scenery, a live view is displayed in the main display area 10 to confirm and photograph the image to be photographed.

However, the present invention is not limited thereto, and the processor 130 may display a live view in an area opposite to the above-described method, or may display a live view in both the main display area 10 and the sub display area 20. [

Alternatively, the processor 130 may display a live view in an area where a touch operation of the user is performed.

Alternatively, the processor 130 may analyze the image captured by the camera 120 to determine an area for displaying the live view. For example, the processor 130 may recognize the user from the photographed image and determine the area to display the live view.

Alternatively, the processor 130 may determine an area for displaying the live view through iris recognition of the user.

12 is a diagram for explaining a method of detecting a change in the direction of the user terminal 100 according to an embodiment of the present invention.

As shown in FIG. 12, when the direction of the user terminal device 100 is changed, the image of the live view can be changed. The user terminal device 100 may include a plurality of sensors as described above. The processor 130 may determine that the direction of the user terminal device 100 has been changed based on at least one of the position, movement, and grip of the user terminal device 100 sensed by the plurality of sensors. For example, the processor 130 determines the orientation of the user terminal device 100 based on the rotation state and the movement direction of the user terminal device 100 by the geomagnetic sensor and the rotation angle of the user terminal device 100 by the gyro sensor Can be judged to have been changed.

The processor 130 may determine that the direction of the user terminal device 100 has changed when the user terminal device 100 rotates more than a predetermined angle in a specific direction. For example, when the user terminal device 100 rotates by 90 degrees or more, the processor 130 may determine that the direction of the user terminal device 100 has changed and perform a corresponding operation.

If the face of the person is not recognized in a state in which a face of a person of a predetermined size or larger is recognized or a face of a person of a predetermined size or more is recognized in the live view, May be judged to have been changed. 12 shows a plurality of still images of the live view in sequence. The first still image 1210 is a human-free image, the second to fourth still images 1220 to 1240 are human images, and the fifth still image 1250 is a human-free image. That is, FIG. 12 is a diagram in which the camera angle is moved from left to right and then moved to the left.

The processor 130 may determine that the direction of the user terminal 100 is changed at the moment when the second still image 1220 is displayed and at the moment when the fifth still image 1250 is displayed as shown in FIG.

However, the present invention is not limited thereto, and the processor 130 may determine that the direction of the user terminal device 100 is changed based on an object other than a human face. For example, the processor 130 extracts the outline of the slide from a specific still image of the live view, sequentially extracts the outline of the slide of the next still image, and the outline of the slide is moved beyond a predetermined distance on the live view It can be determined that the direction of the user terminal device 100 has been changed.

Alternatively, the processor 130 may continuously detect the absolute posture using various sensors provided in the user terminal device 100, and determine that the direction is changed.

At this time, the processor 130 determines the reference posture and can determine that the direction is changed when the posture is changed from the reference posture to a predetermined value or more. Here, the reference posture may be a posture when the camera application is selected.

Alternatively, the processor 130 may determine that the user terminal device 100 has been changed in accordance with the user's touch operation.

13A and 13B are views for explaining an example of an operation according to a direction change of the user terminal device 100. In FIG.

13A shows that the gallery app is executed when the direction of the user terminal device 100 is changed while the photographing function is being executed. For example, the processor 130 may display a live view in the sub display area 20 and display the gallery app in the main display area 10 when the user terminal device 100 rotates about 180 degrees. Alternatively, the processor 130 may display a telephone function, an Internet function, etc., which are not related to the photographing function, in the main display area 10. Alternatively, the processor 130 may stop the photographing function.

When the user touches the main display area 10 while the image is being displayed and the image stored in the sub display area 20 is being displayed, the processor 130 displays the gallery application or the application for correction in the sub display area (20). Alternatively, the processor 130 may display an image sequentially photographed in the sub display area 20 when the image capturing is performed, and may display the image by changing the image by touching the main display area 10 of the user.

13B shows that the moving picture photographing function is stopped when the direction of the user terminal device 100 is changed while the moving picture photographing function is being executed. 13B shows a shooting time, a GUI 1310 for ending movie shooting, and a GUI 1320 for temporarily stopping movie shooting. The user may touch the GUI 1310 to end the moving picture shooting to terminate the moving picture shooting, but may change the direction of the user terminal device 100 to terminate the moving picture shooting. 13B shows a thumbnail image of a moving image which has been photographed at present, and a moving image can be executed by touching the GUI 1330 for executing the moving image.

However, the present invention is not limited thereto, and the processor 130 may pause or terminate the motion picture shooting by touching the main display area 10 during motion picture shooting.

14A and 14B are diagrams for explaining an example of a method for executing the photographing function. The processor 130 can execute the photographing function by a method other than touching the icon indicating the photographing function.

14A shows that the photographing function is executed when the direction of the user terminal device 100 is changed. For example, in a state where the lock screen is displayed in the main display area 10, the processor 130 executes the shooting function when the direction of the user terminal device 100 is changed, You can display the view. Alternatively, when the direction of the user terminal device 100 is changed, the processor 130 executes the photographing function while displaying the live view in the sub display area 20 without providing information in the main display area 10, You may.

14B shows that the photographing function is executed when a swipe interaction is detected in the sub display area 20. [ For example, when the swipe interaction is detected in the sub display area 20 in a state where the lock screen is displayed in the sub display area 20 or the information is not provided in the sub display area 20 The photographing function can be executed. FIG. 14B shows the swipe interaction from left to right, but it is not limited to a specific direction. It is also possible to perform shooting functions through swiping interactions as well as other interactions.

14A and 14B, the photographing function can be executed by various methods. For example, when the processor 130 activates at least one of the main display region 10 and the sub display region 20 in a state in which the main display region 10 and the sub display region 20 do not provide information , The user can recognize the intention and execute the photographing function. The processor 130 may recognize at least one of the face of the user, the gesture of the user, and the grip shape at the time when the specific area is activated, and execute the shooting function. For example, if the user is making a "V" shape using his or her finger, the processor 130 may execute a photographing function.

15 is a diagram for explaining a method of using the round display area 30 according to an embodiment of the present invention.

As shown in FIG. 15, the round display area 30 is located between the main display area 10 and the sub display area 20. The UI for controlling the user terminal device 100 may also be displayed in the round display area 30. For example, when the photographing function is activated, the processor 130 may display the GUI 1520 for photographing and storing the image in the round display area 30. [ The processor 130 can photograph and store an image when the user touches the GUI 1520 for shooting and storing the image.

The processor 130 can display the live view by changing the exposure value when the swipe interaction of the left and right directions 1510-1 and 1510-2 of the round display area 30 is detected. Alternatively, the processor 130 may display a live view by zooming in or out. However, the present invention is not limited thereto, and the processor 130 may change the focus or change the white balance through swipe interaction.

However, the present invention is not limited thereto, and the processor 130 may execute the photographing function when a swipe interaction with the round display area 30 is detected. Alternatively, the processor 130 may touch the round display area 30 to change the area where the live view is displayed.

The processor 130 may display a specific notification in the round display area 30. [ For example, when the subject 130 is not focused on the subject, the processor 130 may display a notice in the round display area 30 to inform the subject. When the user touches the notification, the processor 130 may change settings or retry to focus the subject.

FIG. 16 is a view for explaining a case where a camera 120 is provided on a front surface of a user terminal 100 according to another embodiment of the present invention.

As shown in FIG. 16, the user terminal 100 may include a camera 120 below the main display area 10. However, most of the operations of the embodiment described above can be applied to the main display region 10 and the sub display region 20 in the opposite manner, and a detailed description thereof will be omitted.

Although the camera application has been described above, similar operations are possible for other applications.

17A to 17C are diagrams for explaining an example in which other applications are used.

As shown in FIG. 17A, when an event occurs, the processor 130 can determine the display area based on the direction of the user terminal device 100 at the time when the event occurs. For example, when a telephone or a message is received, the processor 130 senses the absolute posture of the user terminal device 100 at the time the telephone or message is received, and displays the contents of the telephone or message in the predetermined display area . In particular, the processor 130 may display the received contents of the telephone or message in the sub display area 20 when the main display area 10 is in contact with the table.

Alternatively, as shown in FIG. 17B, the processor 130 may determine the display area based on the user position at the time when the event occurs, when the event occurs. For example, when a telephone or a message is received, the processor 130 recognizes the position of the user at the time when the telephone or message is received through the camera 120, and when the user is recognized, And displays the received contents of the telephone or message in the main display area 10 if the user is not recognized.

Alternatively, as shown in FIG. 17C, the processor 130 may determine the display area based on the use state of the user terminal device 100 at the time when the event occurs, when the event occurs. For example, when a telephone or a message is received while the user is viewing a moving picture through the main display area 10, the processor 130 displays a telephone or a message (e.g., a message) in the sub display area 20 or the round display area 30 Can be displayed.

Meanwhile, the processor 130 may change the display area when the direction of the user terminal device 100 is changed. For example, when the user views the sub display area 20 by changing the direction of the user terminal device 100 while checking the content of the text message through the main display area 10, 20 to display the content of the text message.

As described above, the processor 130 can determine that the direction of the user terminal device 100 has been changed by using various sensors or using the camera 120. The detailed description of the direction change of the user terminal device 100 is redundant and thus omitted.

The processor 130 may change the UI when the direction of the user terminal device 100 is changed to change the display area. Specifically, the processor 130 may change at least one of the amount of information, the type of information, and the layout included in the UI. For example, when displaying a text message on the main display area 10, the processor 130 can additionally display not only the contents of the text message, but also a UI, counterpart information, etc. for creating a new text message, When displaying in the sub display area 20, only the contents of the text message can be displayed. In addition, the processor 130 may sequentially display a part of the text message when the text message is displayed in the round display area 30.

In particular, the processor 130 may change the functionality provided by the display area. For example, when displaying the received text message on the main display area 10, the processor 130 may display a UI including a reply message input function, a received message storage function, a received message deletion function, When the displayed text message is displayed in the round display area 30, only the received text message content may be displayed, and no other function may be provided. FIGS. 18A and 18B are views for explaining an example of the configuration of the display.

18A shows a flexible display. The user can fold and use the flexible display. In this case, the folded and inverted area 1810 corresponds to the sub display area 20, and the unfolded area 1820 corresponds to the main display area 10. [

18A, the area of the inverted area 1810 is shown to be smaller than the area of the unfolded area 1820, but is not limited thereto. For example, the user may fold the flexible display exactly halfway.

Except for the round display area 30 of the above-described embodiments, most embodiments are applicable to the flexible display and are redundant descriptions, so will be omitted. However, in the case where the round display region 30 is formed by folding the flexible display twice, the above-described embodiment for the round display region 30 is also applicable.

18B shows a user terminal 100 in which a plurality of displays are provided back and forth. In this case, the front display 1830 corresponds to the main display area 10, and the rear display 1840 corresponds to the sub display area 20. [ Likewise, most of the embodiments are applicable to the user terminal device 100 in which a plurality of displays are provided back and forth, except for the round display area 30 in the above-described embodiments, and are redundant explanations.

19 is a flowchart illustrating a method of controlling a user terminal according to an exemplary embodiment of the present invention.

First, a live view obtained through a camera is displayed in one of a main display area disposed on the front side of the user terminal and a sub display area extending from one side of the main display area and disposed in at least one area on the back side of the user terminal (S1910). When the orientation of the user terminal device is changed, the live view is displayed on the other of the main display area and the sub display area (S1920).

Here, the step of displaying (S1910) in one of the main display region and the sub display region displays the live view in the main display region, does not provide information in the sub display region, , The live view may be displayed in the sub display area and the information may not be provided in the main display area if the direction of the user terminal device is changed.

On the other hand, in step S1420 of displaying the main display area and the sub display area in another one, the live view is displayed in the sub display area and information is not provided in the main display area. When a user touch is detected in the main display area, And photographing and storing the image in response to the touch.

Here, if the sweeping interaction in a predetermined direction is detected from any one point in the main display area, the step of changing the shooting setting value may correspond to a predetermined direction.

On the other hand, the step S1920 of displaying the live view on the other of the main display region and the sub display region displays the live view on the main display region. If the face of the person is recognized as being included in the live view, In the sub display area.

Here, the step of displaying in the sub display area may crop one area of the live view so as to correspond to the size of the sub display area and display it in the sub display area.

In addition, displaying the live view on the other of the main display area and the sub display area (S1920) may display the live view on the main display area, and displaying the animation on the sub display area when the live view is recognized as having a person .

On the other hand, the step S1910 of displaying one of the main display region and the sub display region determines the distance to the subject, displays the live view in the sub display region if the determined distance is smaller than the preset distance, If the distance is greater than the preset distance, the live view can be displayed in the main display area.

The method may further include detecting at least one of a position, a motion, and a user grip of the user terminal, and displaying the other one of the main display region and the sub display region (S1920) And a grip of the user, based on at least one of the grip and the grip of the user.

In addition, displaying (S1920) in the other of the main display area and the sub display area may be performed in a state in which a human face of a predetermined size or larger is recognized in the live view, If it is not recognized, it can be determined that the direction of the user terminal device has been changed.

According to various embodiments of the present invention as described above, the user terminal device can easily perform the shooting by controlling the display disposed on the front and rear sides of the user terminal device based on various shooting situations.

While the above description has focused on the case where the photographing function is executed, the present invention is not limited thereto. For example, in a state where a lock screen is displayed in the main display area, the processor may execute an application previously set when the direction of the user terminal device is changed. Alternatively, the processor can connect the telephone when the direction of the user terminal is changed while the UI in which the telephone is being received is displayed in the main display area. The processor may display a contact or the like when the user terminal is out of the ear during a call.

Alternatively, the processor may display an execution state of an application executed in the background in the sub display area. For example, the processor can display the execution status of a specific game or music application in the sub display area to minimize power consumption. Alternatively, in a state where a specific application is executed and displayed in the main display area, the processor may display a UI displaying essential functions of the application in the sub display area when the direction of the user terminal is changed.

In the above description, only the direction of the user terminal is changed. However, other operations may be performed according to the changed direction. For example, different functions may be performed for each rotation of the user terminal device in the upward direction, the downward direction, the rightward direction, and the leftward direction. Alternatively, even if the user terminal device rotates in the upward direction, other functions may be performed depending on whether the user terminal device is clockwise or counterclockwise.

The display device includes a main display area disposed on a front surface of a user terminal device according to various embodiments of the present invention and a sub display area extending from one side of the main display area, And a camera for capturing an image may be implemented in a computer-executable program code and stored in various non-transitory computer readable media such that each server Or devices.

For example, displaying the live view acquired through the camera in one of the main display area and the sub display area and displaying the live view on the other one of the main display area and the sub display area when the orientation of the user terminal device is changed A non-transitory computer readable medium may be provided in which a program for sequentially performing the displaying steps is stored.

A non-transitory readable medium is a medium that stores data for a short period of time, such as a register, cache, memory, etc., but semi-permanently stores data and is readable by the apparatus. In particular, the various applications or programs described above may be stored on non-volatile readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM,

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

100: user terminal device 110: display
10: main display area 20: sub display area
30: Round display area 120: Camera
130: Processor 140: Storage
145: GPS chip 150:
155: user interface unit 160: audio processing unit
170: video processor 180: speaker
181: button 182: microphone

Claims (20)

In a user terminal,
A display configured of a main display area disposed on a front surface of the user terminal device and a sub display area extending from one side of the main display area and disposed in at least one area of a rear surface of the user terminal device;
A camera for capturing an image; And
Displaying the live view acquired through the camera in one of the main display area and the sub display area, and when the orientation of the user terminal is changed, the live view is displayed in the main display area and the sub display area And control the display to display the other one. The method according to claim 1,
The processor comprising:
Wherein the live view is displayed in the main display area and information is not provided in the sub display area, and when the direction of the user terminal is changed, the live view is displayed in the sub display area, Wherein the user terminal device does not provide the service. The method according to claim 1,
The processor comprising:
Wherein the sub-display area displays the live view and does not provide information to the main display area,
Wherein the controller captures and stores an image in response to the user touch when the user touch is detected in the main display area. The method of claim 3,
The processor comprising:
And changes an imaging setting value to correspond to the predetermined direction when a swipe interaction in a predetermined direction is sensed from any one point in the main display area. The method according to claim 1,
The processor comprising:
And controls the display to display the face area included in the live view on the sub display area when the live view is recognized as including a human face in the live view. . 6. The method of claim 5,
The processor comprising:
And controls the display to crop one area of the live view so as to correspond to the size of the sub display area and display the cropped image in the sub display area. The method according to claim 1,
The processor comprising:
Controls the display to display the live view in the main display area and to display an animation in the sub display area if it is recognized that the live view is included in the live view. The method according to claim 1,
The processor comprising:
And displays the live view on the sub display area if the determined distance is less than the preset distance, and if the determined distance is greater than the preset distance, the live view is displayed on the main display area And controls the display to display the display. The method according to claim 1,
Further comprising: a plurality of sensors,
The processor comprising:
Wherein the controller determines that the direction of the user terminal is changed based on at least one of a position, a motion, and a grip of the user terminal detected by the plurality of sensors. The method according to claim 1,
The processor comprising:
If it is determined that the face of the user is not recognized in a state where a face of a person of a predetermined size or larger is recognized in the live view or a face of a person of the predetermined size or more is recognized, And the user terminal device. And a sub display area extending from one side of the main display area and disposed in at least one area of the rear surface of the user terminal device, A method of controlling a terminal apparatus,
Displaying a live view obtained through the camera in one of the main display area and the sub display area; And
And displaying the live view on the other of the main display area and the sub display area when the orientation of the user terminal device is changed. 12. The method of claim 11,
Wherein the displaying in one of the main display region and the sub display region comprises:
Wherein the main display area displays the live view and does not provide information to the sub display area,
Wherein the displaying on the other of the main display area and the sub display area comprises:
Wherein when the direction of the user terminal device is changed, the live view is displayed in the sub display area and information is not provided in the main display area. 12. The method of claim 11,
Wherein the displaying on the other of the main display area and the sub display area comprises:
Wherein the sub-display area displays the live view and does not provide information to the main display area,
Further comprising photographing and storing an image in response to the user touch when the user touch is detected in the main display area. 14. The method of claim 13,
Further comprising changing a photographing setting value to correspond to the predetermined direction when a sweeping interaction in a predetermined direction is sensed from any one point in the main display area. 12. The method of claim 11,
Wherein the displaying on the other of the main display area and the sub display area comprises:
Displays the live view in the main display area,
And displaying the face area included in the live view in the sub display area if it is recognized that the face of the person is included in the live view. 16. The method of claim 15,
Wherein the displaying in the sub display area comprises:
Wherein one area of the live view is cropped so as to correspond to the size of the sub display area and displayed in the sub display area. 12. The method of claim 11,
Wherein the displaying on the other of the main display area and the sub display area comprises:
Displays the live view in the main display area,
And displaying an animation in the sub display area if it is recognized that the live view includes a person. 12. The method of claim 11,
Wherein the displaying in one of the main display region and the sub display region comprises:
And displays the live view on the sub display area if the determined distance is less than the preset distance, and if the determined distance is greater than the preset distance, the live view is displayed on the main display area And displaying the image. 12. The method of claim 11,
Detecting at least one of a position, a motion of the user terminal, and a grip of the user;
Wherein the displaying on the other of the main display area and the sub display area comprises:
Wherein the control unit determines that the direction of the user terminal is changed based on at least one of a position, a motion, and a grip of the user terminal. 12. The method of claim 11,
Wherein the displaying on the other of the main display area and the sub display area comprises:
If it is determined that the face of the user is not recognized in a state where a face of a person of a predetermined size or larger is recognized in the live view or a face of a person of the predetermined size or more is recognized, Wherein the control method comprises:

Images