WO2024019190A1

WO2024019190A1 - Display device and operation method thereof

Info

Publication number: WO2024019190A1
Application number: PCT/KR2022/010631
Authority: WO
Inventors: 민정은
Original assignee: 엘지전자 주식회사
Priority date: 2022-07-20
Filing date: 2022-07-20
Publication date: 2024-01-25

Abstract

A display device according to an embodiment of the present disclosure may comprise: a wireless communication interface that receives an image signal corresponding to an image which a mobile terminal is playing back; a display that displays a mirroring image on the basis of the received image signal; and a controller that determines whether a touchpad mode event is detected, and when the touchpad mode event is detected, transmits touchpad mode information for switching the operation mode of the mobile terminal to the touchpad mode to the mobile terminal through the wireless communication interface.

Description

Display device and method of operation thereof

This disclosure relates to a display device that provides a screen mirroring service.

Digital TV services using wired or wireless communication networks are becoming popular. Digital TV services can provide a variety of services that cannot be provided by existing analog broadcasting services.

For example, IPTV (Internet Protocol Television), a type of digital TV service, and smart TV service provide interactivity that allows users to actively select the type of program to watch and the viewing time. IPTV and smart TV services can provide various additional services, such as Internet search, home shopping, and online games, based on this interactivity.

Additionally, recently, a multi-view function has been provided that runs two apps simultaneously and provides images of each app. In the multi-view function, a broadcast video can be displayed on one screen, and a mirrored video from a smart phone or smart pad can be displayed on another screen.

However, conventionally, when receiving a notification or performing other operations on a smart phone or smart pad, the user must hold and look at the smart phone or smart pad to check the notification or perform the necessary operations.

In this case, there is a problem that interferes with the viewing of broadcast images such as sports games or movies.

When providing the screen mirroring service of the present disclosure, the purpose is to be able to utilize the screen of the mobile terminal transmitting the source video as a touch pad.

The purpose of the present disclosure is to provide a method for performing notifications or operations on a mobile terminal that provides a mirroring service without other interference while watching a broadcast video when executing the multi-view function.

A display device according to an embodiment of the present disclosure includes a wireless communication interface for receiving an image signal corresponding to an image being played by a mobile terminal, a display for displaying a mirrored image based on the received image signal, and a touch pad mode for detecting whether an event is detected. It may include a controller that determines and, when the touch pad mode event is detected, transmits touch pad mode information to the mobile terminal through the wireless communication interface for switching the operation mode of the mobile terminal to the touch pad mode. .

A method of operating a display device according to an embodiment of the present disclosure includes receiving an image signal corresponding to an image being played by a mobile terminal, displaying a mirrored image based on the received image signal, and determining whether a touch pad mode event is detected. It may include determining and, when the touch pad mode event is detected, transmitting touch pad mode information to the mobile terminal for switching the operation mode of the mobile terminal to the touch pad mode.

According to an embodiment of the present disclosure, a user can operate a mobile terminal while watching a broadcast video in a mirroring state or a multi-view state without looking at the screen of the mobile terminal.

Accordingly, users can enjoy an improved user experience by being able to operate the mobile terminal without being interrupted while watching broadcasts.

Figure 1 shows a block diagram of the configuration of a display device according to an embodiment of the present invention.

Figure 2 is a block diagram of a remote control device according to an embodiment of the present invention.

Figure 3 shows an example of the actual configuration of a remote control device according to an embodiment of the present invention.

Figure 4 shows an example of utilizing a remote control device according to an embodiment of the present invention.

Figure 5 is a block diagram explaining the configuration of a mobile terminal according to an embodiment of the present invention.

Figure 6 is a sequence diagram for explaining a method of operating a system according to an embodiment of the present disclosure.

FIG. 7 is a diagram illustrating an example of displaying a mirrored image of a mobile terminal in a multi-view state according to an embodiment of the present disclosure.

Figure 8 is a diagram explaining touch pad mode information according to an embodiment of the present disclosure.

Figure 9 shows an example of touch pad mode release information according to an embodiment of the present disclosure.

FIG. 10 is a diagram illustrating a process for switching the operation mode of a mobile terminal according to an embodiment of the present disclosure.

FIG. 11 is a diagram illustrating a process for switching the operation mode of a mobile terminal according to another embodiment of the present disclosure.

12A and 12B are diagrams illustrating a process in which a mobile terminal operates in a touch pad mode and a process in which a mobile terminal switches from the touch pad mode to the mirroring mode according to an embodiment of the present disclosure.

FIGS. 13A and 13B are diagrams illustrating that the operation mode of a mobile terminal is changed by selecting either a content image or a casting image in a multi-view state according to an embodiment of the present disclosure.

FIG. 14 is a diagram illustrating a menu screen for setting an operation mode of a mobile terminal according to an embodiment of the present disclosure.

Hereinafter, embodiments related to the present invention will be described in more detail with reference to the drawings. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves.

The display device according to an embodiment of the present invention is, for example, an intelligent display device that adds a computer support function to the broadcast reception function, and is faithful to the broadcast reception function while adding an Internet function, etc., such as a handwriting input device and a touch screen. Alternatively, it can be equipped with a more convenient interface such as a spatial remote control. In addition, by supporting wired or wireless Internet functions, it is possible to connect to the Internet and a computer and perform functions such as email, web browsing, banking, or gaming. A standardized general-purpose OS can be used for these various functions.

Accordingly, in the display device described in the present invention, for example, various applications can be freely added or deleted on a general-purpose OS kernel, so various user-friendly functions can be performed. More specifically, the display device may be, for example, a network TV, HBBTV, smart TV, LED TV, OLED TV, etc., and in some cases, may also be applied to a smartphone.

Referring to FIG. 1, the display device 100 includes a broadcast receiver 130, an external device interface 135, a memory 140, a user input interface 150, a controller 170, a wireless communication interface 173, and a display. It may include (180), a speaker (185), and a power supply circuit (190).

The broadcast receiver 130 may include a tuner 131, a demodulator 132, and a network interface 133.

The tuner 131 can select a specific broadcast channel according to a channel selection command. The tuner 131 may receive a broadcast signal for a specific selected broadcast channel.

The demodulator 132 can separate the received broadcast signal into a video signal, an audio signal, and a data signal related to the broadcast program, and can restore the separated video signal, audio signal, and data signal to a form that can be output.

The external device interface 135 may receive an application or application list within an adjacent external device and transfer it to the controller 170 or memory 140.

The external device interface 135 may provide a connection path between the display device 100 and an external device. The external device interface 135 may receive one or more of video and audio output from an external device connected wirelessly or wired to the display device 100 and transmit it to the controller 170. The external device interface 135 may include a plurality of external input terminals. The plurality of external input terminals may include an RGB terminal, one or more High Definition Multimedia Interface (HDMI) terminals, and a component terminal.

An image signal from an external device input through the external device interface 135 may be output through the display 180. A voice signal from an external device input through the external device interface 135 may be output through the speaker 185.

An external device that can be connected to the external device interface 135 may be any one of a set-top box, Blu-ray player, DVD player, game console, sound bar, smartphone, PC, USB memory, or home theater, but this is only an example.

The network interface 133 may provide an interface for connecting the display device 100 to a wired/wireless network including an Internet network. The network interface 133 may transmit or receive data with other users or other electronic devices through a connected network or another network linked to the connected network.

Additionally, some of the content data stored in the display device 100 may be transmitted to a selected user or selected electronic device among other users or other electronic devices pre-registered in the display device 100.

The network interface 133 can access a certain web page through a connected network or another network linked to the connected network. In other words, you can access a certain web page through a network and transmit or receive data with the corresponding server.

And, the network interface 133 can receive content or data provided by a content provider or network operator. That is, the network interface 133 can receive content and information related thereto, such as movies, advertisements, games, VODs, and broadcast signals, provided from a content provider or network provider through a network.

Additionally, the network interface 133 can receive firmware update information and update files provided by a network operator, and can transmit data to the Internet, a content provider, or a network operator.

The network interface 133 can select and receive a desired application from among applications open to the public through a network.

The memory 140 stores programs for processing and controlling each signal in the controller 170, and can store signal-processed video, voice, or data signals.

In addition, the memory 140 may perform a function for temporary storage of video, voice, or data signals input from the external device interface 135 or the network interface 133, and may store information about a predetermined image through a channel memory function. You can also store information.

The memory 140 may store an application or application list input from the external device interface 135 or the network interface 133.

The display device 100 can play content files (video files, still image files, music files, document files, application files, etc.) stored in the memory 140 and provide them to the user.

The user input interface 150 may transmit a signal input by the user to the controller 170 or transmit a signal from the controller 170 to the user. For example, the user input interface 150 can be used remotely according to various communication methods such as Bluetooth, Ultra Wideband (WB), ZigBee, Radio Frequency (RF) communication, or infrared (IR) communication. Control signals such as power on/off, channel selection, and screen settings can be received and processed from the control device 200, or control signals from the controller 170 can be processed to be transmitted to the remote control device 200.

Additionally, the user input interface 150 can transmit control signals input from local keys (not shown) such as power key, channel key, volume key, and setting value to the controller 170.

The video signal processed by the controller 170 may be input to the display 180 and displayed as an image corresponding to the video signal. Additionally, the image signal processed by the controller 170 may be input to an external output device through the external device interface 135.

The voice signal processed by the controller 170 may be output as audio to the speaker 185. Additionally, the voice signal processed by the controller 170 may be input to an external output device through the external device interface 135.

In addition, the controller 170 may control overall operations within the display device 100.

In addition, the controller 170 can control the display device 100 by a user command or internal program input through the user input interface 150, and connects to the network to display the application or application list desired by the user on the display device ( 100) You can make it available for download.

The controller 170 allows channel information selected by the user to be output through the display 180 or speaker 185 along with the processed video or audio signal.

In addition, the controller 170 controls video signals from an external device, for example, a camera or camcorder, input through the external device interface 135, according to an external device video playback command received through the user input interface 150. Alternatively, the voice signal can be output through the display 180 or speaker 185.

Meanwhile, the controller 170 can control the display 180 to display an image, for example, a broadcast image input through the tuner 131, an external input image input through the external device interface 135, Alternatively, an image input through the network interface unit or an image stored in the memory 140 may be controlled to be displayed on the display 180. In this case, the image displayed on the display 180 may be a still image or a moving image, and may be a 2D image or a 3D image.

In addition, the controller 170 can control the playback of content stored in the display device 100, received broadcast content, or external input content, which includes broadcast video, external input video, and audio files. , can be in various forms such as still images, connected web screens, and document files.

The wireless communication interface 173 can communicate with external devices through wired or wireless communication. The wireless communication interface 173 can perform short range communication with an external device. For this purpose, the wireless communication interface 173 includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), and Wi-Fi. Short-distance communication can be supported using at least one of Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. This wireless communication interface 173 is between the display device 100 and a wireless communication system, between the display device 100 and another display device 100, or between the display device 100 through wireless area networks. It can support wireless communication between the network and the display device 100 (or external server). Local area wireless networks may be wireless personal area networks.

Here, the other display device 100 is a wearable device capable of exchanging data with (or interoperating with) the display device 100 according to the present invention, for example, a smartwatch, smart glasses. It can be a mobile terminal such as (smart glass), HMD (head mounted display), or smart phone. The wireless communication interface 173 may detect (or recognize) a wearable device capable of communication around the display device 100 .

Furthermore, if the detected wearable device is a device authenticated to communicate with the display device 100 according to the present invention, the controller 170 sends at least a portion of the data processed by the display device 100 to the wireless communication interface 173. It can be transmitted to a wearable device through . Accordingly, a user of a wearable device can use data processed by the display device 100 through the wearable device.

The display 180 converts the video signal, data signal, and OSD signal processed by the controller 170 or the video signal and data signal received from the external device interface 135 into R, G, and B signals, respectively, and provides a driving signal. can be created.

Meanwhile, the display device 100 shown in FIG. 1 is only an embodiment of the present invention. Some of the illustrated components may be integrated, added, or omitted depending on the specifications of the display device 100 that is actually implemented.

That is, as needed, two or more components may be combined into one component, or one component may be subdivided into two or more components. In addition, the functions performed by each block are for explaining embodiments of the present invention, and the specific operations or devices do not limit the scope of the present invention.

According to another embodiment of the present invention, unlike shown in FIG. 1, the display device 100 does not have a tuner 131 and a demodulator 132 but has a network interface 133 or an external device interface 135. You can also receive and play video through the device.

For example, the display device 100 is divided into an image processing device such as a set-top box for receiving broadcast signals or contents according to various network services, and a content playback device for playing content input from the image processing device. It can be implemented.

In this case, the method of operating a display device according to an embodiment of the present invention to be described below includes not only the display device 100 as described with reference to FIG. 1, but also an image processing device such as the separated set-top box or the display 180. ) and a content playback device having an audio output unit 185.

Next, with reference to FIGS. 2 and 3, a remote control device according to an embodiment of the present invention will be described.

Figure 2 is a block diagram of a remote control device according to an embodiment of the present invention, and Figure 3 shows an example of the actual configuration of the remote control device 200 according to an embodiment of the present invention.

First, referring to FIG. 2, the remote control device 200 includes a fingerprint reader 210, a wireless communication circuit 220, a user input interface 230, a sensor 240, an output interface 250, and a power supply circuit ( 260), memory 270, controller 280, and microphone 290.

Referring to FIG. 2, the wireless communication circuit 220 transmits and receives signals to and from any one of the display devices according to the embodiments of the present invention described above.

The remote control device 200 has an RF circuit 221 capable of transmitting and receiving signals to and from the display device 100 in accordance with RF communication standards, and is capable of transmitting and receiving signals to and from the display device 100 in accordance with IR communication standards. An IR circuit 223 may be provided. Additionally, the remote control device 200 may be provided with a Bluetooth circuit 225 capable of transmitting and receiving signals to and from the display device 100 according to the Bluetooth communication standard. In addition, the remote control device 200 is provided with an NFC circuit 227 capable of transmitting and receiving signals with the display device 100 according to the NFC (Near Field Communication) communication standard, and displays the display device 100 according to the WLAN (Wireless LAN) communication standard. A WLAN circuit 229 capable of transmitting and receiving signals to and from the device 100 may be provided.

In addition, the remote control device 200 transmits a signal containing information about the movement of the remote control device 200 to the display device 100 through the wireless communication circuit 220.

Meanwhile, the remote control device 200 can receive a signal transmitted by the display device 100 through the RF circuit 221 and, if necessary, turn on/off the display device 100 through the IR circuit 223. Commands for turning off, changing channels, changing volume, etc. can be sent.

The user input interface 230 may be comprised of a keypad, button, touch pad, or touch screen. The user can input commands related to the display device 100 into the remote control device 200 by manipulating the user input interface 230. If the user input interface 230 has a hard key button, the user can input a command related to the display device 100 to the remote control device 200 through a push operation of the hard key button. This will be explained with reference to FIG. 3 .

Referring to FIG. 3, the remote control device 200 may include a plurality of buttons. The plurality of buttons include a fingerprint recognition button (212), power button (231), home button (232), live button (233), external input button (234), volume control button (235), voice recognition button (236), It may include a channel change button 237, a confirmation button 238, and a back button 239.

The fingerprint recognition button 212 may be a button for recognizing the user's fingerprint. In one embodiment, the fingerprint recognition button 212 is capable of a push operation and may receive a push operation and a fingerprint recognition operation.

The power button 231 may be a button for turning on/off the power of the display device 100.

The home button 232 may be a button for moving to the home screen of the display device 100.

The live button 233 may be a button for displaying a real-time broadcast program.

The external input button 234 may be a button for receiving an external input connected to the display device 100.

The volume control button 235 may be a button for adjusting the volume of sound output by the display device 100.

The voice recognition button 236 may be a button for receiving the user's voice and recognizing the received voice.

The channel change button 237 may be a button for receiving a broadcast signal of a specific broadcast channel.

The confirmation button 238 may be a button for selecting a specific function, and the back button 239 may be a button for returning to the previous screen.

Figure 2 will be described again.

If the user input interface 230 has a touch screen, the user can input commands related to the display device 100 through the remote control device 200 by touching a soft key on the touch screen. Additionally, the user input interface 230 may be provided with various types of input means that the user can operate, such as scroll keys and jog keys, and this embodiment does not limit the scope of the present invention.

The sensor 240 may include a gyro sensor 241 or an acceleration sensor 243, and the gyro sensor 241 may sense information about the movement of the remote control device 200.

For example, the gyro sensor 241 can sense information about the operation of the remote control device 200 based on the x, y, and z axes, and the acceleration sensor 243 measures the moving speed of the remote control device 200. Information about such things can be sensed. Meanwhile, the remote control device 200 may further include a distance measurement sensor and can sense the distance from the display 180 of the display device 100.

The output interface 250 may output a video or audio signal corresponding to a manipulation of the user input interface 230 or a signal transmitted from the display device 100.

The user can recognize whether the output interface 250 is manipulating the user input interface 230 or controlling the display device 100.

For example, the output interface 250 includes an LED 251 that turns on when the user input interface 230 is manipulated or a signal is transmitted and received with the display device 100 through the wireless communication unit 225, and a vibrator 253 that generates vibration. ), a speaker 255 that outputs sound, or a display 257 that outputs an image.

In addition, the power supply circuit 260 supplies power to the remote control device 200, and stops power supply when the remote control device 200 does not move for a predetermined period of time, thereby reducing power waste.

The power supply circuit 260 can resume power supply when a predetermined key provided in the remote control device 200 is operated.

The memory 270 may store various types of programs, application data, etc. necessary for controlling or operating the remote control device 200.

When the remote control device 200 transmits and receives signals wirelessly through the display device 100 and the RF circuit 221, the remote control device 200 and the display device 100 transmit and receive signals through a predetermined frequency band. .

The controller 280 of the remote control device 200 stores and references information about the display device 100 paired with the remote control device 200 and the frequency band capable of wirelessly transmitting and receiving signals in the memory 270. You can.

The controller 280 controls all matters related to the control of the remote control device 200. The controller 280 sends a signal corresponding to a predetermined key operation of the user input interface 230 or a signal corresponding to the movement of the remote control device 200 sensed by the sensor 240 through the wireless communication unit 225. 100).

Additionally, the microphone 290 of the remote control device 200 can acquire voice.

There may be a plurality of microphones 290.

Next, Figure 4 will be described.

(a) of FIG. 4 illustrates that a pointer 205 corresponding to the remote control device 200 is displayed on the display 180.

The user can move or rotate the remote control device 200 up and down, left and right. The pointer 205 displayed on the display 180 of the display device 100 corresponds to the movement of the remote control device 200. This remote control device 200 can be called a spatial remote control because the corresponding pointer 205 is moved and displayed according to movement in 3D space, as shown in the drawing.

(b) of FIG. 4 illustrates that when the user moves the remote control device 200 to the left, the pointer 205 displayed on the display 180 of the display device 100 also moves to the left correspondingly.

Information about the movement of the remote control device 200 detected through the sensor of the remote control device 200 is transmitted to the display device 100. The display device 100 may calculate the coordinates of the pointer 205 from information about the movement of the remote control device 200. The display device 100 may display the pointer 205 to correspond to the calculated coordinates.

(c) of FIG. 4 illustrates a case where a user moves the remote control device 200 away from the display 180 while pressing a specific button in the remote control device 200. As a result, the selected area in the display 180 corresponding to the pointer 205 can be zoomed in and displayed enlarged.

Conversely, when the user moves the remote control device 200 closer to the display 180, the selected area in the display 180 corresponding to the pointer 205 may be zoomed out and displayed in a reduced size.

Meanwhile, when the remote control device 200 moves away from the display 180, the selected area may be zoomed out, and when the remote control device 200 approaches the display 180, the selected area may be zoomed in.

Additionally, when a specific button in the remote control device 200 is pressed, recognition of up-down, left-right movement may be excluded. That is, when the remote control device 200 moves away from or approaches the display 180, up, down, left, and right movements may not be recognized, and only forward and backward movements may be recognized. When a specific button in the remote control device 200 is not pressed, only the pointer 205 moves as the remote control device 200 moves up, down, left, and right.

Meanwhile, the moving speed or direction of the pointer 205 may correspond to the moving speed or direction of the remote control device 200.

Meanwhile, a pointer in this specification refers to an object displayed on the display 180 in response to the operation of the remote control device 200. Accordingly, the pointer 205 can be an object of various shapes other than the arrow shape shown in the drawing. For example, concepts may include dots, cursors, prompts, thick outlines, etc. In addition, the pointer 205 can be displayed not only in response to one of the horizontal and vertical axes on the display 180, but also in response to multiple points, such as a line or surface. .

The mobile terminal 500 includes a projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, personal digital assistants (PDA), portable multimedia player (PMP), navigation, tablet PC, wearable device, set-top box (STB), It can be implemented as a fixed device or a movable device, such as a DMB receiver, radio, or desktop computer.

Referring to FIG. 5 , the mobile terminal 500 may include a communication circuit 510, an input unit 520, a memory 530, a display 540, and a processor 590.

The communication circuit 510 can transmit and receive data with external devices such as other mobile terminals or servers using wired or wireless communication technology.

The communication circuit 510 includes GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), and Bluetooth. Communication can be performed using any one of the following communication standards: RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, and NFC (Near Field Communication).

The input unit 520 may include a camera for inputting video signals, a microphone for receiving audio signals, and a user input unit for receiving information from the user.

Here, the camera or microphone may be treated as a sensor, and the signal obtained from the camera or microphone may be referred to as sensing data or sensor information.

The memory 530 can store various software and data related to the operation of the mobile terminal 500.

The display 540 can display an image signal received from the outside.

The processor 590 can control the overall operation of the mobile terminal 500.

If linkage with an external device is necessary to perform an operation of the mobile terminal 500, the processor 590 may generate a control signal to control the external device and transmit the generated control signal to the external device.

The processor 590 may obtain intent information regarding user input and determine the user's request based on the obtained intent information.

The processor 590 may control at least some of the components of the mobile terminal 500 to run an application program stored in the memory 170.

The processor 590 may operate by combining two or more of the components included in the mobile terminal 500 to run the application program.

A system according to an embodiment of the present disclosure may include the display device 100 shown in FIG. 1 and the mobile terminal 500 shown in FIG. 5.

The display device 100 and the mobile terminal 500 may be connected to each other through a short-range wireless communication standard. The short-range wireless communication standard may be either the Wi-Fi standard or the Wi-Fi Direct standard.

이동 단말기(500)의 프로세서(590)는 디스플레이(540)에 표시 중인 영상에 상응하는 미러링 영상 신호를 통신 회로(510)를 통해 디스플레이 장치(100)에 전송한다(S601).The processor 590 of the mobile terminal 500 transmits a mirroring image signal corresponding to the image being displayed on the display 540 to the display device 100 through the communication circuit 510 (S601).

The communication circuit 510 may include a Wi-Fi circuit that transmits a mirroring image signal through the Wi-Fi standard.

디스플레이 장치(100)의 컨트롤러(170)는 수신된 미러링 영상 신호에 기초한 미러링 영상을 디스플레이(180) 상에 표시한다(S603).The controller 170 of the display device 100 displays a mirrored image based on the received mirrored image signal on the display 180 (S603).

The controller 170 can convert the mirroring image signal to match the resolution of its display 180 and display the converted mirroring image on the display 180.

In one embodiment, the controller 170 may display the mirrored image on the entire screen of the display 180. In some cases, a black image may be displayed on either left/right/top/bottom to match the resolution.

In another embodiment, the display device 100 may be in a state in which a multi-view function is being executed. The multi-view function may be a function of displaying a plurality of images through each of a plurality of split screens. Each split screen may be an execution screen of an application.

When the display device 100 is executing the multi-view function, the controller 170 can display the content image and the mirroring image on each of the split screens.

In one embodiment, the content image may be a broadcast image resulting from execution of a live broadcast application.

The broadcast video may be video received from a set-top box connected to the external device interface 135.

Referring to FIG. 7, the display device 100 and the mobile terminal 500 are connected to each other through the Wi-Fi standard or the Wi-Fi Direct standard.

The mobile terminal 500 may be displaying the home screen image 700 on the display 540 and transmitting a mirroring image signal corresponding to the home screen image 700 to the display device 100.

The display device 100 displays a content image 711 on the first split screen 710 according to execution of the multi-view function, and displays a mirrored image 731 based on the mirroring image signal on the second split screen 730. can be displayed.

Meanwhile, the user of the mobile terminal 500 may apply a touch input to a point 701 of the display 540. The mobile terminal 500 may transmit coordinate information about a point 701 to the display device 100. The display device 100 may display an indicator 733 corresponding to a point 701 on the second split screen 730 based on the received coordinate information.

Operation modes of the mobile terminal 500 may include mirroring mode and touch pad mode.

The mirroring mode may be a mode in which the mirroring image 730 can be controlled according to the user's touch input during mirroring. Under the mirroring mode, an indicator 733 corresponding to the user's touch input may be displayed on the mirroring image 730.

Under mirroring mode, the user's drag input may be reflected in the mirroring image 730 through the indicator 733.

The touch pad mode may be a mode in which the display 540 of the mobile terminal 500 is used as a touch pad during mirroring. Even under the touch pad mode, an indicator corresponding to the user's touch input may be displayed on the mirroring image 730.

Under touch pad mode, the user's drag input is ignored, and only touch down input or touch up input can be recognized.

Figure 7 shows an example of the mobile terminal 500 operating in mirroring mode.

Again, Figure 6 will be described.

디스플레이 장치(100)의 컨트롤러(170)는 미러링 영상의 표시 중 이동 단말기(500)의 터치 패드 모드 이벤트가 감지되었는지를 판단한다(S605).The controller 170 of the display device 100 determines whether a touch pad mode event of the mobile terminal 500 is detected while displaying the mirrored image (S605).

In one embodiment, the controller 170 may detect a touch pad mode event of the mobile terminal 500 based on information received from the mobile terminal 500.

The touch pad mode event may be an event for switching the operation mode of the mobile terminal 500 to the touch pad mode.

The controller 170 may detect a touch pad mode event of the mobile terminal 500 based on movement information received from the mobile terminal 500. When the mobile terminal 500 determines that there is no movement of the mobile terminal 500 for a certain period of time, it can transmit movement information indicating this to the display device 100.

The mobile terminal 500 may measure its movement through a gyro sensor or an acceleration sensor.

Based on the received motion information, the controller 170 may determine that there is no movement of the mobile terminal 500 for a certain period of time and determine that a touch pad mode event of the mobile terminal 500 has been detected.

In another embodiment, when multi-touch input information is received from the mobile terminal 500, the controller 170 may determine that a touch pad mode event of the mobile terminal 500 has been detected. When two or more touch inputs are maintained for a certain period of time on the display 540 of the mobile terminal 500 being mirrored, the mobile terminal 500 displays multi-touch input information indicating that a multi-touch input has been generated. can be transmitted to.

In another embodiment, the controller 170 may detect a touch pad mode event of the mobile terminal 500 through a pop-up window. When providing a mirroring service with the mobile terminal 500, the controller 170 may display a pop-up window on the display 180 that induces the mobile terminal 500 to operate in a touch pad mode. The pop-up window may be displayed after the mirroring service is provided.

When the confirmation button included in the pop-up window is selected, the controller 170 may determine that the touch pad mode of the mobile terminal 500 has been detected.

디스플레이 장치(100)의 컨트롤러(170)는 이동 단말기(500)의 터치 패드 모드 이벤트가 감지된 경우, 터치 패드 모드 정보를 무선 통신 인터페이스(173)를 통해 이동 단말기(500)에 전송한다(S607).When a touch pad mode event of the mobile terminal 500 is detected, the controller 170 of the display device 100 transmits touch pad mode information to the mobile terminal 500 through the wireless communication interface 173 (S607). .

In one embodiment, the touch pad mode information may include information for switching the operation mode of the mobile terminal 500 to the touch pad mode.

The touch pad mode information may be information used to switch the operation mode of the mobile terminal 500 from the mirroring mode to the touch pad mode.

The touch pad mode information may include a mode parameter for setting the operation mode of the mobile terminal 500 to the touch pad mode and a drag input parameter indicating whether drag input is possible.

The touch pad mode information may further include a touch input parameter indicating whether touch up/down input is possible.

Referring to FIG. 8, touch pad mode information 800 may include mode parameters, drag input parameters, and touch up/down input parameters.

The mode parameter may have a touch pad mode value that requests operation in touch pad mode.

The drag input parameter may have a disable value indicating that drag input is not possible (or ignored).

The touch up/down input parameter can have an enable value indicating that touch up/down input is possible.

Again, Figure 6 will be described.

이동 단말기(500)의 프로세서(590)는 수신된 터치 패드 모드 정보에 기반하여 이동 단말기(500)의 동작 모드를 터치 패드 모드로 동작 시킨다(S609).The processor 590 of the mobile terminal 500 operates the mobile terminal 500 in a touch pad mode based on the received touch pad mode information (S609).

The processor 590 of the mobile terminal 500 may change the operation mode of the mobile terminal 500 to the touch pad mode based on the value of the mode parameter included in the received touch pad mode information.

The mobile terminal 500 may ignore the user's drag input under the touch pad mode. The mobile terminal 500 can only recognize the user's touch up input and touch down input under the touch pad mode.

The user can manipulate the mirroring screen by only viewing the mirroring screen of the display device 100.

이동 단말기(500)의 프로세서(590)는 통신 회로(510)를 통해 터치 입력에 대한 터치 입력 정보를 디스플레이 장치(100)에 전송한다(S611).The processor 590 of the mobile terminal 500 transmits touch input information about the touch input to the display device 100 through the communication circuit 510 (S611).

In one embodiment, the touch input information may include a touch up input or a touch down input and coordinate information of a point where the corresponding input was received. Coordinate information may include location information of pixels included in the display 540 of the mobile terminal 500.

디스플레이 장치(100)의 컨트롤러(170)는 터치 입력 정보에 기반하여 이동 단말기(500)의 터치 패드 모드의 해제 이벤트가 감지되었는지를 판단한다(S613).The controller 170 of the display device 100 determines whether a release event of the touch pad mode of the mobile terminal 500 has been detected based on the touch input information (S613).

The controller 170 may select an item included in the mirroring image based on touch input information. Items included in the mirroring image may be either an app icon or a notification item corresponding to an application installed on the mobile terminal 500.

The touch pad mode release event may be an event for releasing the touch pad mode of the mobile terminal 500 and switching back to the mirroring mode.

When a specific event occurs based on the touch input information, the controller 170 may determine that the touch pad mode release event of the mobile terminal 500 has been detected.

In one embodiment, the specific event may be an event for selecting an application icon included in the mirroring image. The application may be an application installed on the mobile terminal 500.

That is, when the application icon included in the mirroring image is selected according to the touch input information, the controller 170 may determine that the touch pad mode release event of the mobile terminal 500 has been detected.

디스플레이 장치(100)의 컨트롤러(170)는 이동 단말기(500)의 터치 패드 모드의 해제 이벤트가 감지된 경우, 터치 패드 모드 해제 정보를 이동 단말기(500)에 전송한다(S615).When a touch pad mode release event of the mobile terminal 500 is detected, the controller 170 of the display device 100 transmits touch pad mode release information to the mobile terminal 500 (S615).

The touch pad mode release information may include information to release the touch pad mode of the mobile terminal 500.

The touch pad mode release information may include information for switching the operation mode of the mobile terminal 500 to the mirroring mode.

Referring to FIG. 9 , touch pad mode release information 900 may include mode parameters, drag input parameters, and touch up/down input parameters.

The mode parameter may have a mirroring mode value that requests operation in mirroring mode.

The drag input parameter can have an enable value indicating that drag input is possible.

Again, Figure 6 will be described.

이동 단말기(500)의 프로세서(590)는 디스플레이 장치(100)로부터 수신된 터치 패드 모드 해제 정보에 따라 동작 모드를 터치 패드 모드에서 미러링 모드로 전환한다(S617).The processor 590 of the mobile terminal 500 switches the operation mode from the touch pad mode to the mirroring mode according to the touch pad mode release information received from the display device 100 (S617).

In this case, the display device 100 may display the indicator 733 shown in FIG. 7 at a position corresponding to the received touch point on the screen of the mobile terminal 500.

Hereinafter, embodiments of the present disclosure will be described in more detail.

Referring to FIG. 10, the display device 100 and the mobile terminal 500 are connected to each other through the Wi-Fi standard or the Wi-Fi Direct standard.

In Figure 10, the mobile terminal 500 operates in mirroring mode.

Referring to FIG. 10 , the mobile terminal 500 may receive two long touch down/up inputs and transmit the received touch input information to the display device 100.

The display device 100 may determine that a touch pad mode event of the mobile terminal 500 has been detected based on two long touch down/up input information included in the received touch input information.

The display device 100 may transmit touch pad mode information indicating a switch to the touch pad mode to the mobile terminal 500, and the mobile terminal 500 may change the operation mode from the mirroring mode to the touch pad based on the touch pad mode information. You can switch to mode.

Additionally, when two long touch down/up inputs are received while the mobile terminal 500 is switched to the touch pad mode, the mobile terminal 500 can switch the touch pad mode back to the mirroring mode.

As such, according to an embodiment of the present disclosure, a user can easily switch the operation mode of the mobile terminal 500 with a simple multi-touch input. Accordingly, user convenience can be greatly improved.

Referring to FIG. 11, the display device 100 and the mobile terminal 500 are connected to each other through the Wi-Fi standard or the Wi-Fi Direct standard.

When providing a mirroring service with the mobile terminal 500, the display device 100 may display a pop-up window 1100 on the display 180 that induces the operation mode of the mobile terminal 500 to be switched to the touch pad mode. .

The pop-up window 1100 may be displayed on the first split screen 710 or the second split screen 730. As another example, the pop-up window 1100 may be displayed in an area other than the first split screen 710 or the second split screen 730.

When the display device 100 receives a command to select the confirmation button 1110 included in the pop-up window 1100, it may determine that a touch pad mode event has been detected. The display device 100 may transmit touch pad mode information to the mobile terminal 500 to change the operation mode of the mobile terminal 500 to the touch pad mode according to selection of the confirmation button 1110.

Referring to FIG. 12A, the display device 100 and the mobile terminal 500 are connected to each other through the Wi-Fi standard or the Wi-Fi Direct standard.

The display device 100 may transmit touch pad mode information to the mobile terminal 500 according to the occurrence of a touch pad mode event.

The mobile terminal 500 may switch the operation mode to the touch pad mode according to the received touch pad mode information. At the same time, the mobile terminal 500 may display a blank screen 1200 on which no app icons are displayed. This is to inform the user that the mobile terminal 500 is in touch pad mode. Additionally, this is to prevent visual interference from the user's touch input by displaying app icons.

The user may apply a touch down/up input on the blank screen 1200. That is, while watching the content image 710, the user can apply a touch down/up input on the display 540 of the mobile terminal 500 operating in touch pad mode.

The mobile terminal 500 may output a haptic signal when a touch down/up input is applied to a point where the icon of a specific app is located for a certain period of time.

The display device 100 may display an indicator 733 on the mirroring image 730 at a position corresponding to the point based on the touch down/up input information received from the mobile terminal 500. The indicator 733 may be displayed at a location corresponding to the user's touch point.

When a touchdown input is received from the mobile terminal 500 for more than a certain period of time, the display device 100 may display an indicator 733 at the corresponding location.

The display device 100 may select the message icon 735 corresponding to the message application where the indicator 733 is located based on the touch input information received from the mobile terminal 500.

The message application may be an application installed on the mobile terminal 500.

When a message application is executed, the display device 100 may determine that a touch pad mode release event has been detected.

That is, when the display device 100 detects execution of an application based on touch input information received from the mobile terminal 500, it may generate a touch pad mode release event.

The display device 100 may transmit touch pad mode release information to the mobile terminal 500.

The mobile terminal 500 may switch the operation mode to the mirroring mode based on the touch pad mode release information. As shown in FIG. 12B, the mobile terminal 500 may execute a message application corresponding to the message icon 735 and display an execution image 1210 on the display 540.

The mobile terminal 500 transmits an image signal corresponding to the execution image 1210 to the display device 100, and the display device 100 displays a mirroring image 1230 based on the received image signal as a second split screen ( 730).

In this way, the user can operate the mobile terminal while watching a broadcast video in a mirroring state or a multi-view state without looking at the screen of the mobile terminal.

Referring to FIG. 13A, the mobile terminal 500 may have its screen turned off or operate in a casting mode. The casting mode may be a mode similar to the mirroring mode described above. The difference from the mirroring mode is that the mobile terminal 500 does not display a mirrored image under the casting mode.

Casting mode may be a mode in which an image being played on the mobile terminal 500 is transmitted to the display device 100 and the display device 100 plays the image. Under casting mode, the screen of the mobile terminal 500 may be turned off.

The display device 100 may display the broadcast image 1301 on the first split screen 710 and the casting image 1303 on the second split screen 730 according to the execution of the multi-view function. . The casting image 1303 may be an image being played on the mobile terminal 500.

When receiving a command to select the first split screen 710, the display device 100 transmits touch pad mode information to the mobile terminal 500 to change the operation mode of the mobile terminal 500 from the casting mode to the touch pad mode. ) can be transmitted to.

The display device 100 may display the broadcast image 1301 on the entire screen according to a command for selecting the first split screen 710. The mobile terminal 500 may switch the casting mode to the touch pad mode according to the touch pad mode information received from the display device 100.

When the first split screen 710 is selected, the display device 100 may determine that the touch pad mode event is detected in the mobile terminal 500.

The mobile terminal 500 operating in touch pad mode can perform the same role as the remote control device 200.

Referring to FIG. 13B, unlike FIG. 13A, the display device 100 may receive a command to select the second split screen 730.

The display device 100 may display the casting image 1303 on the entire screen according to a command for selecting the second split screen 730.

In this case, the operation mode of the mobile terminal 500 may be maintained in casting mode. This is to manipulate the casting image 1303 since the casting image 1303 is being displayed on the entire screen of the display device 100.

Meanwhile, in FIGS. 13A and 13B, casting mode can be used instead of mirroring mode.

The display device 100 may display a menu screen 1400 for setting the operation mode of the mobile terminal 500 on the display 180 according to a command received from the remote control device 200.

The menu screen 1400 may include a first menu item 1410 for setting the mirroring mode and a second menu item 1430 for setting the touch pad mode.

When the mirroring mode is checked through the first menu item 1410, the display device 100 may transmit information to operate in the mirroring mode to the mobile terminal 500. The corresponding information may be touch pad mode release information in FIG. 6.

When the touch pad mode is checked through the second menu item 1430, the display device 100 may transmit information to operate in the touch pad mode to the mobile terminal 500. The corresponding information may be touch pad mode information in FIG. 6.

As such, according to the embodiment of the present disclosure, the user may easily set the operation mode of the mobile terminal 500 through the menu screen 1400.

According to an embodiment of the present disclosure, the above-described method can be implemented as processor-readable code on a program-recorded medium. Examples of media that the processor can read include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices.

The display device described above is not limited to the configuration and method of the above-described embodiments, and the embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made. It may be possible.

Claims

In the display device,

A wireless communication interface for receiving a video signal corresponding to the video being played by the mobile terminal;

a display that displays a mirrored image based on the received image signal; and

It determines whether a touch pad mode event is detected, and when the touch pad mode event is detected, touch pad mode information for switching the operation mode of the mobile terminal to the touch pad mode is sent to the mobile terminal through the wireless communication interface. Contains a controller that transmits

Display device.
According to paragraph 1,

The touch pad mode information is

Includes a mode parameter for setting the operation mode of the mobile terminal to the touch pad mode and a drag input parameter indicating the impossibility of drag input.

Display device.
According to paragraph 1,

The controller is

Based on information received from the mobile terminal, detecting the touch pad mode event

Display device.
According to paragraph 3,

The controller is

When motion information indicating that there is no movement of the mobile terminal for a certain period of time is received from the mobile terminal, it is determined that the touch pad mode event is detected.

Display device.
According to paragraph 3,

The controller is

When receiving multi-touch input information indicating that two or more touch inputs were maintained for a certain period of time from the mobile terminal, determining that the touch pad mode event has been detected

Display device.
According to paragraph 1,

The controller is

Displaying a pop-up window to induce operation in the touch pad mode on the display, and determining that the touch pad mode event is detected through the pop-up window

Display device.
According to paragraph 1,

The controller is

After the operation mode of the mobile terminal is set to the touch pad mode according to the touch pad mode information, touch input information is received from the mobile terminal,

Detecting a touch pad mode release event indicating release of the touch pad mode based on received touch input information

Display device.
In clause 7,

The controller is

When an app icon included in the mirroring image is selected based on the touch input information, it is determined that a release event of the touch pad mode is detected.

Display device.
According to clause 8,

The controller is

Transmitting touch pad mode release information indicating release of the touch pad mode to the mobile terminal

Display device.
According to clause 9,

The controller is

Receiving an execution image of an application corresponding to the app icon from the mobile terminal and displaying the received execution image on the display

Display device.
According to clause 9,

The touch pad mode release information is

Includes a mode parameter for setting the operation mode of the mobile terminal to mirroring mode and a drag input parameter indicating the possibility of drag input.

Display device.
According to paragraph 1,

The controller is

Displaying the content image on the first split screen and the mirroring image on the second split screen according to the execution of the multi-view function

Display device.
According to clause 12,

The controller is

Receiving touch input information from the mobile terminal, and displaying an indicator at a position of the mirroring image corresponding to a point of the received touch input on the display of the mobile terminal based on the received touch input information

Display device.
According to clause 12,

The controller is

When the first split screen is selected, determining that the touch pad mode event is detected in the mobile terminal

Display device.
In a method of operating a display device,

Receiving a video signal corresponding to the video being played by the mobile terminal;

displaying a mirrored image based on the received image signal;

determining whether a touch pad mode event is detected; and

When the touch pad mode event is detected, transmitting touch pad mode information to the mobile terminal to change the operation mode of the mobile terminal to the touch pad mode.

How the display device operates.