WO2024158074A1 - Portable controller - Google Patents

Abstract

A portable controller comprises a basic controller, and an expansion module that can be attached to or detached from the basic controller, wherein the basic controller includes a first interface unit, a wireless communication unit for communicating with a master device, and a first processor that receives an identifier of the expansion module from the expansion module through the first interface unit and controls the basic controller to operate to match the type of the expansion module, and the expansion module includes a second interface unit, and a second processor that controls the identifier of the expansion module to be provided to the basic controller through the second interface unit. The expansion module is easily attached to or detached from the portable controller such that functions of the portable controller can be expanded and changed.

Description

portable controller

The present invention relates to a controller for controlling extended reality (XR: eXtended Reality) devices.

Virtual Reality (VR) technology provides objects and backgrounds in the real world only as computer graphics (CG) images, and Augmented Reality (AR) technology provides virtual computers created on images of real objects. It also provides graphic images, and mixed reality (MR: Mixed Reality) technology is a computer graphics technology that mixes and combines virtual objects in the real world. The aforementioned VR, AR, MR, etc. are all simply referred to as XR (extended reality) technologies.

XR technology includes devices that provide images in graphics, and as a representative example, a device worn directly on the head, such as a head mounted display (HMD), can be used.

A head-mounted display that is worn on the head can use a portable controller that the user holds in his hand to input commands. There is also a wired type, but recently wireless type portable controllers that exchange data wirelessly are becoming popular.

Portable controllers go beyond simple user input functions and provide feedback to the user through vibration. In addition, user input through gestures can also be performed by detecting the movement of the user's hand (or arm) other than when the user directly manipulates buttons or wheels.

Additionally, different gestures and feedback are performed depending on the content being used, and the method of recognizing gestures is different depending on the type of XR device, so there is a problem of having to use different controllers depending on the situation.

Accordingly, the purpose of the present invention is to provide a portable controller whose functions can be easily expanded and changed.

A basic controller including a handle portion; and an expansion module detachable from the basic controller, wherein the basic controller includes: a handle portion that extends in the longitudinal direction and is held by a user's hand; An operating body located in front of the handle portion; a user input unit located on the operation body; a first connector formed on the operating body; and a first coupling rail formed on the manipulation body, wherein the expansion module includes a module body including a coupling portion in contact with the manipulation body; a second connector formed in the coupling portion and coupled to the first connector; And it provides a portable controller including a second coupling rail that is slide fastened to the longitudinal direction of the first coupling rail.

The first connector is located in the front of the manipulation body, the first coupling rail is located in the left and right directions of the manipulation body, and the coupling portion of the expansion module may have a U-shape surrounding the front and left and right sides of the manipulation body. there is.

The second connector protrudes and can be inserted into the first connector.

The first coupling rail may include a groove that is wide at the front and narrows toward the rear, and the second coupling rail may have a bar shape that is thin at the rear and thick at the front.

The expansion module may include a locking hook formed on the coupling portion, and the basic controller may include a locking protrusion to which the locking hook is coupled.

The locking hook is formed at one end of a hook lever rotatably coupled to the module body, and includes a unlocking button coupled to the other side of the hook lever. When the unlocking button is operated, the hook lever rotates and the The locking hook may be separated from the locking protrusion.

The locking protrusion includes an inclined surface extending from front to rear; And it may include a locking part that is bent on an inclined plane and catches the hook.

The locking hook includes a hook surface in contact with the catching portion, and a line extending from the rotation center of the hook lever to the hook surface may be perpendicular to the hook surface.

It includes a locking hole formed in the basic controller, and the locking protrusion may be located on one side of the locking hole.

The locking protrusion may be located at a lower portion of the second connector.

The upper surface of the manipulation body and the upper surface of the module body may be parallel.

It is located on the upper surface of the module body and further includes a second user input unit, and the first user input unit and the second user input unit may be located within a movable range of the user's thumb.

The basic controller may be left-right symmetrical.

The expansion module includes a ring case connected to both sides of the module body; And it may include a plurality of infrared LEDs (IR LEDs) mounted along the outer surface of the ring case.

The ring case is located at the bottom of the module body, and an outer surface of the ring case may be inclined toward the front.

The angle formed between the upper surface of the operating body and the front surface of the ring case may be tilted toward the rearward direction than 90°.

It includes a basic controller and an expansion module detachable from the basic controller, wherein the basic controller receives a first interface unit, a wireless communication unit for communicating with a master device, and an identifier of the expansion module from the expansion module through the first interface unit. and a first processor that receives and controls the basic controller to operate to match the type of the expansion module, wherein the expansion module receives the identifier of the expansion module through a second interface unit and a second interface unit. A portable controller may be provided, including a second processor that provides control to the basic controller.

The first processor may control to receive the identifier from the expansion module when the expansion module is mounted on the basic controller and the portable controller is turned on.

The expansion module further includes an optical output unit including a plurality of optical element markers, and the second processor may be controlled to provide arrangement information of the plurality of optical element markers to the basic controller through a second interface unit. .

The plurality of optical device markers may include a plurality of infrared LED markers.

The first processor may control the arrangement information of the plurality of optical element markers to be transmitted to the master device.

The master device may include a wearable XR device and an external display device.

The arrangement information of the plurality of optical element markers may include three-dimensional spatial coordinate information where the plurality of optical element markers are arranged in the expansion module.

The basic controller further includes a first sensor for sensing real-time movement of 3-DoF (Degrees of Freedom) of the basic controller, and the first processor transmits the sensed 3-DoF real-time movement to the master device. You can control transmission.

The first sensor may include at least one of an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, and a motion sensor.

The expansion module may further include a plurality of cameras, and the second processor may control the basic controller to provide characteristic information of the plurality of cameras.

The basic controller further includes a first sensor for sensing real-time movement of 3-DoF (Degrees of Freedom) of the basic controller, and the first processor is configured to detect images captured by the plurality of cameras received from the expansion module. Based on a plurality of camera images and the 3-DoF real-time movement, the 6-DoF real-time movement of the portable controller can be calculated, and the calculated 6-DoF real-time movement can be controlled to be transmitted to the master device.

The first processor may control real-time motion information of the portable controller to be calculated based on motion information of feature points in the plurality of camera images.

The plurality of cameras may include three cameras having a field of view of at least 120 degrees.

The expansion module may include an impact actuator, and the second processor may include an impact actuator driver IC.

The temperature sensor driver IC includes a memory to include an identifier of the expansion module and temperature sensor characteristic information, and controls to provide the temperature sensor characteristic information and temperature data sensed through the temperature sensor to the basic controller. You can.

The first processor may control transmission of the temperature center characteristic information and the temperature data to the master device.

According to at least one embodiment of the present invention, the portable controller can expand and change functions by easily attaching and detaching an expansion module.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 is a perspective view showing a portable controller according to one aspect of the present invention.

Figure 2 is a diagram showing a basic controller and an expansion module of a portable controller according to one aspect of the present invention.

Figure 3 is an exploded perspective view showing the basic controller of a portable controller according to one aspect of the present invention.

Figure 4 is an exploded perspective view of an expansion module of a portable controller according to one aspect of the present invention.

Figure 5 is a side view showing an operating body of a portable controller according to one aspect of the present invention.

Figure 6 is a plan view showing the operation body and module body of the portable controller according to one aspect of the present invention.

Figure 7 is a diagram showing a connection part and a locking part of a portable controller according to one aspect of the present invention.

Figure 8 is a diagram showing a coupling rail of a portable controller according to one aspect of the present invention.

Figure 9 is a diagram showing a method of fastening a coupling rail of a portable controller according to an aspect of the present invention.

Figure 10 is a diagram showing a locking method of the basic controller and expansion module of the portable controller according to one aspect of the present invention.

Figure 11 is a diagram illustrating a method of unlocking the basic controller and expansion module of a portable controller according to one aspect of the present invention.

Figure 12 is a block diagram for explaining the operation of a portable controller according to one aspect of the present invention.

Figure 13 shows a portable controller and a master device linked thereto according to an embodiment of the present invention.

Figure 14 shows an interactive operation process between a portable controller and a master device linked thereto according to an embodiment of the present invention.

15 to 17 are detailed block diagrams for explaining the operation of a portable controller according to one aspect of the present invention.

Figure 18 is a top view, left side view, and right side view of the expansion module of the portable controller according to one aspect of the present invention.

Figure 19 shows an interactive operation process between a portable controller and a master device linked thereto according to an embodiment of the present invention.

Figure 20 is a detailed block diagram for explaining the operation of a portable controller according to an aspect of the present invention.

Figure 21 is a plan view and a front view of an expansion module of a portable controller according to an aspect of the present invention.

Figure 22 is a detailed block diagram for explaining the operation of a portable controller according to one aspect of the present invention.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numbers regardless of reference numerals, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

FIG. 1 is a perspective view showing a portable controller 10 according to an aspect of the present invention, and FIG. 2 is a diagram showing a basic controller and an expansion module of the portable controller according to an aspect of the present invention.

XR technology may include XR devices that provide images to provide images graphically. XR devices can use a screen located in front of or around the user, and as a representative XR device, devices worn directly on the head, such as a head mounted display (HMD), can be used.

Since the XR device is used while the user moves as if he or she were actually experiencing it, a portable controller 10 that the user can hold and operate in order to input commands is required.

The controller 10 of the present invention may include a handle 102 that is wrapped and grasped by hand, and an operating body 101 located in front of the handle 102. The handle part 102 can be wrapped with the lower palm of the thumb, middle finger, ring finger, and little finger, and the button located on the operating body 101 can be operated using the index finger and thumb.

Use your thumb to operate the function key 123 and the trackball 121 or jog stick 122 located at the top of the operating body 101, and use your index finger to operate the trigger button located in front of the operating body 101. (125) can be manipulated.

Basically, a pair of controllers 10a and 10b can be configured as a set so that the user can hold and use them in his left and right hands, respectively. A pair of controllers 10a and 10b may have the same buttons, but may be configured to enable various operations through different buttons.

For example, according to the embodiment shown in FIG. 1, the left side may include input units 121, 122, 123, and 125 in the form of a jogstick 122, and the right side may include an input unit in the form of a trackball 121. The jogstick 122 is advantageous for two-dimensional movement manipulation, and the trackball 121 can be used for rotation and gaze movement.

Depending on the application being used, the number of portable controllers 10 required may vary. If the operation is a simple application, it can be controlled with just one basic controller 100, but if various controls are required, it may be operated by holding it with both hands.

When using an application that searches space by moving the gaze, both the jogstick 122 and the trackball 121 are required, so the user must hold the portable controllers 10a and 10b in both hands. In this way, one or two of the pair of controllers (10a, 10b) can be used depending on the situation.

The portable controller 10 can go beyond simply manipulating a cursor or pointer on the screen to detect a user's gesture and recognize it as a user input.

The user's gesture style may vary depending on the content of the application being used. For example, when using a sports application, the user's movements can be detected through the portable controller 10. The functions required for the portable controller 10 may vary depending on how the XR device detects the user's gesture and the sensor location of the XR device.

Conventionally, a position sensor such as a gyro sensor of the portable controller 10 itself was used to detect the user's movement (gesture) based on a change in the position of the portable controller 10.

However, in the case of XR applications, there is a need to detect more precise user movements. In order to detect not only large gestures such as arm movements, but also small gestures such as wrist bending and wrist rotation, a method of externally tracking the movement of the controller 100 can be used.

In addition, in addition to receiving user commands through the portable controller 10, it can function as an output unit that provides tactile, a sense beyond sight and hearing, to the user.

The sense of touch provides not only a tremor that vibrates in one direction, but also other sensations depending on the direction, speed, and size of the vibration. The actuator 141, which provides a rich sense of touch, requires a plurality of actuators to provide various types of sensation.

However, in order to provide various input and output functions as described above, the problem arises that the size of the portable controller 10 increases.

Accordingly, the portable controller 10 of the present invention can be composed of a basic controller 100 and an expansion module 200 that is detachable from the basic controller 100, as shown in FIG. 2.

Basic functions can be used only with the basic controller 100, but an expansion module 200 equipped with specialized functions can be added to the basic controller 100 for more accurate control and to provide various senses.

The expansion module 200 has a form that can be inserted from the front of the basic controller 100, as shown in FIG. 2. It can be coupled at the front of the manipulation body 101 so as not to interfere with the handle portion 102 extending toward the rear of the manipulation body 101.

In the combined state as shown in Figure 1, the functions mounted on the expansion module 200 can be additionally utilized. The expansion module 200 of this embodiment includes a ring case 220 (LED marking) in which an LED marker 224 capable of tracking the user's gesture is arranged.

The expansion module 200 includes a module body 210 including a U-shaped coupling portion 240 surrounding the front and side surfaces of the manipulation body 101. Modules for additional functions may be mounted inside the module body 210, and for example, additional buttons may be mounted as shown in FIG. 3. The module body 210 may have various shapes depending on the function of the expansion module 200.

A coupling rail is provided so that the basic controller 100 and the expansion module 200 can be stably coupled, and the expansion module 200 and the basic controller 100 are electrically connected to utilize the functions of the expansion module 200. A second connector 233 may be provided. The connector and coupling rail may be located around the side surface of the operating body 101 where the basic controller 100 and the expansion module 200 are in contact and at the coupling portion 240 of the expansion module 200.

Figure 3 is an exploded perspective view showing the basic controller 100 of the portable controller 10 according to one aspect of the present invention.

The basic controller 100 includes cases 111, 112, 113, 114, and 115 that constitute the exterior, and the case may be separately configured with a handle portion 102 and an operating body 101, but in this embodiment, the upper and lower The assembly of parts can be secured through the cases 111 and 112 that are coupled in one direction.

In the operation body 101, a trigger button 125, function keys, and a trackball 121 are mounted on the main board 131. The handle portion 102 may include a battery 139 and a basic actuator 141 that transmits vibration, is mounted on the auxiliary board 132, and is connected to the main board 131 to sense user input.

The cases 111, 112, 113, and 114 are manufactured by injection molding for ease of manufacturing, but may be equipped with a partially rigid middle frame 115 for durability. The middle frame 115 is located below the main board 131 of the operating body 101 and can support the back of the input unit when the input unit is pressed. The trigger button 125 is also combined with the middle frame 115, and the frame can support the force of the trigger button 125 by pressing the button in the direction of the frame.

A top cover 112 having an opening may be coupled to the upper surface of the manipulation body 101 to expose the user input unit. The lower part of the handle portion 102 may further include a bottom cover 114 made of another material to improve grip. The top cover 112 and bottom cover 114 may be omitted.

The basic controller 100 can be configured left and right symmetrically so that it can be used without distinction between left and right hands. When using a pair of basic controllers (100), the basic controller (100) equipped with a trackball (121) and a jog stick (122) can be used as the basic controller (100). Since the basic controller (100) is symmetrical, the left and right sides can be freely controlled by the user. You can change it by setting it.

Figure 4 is an exploded perspective view of the expansion module 200 of the portable controller 10 according to one aspect of the present invention.

The module body 210 of this embodiment may be composed of a lower case 213, a middle frame 215, and an upper cover 211.

It includes ring cases (221, 222) extending from the left and right sides of the module body (210), and the ring cases (221, 222) may be configured as one piece with the module body (210), and are fastened to the module body (210). It can also be configured in the form

In this embodiment, the lower case 213 of the module body 210 and the inner case 221 of the ring case are integrated, and the outer cover 222 surrounding the outer surface of the ring case is coupled to the inner case 221. It can be configured in the form

An LED lamp 224 (LED marker) is mounted between the inner case 221 and the outer cover 222, and a pair of LED lamps 224 can be provided on both left and right sides toward the outside of the ring case 221 and 222. there is. The LED lamp 224 may be connected through the flexible substrate 223. The outer cover 222 may include a light-transmitting material that can pass infrared rays.

The XR device is equipped with a sensor that detects the movement of the LED marker 224 mounted on the ring case 221 and 222, and can detect the user's gesture by tracking the movement of the LED marker 224.

The LED marking 220 shown in FIG. 1 can detect movement through a camera located in the front, and when tracking the LED marker 224 on a head-mounted display worn on the user's head, the LED marker 224 The arrangement of the LED marking 220 must be changed so that it faces the user's face.

In this embodiment, the LED marker 224 is disposed along the outer surface of the ring cases 221 and 222, so that the LED marker 224 located at the rear can also be recognized by the camera located at the front. ) may have a curved shape with the outer surface facing slightly forward. That is, the hole size at the front of the ring cases 221 and 222 may be smaller than the size of the hole at the rear.

The module body 210 may be additionally provided with an expansion button 234, and a module board 231 on which the expansion button 234 is mounted may be mounted on the module body 210. The module board 231 is connected to the LED marker 224 and may further include a second connector 233 to transmit the functions mounted on the expansion module 200 to the basic controller 100. Through the second connector 233, the control unit of the basic controller 100 can control the LED marker 224 and receive input from the expansion button to control the XR controller.

Figure 5 is a side view showing the operation body 101 of the portable controller 10 according to one aspect of the present invention. For better grip, the handle portion 102 has a cylindrical shape with a diameter of 30 to 40 mm and may become thinner toward the lower end.

The average adult finger size is the sum of the widths of the middle ring finger and little finger of about 62.8 mm, so taking this into consideration, the length of the handle portion 102 can be configured to be 71 mm.

The angle of the handle portion 102 can be configured to be bent at about 35° with the upper surface of the operating body 101. The upper surface angle of the handle and the operating body 101 can be determined by considering the manipulability of the thumb.

The front-to-back length of the manipulation body 101 can be determined by considering an appropriate index finger position for manipulating the trigger button 125 located at the front lower portion of the manipulation body 101.

Figure 6 is a plan view showing the operation body 101 and the module body 210 of the portable controller 10 according to one aspect of the present invention.

The user can control the function key 123 and the jog stick 122 (trackball 121) located on the upper surface of the operating body 101 as well as the expansion button 234 located on the expansion module 200 with the thumb. The user input units 121, 122, 123, and 234 can be arranged in consideration of the thumb's range of motion (30°).

Figure 7 is a diagram showing a coupling rail of the portable controller 10 according to an aspect of the present invention, Figure 6 (a) is a first coupling rail 116 of the basic controller 100, Figure 6 (b) is the second coupling rail 216 of the expansion module 200. The first coupling rail 116 may have a concave shape and the second coupling rail 216 may have a protruding shape.

Since there should be no inconvenience when using only the basic controller 100, the basic controller 100 includes a concave structure rather than a protruding structure, and in particular, the coupling portion 240 of the expansion module 200 has a U-shape. , the risk of damage to the protruding joint structure is low.

The first coupling rail 116 may have a groove shape and the second coupling rail 216 may have a bar shape. The groove of the first coupling rail 116 may become narrower toward the rear, and the second coupling rail 216 may also have a shape where the rear is thinner than the front. The second coupling rail 216 may be configured to have a curved surface at the rear to facilitate coupling to the first coupling rail 116.

Figure 8 is a diagram showing a method of fastening the coupling rail of the portable controller 10 according to one aspect of the present invention. The left side is the rear and the right side is the front, and the diagram shows only the second coupling rail 216 of the expansion module 200 based on the basic controller 100.

The rear of the second coupling rail 216 is inserted into the front of the first coupling rail 116, and initially there is a large gap between the first coupling rail 116 and the second coupling rail 216 (gap1, (a) ) When the coupling is completed, the gap between the first coupling rail 116 and the second coupling rail 216 disappears (gap2=0), and the expansion module 200 can be completely coupled to the basic controller 100.

FIG. 9 is a diagram illustrating the connectors 133 and 233 and the locking portion 260 of the portable controller 10 according to one aspect of the present invention. (a) is a front view of the basic controller 100 and (b) is a rear view of the expansion module 200.

A first connector 133 and a locking hole 117 are located on the front of the basic controller 100, and the above-described first coupling rails 116 are located on the left and right sides. The coupling portion 240 of the expansion module 200 has second coupling rails 216 located on the left and right, and a second connector 233 and a locking hook 2611 protrude from the U-shaped inner end.

Like the second coupling rail 216, the connector (second connector 233) and the locking structure (locking hook 2611) of the expansion module 200 have a protruding shape, and the connector (first connector 233) of the basic controller 100 has a protruding shape. The connector 133) and the locking structure (locking protrusion 1172 located in the lock hole 117) may have a shape that is recessed inward rather than protruding.

As shown in FIG. 3, the protruding second connector 233 of the expansion module is inserted into the first connector 133 of the basic controller 100, and the expansion module 200 and the basic controller 100 can be electrically connected. there is.

The coupling is completed when the front of the basic controller 100 and the inner surface of the coupling portion 240 of the expansion module 200 contact each other. The coupling rail assists the coupling of the expansion module 200 and the basic controller 100, but since the portable controller 10 moves while held by the user, it can be equipped with a locking device to ensure stable coupling.

The locking device of the present invention may include a locking hook 2611 protruding from the expansion module 200 and a locking protrusion 1172 of the basic controller 100.

FIG. 10 is a diagram illustrating a locking method of the basic controller 100 and the expansion module 200 of the portable controller 10 according to one aspect of the present invention. The locking hook 2611 of the present invention is formed on one side of the hook lever 261 rotatably coupled to the module body 210 and protrudes from the module body 210.

When the basic controller 100 is inserted into the coupling portion 240 of the expansion module 200 along the coupling rails 116 and 216, the end of the locking hook 2611 and the inclined surface 1174 formed in front of the basic coupling protrusion come into contact with each other. all. When the locking hook 2611 moves along the inclined surface 1174 of the engaging protrusion, the horizontal position changes (moves to the right in the drawing), so the hook lever 261 rotates and the horizontal position of the locking hook 2611 changes. It may vary.

When the locking hook 2611 reaches the end of the inclined surface 1174, the locking hook 2611 is inserted into the locking protrusion 1172, as shown in (b) of FIG. 9, and the basic controller 100 and the expansion module 200 ) can be concluded.

The locking protrusion 1172 includes a locking portion 1173 that is bent from the inclined surface 1174 and located on the opposite side of the inclined surface 1174, and the locking hook 2611 has a hook surface 2612 in contact with the locking portion 1173. It can be included.

The hook surface 2612 is configured to be perpendicular to the line extending from the rotation axis 2615 of the hook lever 261 to the hook surface 2612, thereby preventing the locking hook 2611 from being separated from the locking protrusion 1172.

The locking protrusion 1172 may be formed on the side of the locking hole 117 formed in front of the basic controller 100, and the locking protrusion 1172 may be formed on both the left and right sides of the locking hole 117. The basic controller 100 can be used without distinction between left and right, and both the left-facing expansion module 200 and the right-facing expansion module 200 can be combined.

FIG. 11 is a diagram illustrating a method of unlocking the basic controller 100 and the expansion module 200 of the portable controller 10 according to one aspect of the present invention. The other side 2613 of the hook lever 261 is connected to the unlock button 265.

The lock release button 265 operates in a slide manner, and the hook lever 261 rotates around the rotation axis 2615 according to the slide direction, and the position of the lock hook 2611 located on one side of the hook lever 261 changes. can do.

The unlock button 265 is located in the front of the expansion module 200, and the hook lever 261 is disposed in the front and rear directions of the module body 210 of the expansion module 200. In the locked state as shown in Figure 10 (b), when the user slides the unlock button 265 in Figure 11 (a), the hook lever 261 rotates as shown in Figure 11 (b) and the lock hook 2611 ) can be released from the locking protrusion (1172).

Hereinafter, the operation of the portable controller 10 will be described with reference to FIG. 12. Figure 12 is a block diagram for explaining the operation of a portable controller according to one aspect of the present invention.

As described above, the portable controller 10 may include the basic controller 100 and the expansion module 200 that is detachable therefrom. The expansion module 200 may be implemented according to one of several types, each equipped with specialized functions. That is, the user can select the type of expansion module 200 equipped with the functions he or she needs and install it on the basic controller 100 to use it.

First, let's look at the basic controller 100.

The basic controller 100 includes a wireless communication unit 1110, a first user input unit 1123, a first sensor 1140, a first audio output unit 1152, a first haptic module 1153, and a first interface unit ( 1160), a first processor 1180, and a first power supply unit 1190.

Not all components of the basic controller 100 shown in FIG. 12 are essential for configuring the basic controller 100. The basic controller 100 may have more or fewer components than those listed above.

Let's take a look at each of these components.

The wireless communication unit 1110 is used for the portable controller 10 to communicate with the master device (20 and 30 in FIG. 13) and may include at least one of a wireless Internet module and a short-range communication module.

The wireless Internet module refers to a module for wireless Internet access and may be built into or external to the basic controller 100. The wireless Internet module is configured to transmit and receive wireless signals in a communication network based on wireless Internet technologies.

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), and WiMAX (Worldwide). Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G, etc., and the wireless Internet The module transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

The short-range communication module is for short-range communication and includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and NFC ( Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology can be used to support short-distance communication.

Here, the master devices 20 and 30 are wearable devices (for example, smartwatches) capable of exchanging data with (or interoperating with) the portable controller 10 according to the present invention. , it may be smart glass, head mounted display (HMD) 20, or an external display device 30. The short-range communication module may detect (or recognize) a master device around the portable controller 10 that can communicate with the portable controller 10. Furthermore, if the detected master device is a device authenticated to communicate with the portable controller 10 according to the present invention, the first processor 1180 stores at least a portion of the data processed by the portable controller 10, It can be transmitted to the master device through a short-distance communication module. Accordingly, the user of the master device can use the data processed by the portable controller 10 through the master device.

The first user input unit 1123 is for receiving information from the user. When information or a user command is input through the first user input unit 1123, the first processor 1180 processes the input information or user command to correspond to the input information or user command. The operation of the portable controller 10 can be controlled. The first user input unit 1123 includes a mechanical input means (or a mechanical key, for example, a button located on the front/rear or side of the basic controller 100, a dome switch, jog wheel, jog switch, etc.) and touch input means.

The first user input unit 1123 may include at least one of the trackball 121, the jog stick 122, the function key 123, and the trigger button 125.

The first sensor 1140 may include at least one of an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, and a motion sensor. You can. The first processor 1180 may track real-time movements of at least 3-Degrees of Freedom (DoF) of the portable controller 10 or the basic controller 100 through the first sensor 1140.

The first audio output unit 1152 may output audio data received from the wireless communication unit 1110 or stored in the memory of the portable controller 10. The first sound output unit 1152 also outputs sound signals related to functions (eg, sound effects, etc.) performed by the portable controller 10. The second sound output unit 1152 may include a receiver, speaker, buzzer, etc.

The first haptic module 1153 generates various tactile effects that the user can feel. A representative example of a tactile effect generated by the first haptic module 1153 may be vibration. The intensity and pattern of vibration generated by the first haptic module 1153 may be controlled by user selection or settings of the first processor 1180. For example, the first haptic module 1153 may synthesize and output different vibrations or output them sequentially.

In addition to vibration, the first haptic module 1153 performs functions such as pin arrays moving perpendicular to the contact skin surface, blowing force or suction force of air through a nozzle or intake port, grazing the skin surface, contact with electrodes, and electrostatic force. A variety of tactile effects can be generated, including effects by stimulation and effects by reproducing hot and cold sensations using devices capable of absorbing heat or generating heat.

The first haptic module 1153 can not only deliver a tactile effect through direct contact, but can also be implemented so that the user can feel the tactile effect through muscle senses such as fingers or arms. Two or more first haptic modules 1153 may be provided depending on the configuration of the portable controller 10 or the basic controller 100.

The first interface unit 1160 serves as a passageway for all external devices (eg, expansion module 200) connected to the basic controller 100. The first interface unit 1160 receives data from an external device, supplies power to an external device, or transmits data inside the basic controller 100 to an external device. For example, a wired/wireless data port, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, etc. are connected to the first interface unit 160. may be included.

The first processor 1180 controls the overall operation of the portable controller 10. The first processor 1180 provides appropriate information or functions to the user by processing signals, data, and information input or output through the components described above or by running an application program stored in the memory of the portable controller 10. Or you can process it.

The first processor 1180 may control at least some of the components to run the application program stored in the memory. Furthermore, the first processor 1180 can operate at least two of the components included in the portable controller 10 in combination with each other in order to run the application program.

The memory may be included in the first processor 1180, or may be configured as a separate component within the basic controller 100 that is independent of the first processor 1180.

When the expansion module 200 is connected to the basic controller 100, the basic (or when the portable controller 10 is turned on after the expansion module 200 and the basic controller 100 are interconnected), The expansion module 200 can be recognized by the basic controller 100. Then, the first processor 1180 can receive the identifier (ID) of the expansion module from the expansion module 200 through the first interface unit 1160. The identifier (ID) of the expansion module is used to identify the type of the expansion module 200.

The first processor 1180 may control the overall operation of the portable controller 10 to match the type of the connected expansion module 200 based on the identifier (ID) of the expansion module.

The first power supply unit 1190 receives external power and internal power under the control of the first processor 1180 and supplies power to each component included in the portable controller 10. The first power supply unit 1190 may include a battery, and the battery may be a built-in battery or a replaceable battery. The first power supply unit 1190 may include a Power Management IC (PMIC) for power management of mobile devices.

Hereinafter, we will look at the expansion module 200.

The expansion module 200 includes a second sensor 2140, a second user input unit 2123, a second interface unit 2160, a display unit 2151, a second sound output unit 2152, and a second haptic module ( 2153), an optical output unit 2154, a second processor 2180, and a second power supply unit 2190.

Not all components of the expansion module 200 shown in FIG. 12 are essential for configuring the expansion module 200. The expansion module 200 may have more or fewer components than those listed above. In particular, the expansion module 200 may be configured to further include other components to meet specialized functions for each type.

Let's take a look at each of these components.

The second sensor 2140 may include environmental sensors (e.g., barometer, hygrometer, thermometer, radiation detection sensor, heat detection sensor, gas detection sensor, etc.) and chemical sensors (e.g., electronic nose sensor, thermometer, heart rate sensor, etc.). It may include at least one of a healthcare sensor, a biometric sensor such as iris recognition and fingerprint recognition, etc.). For example, at least one of the user's body temperature and heart rate may be measured through the second sensor 2140.

The second user input unit 2123 is for receiving information from the user. When information or a user command is input through the second user input unit 2123, the second processor 2180 is configured to respond to the input information or user command. The operation of the expansion module 200 can be controlled. This second user input unit 2123 may include a mechanical input means (or a mechanical key, for example, a button located on the front/rear or side of the expansion module 200, a dome switch, jog wheel, jog switch, etc.) and touch input means.

The second user input unit 2123 may include the expansion button 234.

The second interface unit 2160 serves as a passageway for all external devices (for example, the basic controller 100) connected to the expansion module 200. The second interface unit 2160 receives data from an external device, receives power from an external device, or transmits data inside the expansion module 200 to an external device. For example, wired/wireless data ports, audio I/O (Input/Output) ports, video I/O (Input/Output) ports, etc. are connected to the second interface unit 2160. May be included.

The display unit 2151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display ( It may include at least one of a flexible display, a 3D display, and an e-ink display.

There may be two or more display units 2151 depending on the implementation type of the expansion module 200. In this case, in the expansion module 200, a plurality of display units may be spaced apart or arranged integrally on one side, or may be arranged on different sides.

The display unit 2151 may include a touch sensor that detects a touch on the display unit 2151 so that control commands can be input by a touch method. Using this, when the display unit 2151 is touched, the touch sensor detects the touch, and the second processor 2180 can generate a control command corresponding to the touch based on this. there is. Content input by the touch method may be letters or numbers, instructions in various modes, or designable menu items.

The touch sensor may be formed integrally with the display. For example, the touch sensor may be placed on a display substrate or provided inside the display.

In this way, the display unit 2151 can form a touch screen together with the touch sensor, and in this case, the touch screen can function as the second user input unit 2123.

The second audio output unit 2152 may output audio data received from the basic controller 100 or stored in the memory of the expansion module 200 through the second interface unit 2160. The second sound output unit 2152 also outputs sound signals related to functions (eg, sound effects, etc.) performed by the portable controller 10. The second audio output unit 2152 may include a receiver, speaker, buzzer, etc.

The second haptic module 2153 generates various tactile effects that the user can feel. A representative example of a tactile effect generated by the second haptic module 2153 may be vibration. The intensity and pattern of vibration generated by the second haptic module 2153 may be controlled by the user's selection or settings of the first processor 1180 or the second processor 2180. For example, the second haptic module 2153 may synthesize and output different vibrations or output them sequentially.

In addition to vibration, the second haptic module 2153 performs functions such as pin arrays that move perpendicular to the contact skin surface, blowing force or suction force of air through a nozzle or inlet, grazing the skin surface, contact with electrodes, and electrostatic force. A variety of tactile effects can be generated, including effects by stimulation and effects by reproducing hot and cold sensations using devices capable of absorbing heat or generating heat.

The second haptic module 2153 can not only deliver a tactile effect through direct contact, but can also be implemented so that the user can feel the tactile effect through muscle senses such as fingers or arms. Two or more second haptic modules 2153 may be provided depending on the configuration of the expansion module 200.

The second haptic module 2153 may include an impact actuator to output a larger vibration than the first haptic module 1153.

The optical output unit 2154 provides at least one of the position, movement, movement direction, rotation, rotation direction, and orientation direction of the portable controller 10 or the expansion module 200 to the master device 20, 30. Light can be output for notification purposes. The light output unit 2154 may include the previously described LED marking 220, which is a type of optical element marker module. The LED marking 220 may include at least one LED marker 224, which is a type of optical device marker. Each LED marker 224 may be composed of an infrared LED. Of course, other types of optical element markers other than the LED marker 224 may be used.

The arrangement of the at least one LED marker in the expansion module 200 may vary depending on the type of the master device 20 or 30. For example, if the master device is a wearable XR device (e.g., HMD, etc.) worn by the user, the LED marker 224 is directed toward the wearable device when the user normally uses the portable controller 10. The LED marker may be placed. Alternatively, if the master device is an external display device equipped with a camera (e.g., TV, monitor, etc.), the LED marker 224 displays the external display device when the user normally uses the portable controller 10. The LED marker may be arranged to face.

The second processor 2180 may control at least some of the components of the expansion module 200 in order to control the overall operation of the expansion module 200.

The second processor 2180 may include a memory for storing data related to the overall operation of the expansion module 200. The memory may be configured as a separate component independent from the second processor 2180.

The identifier (ID) of the expansion module may be stored in the memory. The identifier (ID) of the expansion module is used to identify the type of the expansion module 200.

When the expansion module 200 is connected to the basic controller 100, the basic (or when the portable controller 10 is turned on after the expansion module 200 and the basic controller 100 are interconnected), The expansion module 200 can be recognized by the basic controller 100. Then, the second processor 200 can transmit its expansion module identifier (ID) to the basic controller 100.

The second power supply unit 2190 may supply power to each component included in the expansion module 200 under the control of the first processor 1180 or the second processor 2180.

The portable controller 10 and the master device linked thereto will be described with further reference to FIG. 13. Figure 13 shows a portable controller and a master device linked thereto according to an embodiment of the present invention.

For example, as shown in (13-1) of FIG. 13, the portable controller 10 can link with the wearable XR device 20 worn by the user (U). In this case, the portable controller 10 may be configured by combining a first expansion module 200-1 suitable for interworking with the wearable XR device 20 to the basic controller 100.

In the first expansion module 200-1, the LED marker 224 may be arranged to emit light in a direction toward a user who normally uses the portable controller 10.

The wearable XR device 20 detects the light output from the LED marker 224 through its optical sensor (e.g., camera) positioned to face forward, thereby detecting the position and movement of the LED marker 224. , at least one of the movement direction, rotation, rotation direction, and orientation direction is tracked, and through this, at least one of the position, movement, movement direction, rotation, rotation direction, and orientation direction of the first expansion module 200-1 ( Hereinafter, movement information) can be determined.

Meanwhile, the wearable XR device 20 may receive the 3-DoF movement information from the portable controller 10.

Accordingly, the wearable XR device 20 can track the 6-DoF movement of the portable controller 10 based on the 3-DoF movement information and the expansion module movement information.

This can be understood as a type of inside-out tracking.

Meanwhile, as shown in (13-2) of FIG. 13, the portable controller 10 can interact with an external display device 30 equipped with an optical sensor 35. In this case, the portable controller 10 may be configured by coupling a second expansion module 200-2 suitable for interworking with the external display device 30 to the basic controller 100.

In the second expansion module 200-2, the LED marker 224 may be arranged to emit light in a direction toward the external display device 30 when the portable controller 10 is normally used by the user.

The external display device 30 senses the light output from the LED marker 224 through its optical sensor 35 (e.g., a camera) to determine the position, movement, and direction of movement of the LED marker 224. , at least one of rotation, rotation direction, and orientation direction is tracked, and through this, at least one of the position, movement, movement direction, rotation, rotation direction, and orientation direction of the second expansion module 200-2 (hereinafter referred to as expansion) module movement information) can be identified.

Meanwhile, the external display device 30 may receive the 3-DoF movement information from the portable controller 10.

Accordingly, the external display device 30 can track the 6-DoF movement of the portable controller 10 based on the 3-DoF movement information and the expansion module movement information.

This can be understood as a type of outside-in tracking.

The external display device 30 may include an interlocking module for a portable controller (e.g., a game module) and interoperate with (or communicate with) the portable controller 10 through the portable controller interlocking module. (10) may be connected to a separate external interlocking device (for example, a game console) for communication.

Hereinafter, with reference to FIG. 14, we will look at the mutual operation between the portable controller 10 according to FIG. 13 and the master devices 20 and 30 linked thereto in more detail. Figure 14 shows an interactive operation process between a portable controller and a master device linked thereto according to an embodiment of the present invention.

The expansion module 200 is mounted on the basic controller 100, so that the basic controller 100 and the expansion module 200 can be interconnected [S141]. The basic controller 100 and the expansion module 200 may be wired to each other through a first interface unit 1160 and a second interface unit 2160.

When the expansion module 200 is connected to the basic controller 100, the basic (or when the portable controller 10 is turned on after the expansion module 200 and the basic controller 100 are interconnected), The expansion module 200 can be recognized by the basic controller 100. Then, the expansion module 200 can transmit its expansion module identifier (ID) and placement information of the LED marker 224 to the basic controller 100 [S142]. Alternatively, the basic controller 100 may read the expansion module identifier (ID) and placement information of the LED marker 224 from the expansion module 200. The expansion module identifier (ID) is used to identify the type of expansion module 200 described above. The placement information of the LED marker 224 may include physical three-dimensional space coordinate information where the LED marker 224 is placed on the expansion module 200 or the LED marking 220. The arrangement information of the LED marker 224 may include LED calibration information.

The basic controller 100 may transmit the expansion module identifier (ID) and placement information of the LED marker 224 to the master devices 20 and 30 through short-distance communication (e.g., Bluetooth communication) [ S143].

If the short-range communication is connected between the portable controller 10 and the master devices 20 and 30 before step S142, step S143 can be performed immediately after step S142.

If the short-range communication is not connected between the portable controller 10 and the master devices 20 and 30 before step S142, step S143 may be performed after step S142 and establishing the short-range communication connection.

The master devices 20 and 30 may identify the expansion module 200 based on the expansion module identifier (ID) [S144].

And, the master devices 20 and 30 detect at least one of the position, movement, movement direction, rotation, rotation direction, and orientation direction of the LED marker 224 by sensing the light output from the LED marker 224. It can be traced [S145].

The master devices 20 and 30 may calculate real-time movement information of the expansion module based on the placement information of the LED marker 224 and the tracking information of the LED marker [S146].

Meanwhile, the basic controller 100 can acquire 3-DoF real-time movement information through the first sensor 1140 [S147].

The basic controller 100 performs the 3- DoF real-time motion information can be transmitted to the master devices 20 and 30 through short-distance communication [S148].

Step S148 may be performed after step S143, step S144, step S145, and step S146, simultaneously with any one of these steps, or between any two of these steps.

The master devices 20 and 30 may calculate the 6-DoF real-time movement of the portable controller 10 based on the 3-DoF real-time movement information and the real-time movement information of the expansion module 200.

The master devices 20 and 30 can reflect the 6-DoF real-time movement of the portable controller 10 on the display screen [S149]. At this time, the master devices 20 and 30 may further reflect the user input when reflecting the 6-DoF real-time movement of the portable controller 10 on the display screen.

Hereinafter, with reference to FIG. 15, we will look at the portable controller 10 in more detail. Figure 15 is a detailed block diagram for explaining the operation of a portable controller according to one aspect of the present invention. In the detailed block diagram of FIG. 15, some components of the block diagram of FIG. 12 are omitted.

First, let's look at the basic controller 100.

The first power supply unit 1190 may include, for example, a battery 1191, a power management IC (PMIC) 1192, and a power supply module 1193.

The PMIC 1192 controls the power of the battery 1191 to supply appropriate power to each component of the basic controller 100.

The power supply module 1193 supplies power from the battery 1191 to the expansion module 200 through the first interface unit 1160.

The first processor 1180 may include a Universal Serial Bus (USB) control chip 1181, a main processor chip, and a multiplexer 1183.

The USB control chip 1181 can transmit a USB control signal to the expansion module 200 through the first interface unit 1160 and to other components of the basic controller 100.

The main processor chip 1182 can transmit various data to the expansion module 200. For example, the main processor chip 1182 can output a video signal from the USB3.1/Display Port terminal and provide it to the expansion module 200 through the first interface unit 1160, and audio from the Aux terminal. A signal can be output and provided to the expansion module 200 through the first interface unit 1160. The video signal may be output through the display unit 2151 of the expansion module 200, and the audio signal may be output through the second audio output unit 2152 of the expansion module 200.

In addition, the main processor chip 1182 multiplexes the GIOP data output from the GPIO (general-purpose input/output) terminal and the USB2.0 data output from the USB2.0 terminal by the multiplexer 1183 to form the first interface unit. It can be transmitted to an external device through (1160). Although it is not illustrated in FIG. 15 that the GIPO data and USB2.0 data are used in the expansion module 200, the GIPO data and USB2.0 data can be used depending on the type of the expansion module 200. Of course.

Hereinafter, we will look at the expansion module 200.

The second power unit 2190 may receive power from the basic controller 100 through the second interface 2160 and supply power to each component within the expansion module 200.

The second sensor 2140 may include environmental sensors (e.g., barometer, hygrometer, thermometer, radiation detection sensor, heat detection sensor, gas detection sensor, etc.) and chemical sensors (e.g., electronic nose sensor, thermometer, heart rate sensor, etc.). It may include at least one of a healthcare sensor, a biometric sensor such as iris recognition and fingerprint recognition, etc.). For example, at least one of the user's body temperature and heart rate may be measured through the second sensor 2140. Sensing data sensed by the second sensor 2140 is stored in a memory within the second processor 2180 or a separate memory within the expansion module 200 and then transmitted to the basic controller 100 through the second interface unit 2160. It can be provided as . The USB control chip 1181 may receive the sensing data and transmit it to the main processor chip 1182.

The second user input unit 2123 may include, for example, a button 2123-1 and a touch input means 2123-2 located on the front/rear or side of the expansion module 200. The user command input through the second user input unit 2123 is stored in the memory within the second processor 2180 or a separate memory within the expansion module 200 and then transmitted to the basic controller 100 through the second interface unit 2160. ) can be provided. The USB control chip 1181 can receive the user command and transmit it to the main processor chip 1182.

The display unit 2151 receives and displays the video signal through the second interface unit 2160, and the second audio output unit 2152 receives the audio signal through the second interface unit 2160 and outputs it. can do.

The second processor 2180 may control at least some of the components of the expansion module 200 in order to control the overall operation of the expansion module 200.

The second processor 2180 may include a memory for storing data related to the overall operation of the expansion module 200. The memory may be configured as a separate component independent from the second processor 2180.

In FIG. 15, it is illustrated that the second processor 2180 is configured as a Field Programmable Gate Array (FPGA).

The second power supply unit 2190 may supply power to each component included in the expansion module 200 under the control of the first processor 1180 or the second processor 2180.

Hereinafter, with reference to FIG. 16, we will look at the driving of the LED marker 224 of the portable controller 10 in more detail. Figure 16 is a detailed block diagram for explaining the operation of a portable controller according to one aspect of the present invention. In the detailed block diagram of FIG. 16, some components of the block diagram of FIG. 12 are omitted.

First, let's look at the basic controller 100.

When the first power unit 1190 supplies appropriate power to each component of the basic controller 100, it can supply power to the expansion module 200 through the first interface unit 1160.

The wireless communication unit 1110 is used for the portable controller 10 to communicate with the master devices 20 and 30, and may include at least one of a wireless Internet module and a short-range communication module.

The first user input unit 1123 is for receiving information from the user. When information or a user command is input through the first user input unit 1123, the first processor 1180 processes the input information or user command to correspond to the input information or user command. The operation of the portable controller 10 can be controlled. The first user input unit 1123 includes a mechanical input means (or a mechanical key, for example, a button located on the front/rear or side of the basic controller 100, a dome switch, jog wheel, jog switch, etc.) and touch input means.

The first user input unit 1123 may include at least one of the trackball 121, the jog stick 122, the function key 123, and the trigger button 125.

The first sensor 1140 may include at least one of an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, and a motion sensor. You can. The first processor 1180 may track the movement of at least 3-Degrees of Freedom (DoF) of the portable controller 10 through the first sensor 1140.

The first haptic module 1153 generates vibration as various tactile effects that the user can feel. The intensity and pattern of vibration generated by the first haptic module 1153 may be controlled by user selection or settings of the first processor 1180. For example, the first haptic module 1153 may synthesize and output different vibrations or output them sequentially.

The first interface unit 1160 serves as a passageway for all external devices (eg, expansion module 200) connected to the basic controller 100. The first interface unit 1160 receives data from an external device, supplies power to an external device, or transmits data inside the basic controller 100 to an external device. For example, a wired/wireless data port, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, etc. are connected to the first interface unit 160. may be included.

The first processor 1180 can control the overall operation of the portable controller 10. FIG. 16 illustrates that the first processor 1180 includes a Micro Controller Unit (MCU) and a Bluetooth Low Energy (BLE) control chip for short-distance communication.

Hereinafter, we will look at the expansion module 200.

The second power unit 2190 may receive power from the basic controller 100 through the second interface 2160 and supply power to each component within the expansion module 200.

The second user input unit 2123 is for receiving information from the user. When information or a user command is input through the second user input unit 2123, the second processor 2180 is configured to respond to the input information or user command. The operation of the expansion module 200 can be controlled.

The optical output unit 2154 provides at least one of the position, movement, movement direction, rotation, rotation direction, and orientation direction of the portable controller 10 or the expansion module 200 to the master device 20, 30. Light can be output for notification purposes.

The light output unit 2154 may include at least one LED marker 224.

The second processor 2180 may control at least some of the components of the expansion module 200 in order to control the overall operation of the expansion module 200. FIG. 16 illustrates that the second processor 2180 is configured as an LED marker driver IC for driving the LED marker 224. That is, the LED marker driver IC can function as the second processor 2180.

The LED marker driver IC may include a memory 2181.

The memory 2181 may store expansion module identifier (ID) and LED marker placement information.

When the expansion module 200 is connected to the basic controller 100 (or when the portable controller 10 is turned on after the expansion module 200 and the basic controller 100 are interconnected), the second The processor 2180 may transmit the expansion module identifier (ID) and the LED marker placement information to the basic controller 100 through the second interface unit 2160.

Hereinafter, with reference to FIG. 17, we will look at the operation of the second haptic module 2153 of the portable controller 10 in more detail. Figure 17 is a detailed block diagram for explaining the operation of a portable controller according to one aspect of the present invention. In the detailed block diagram of FIG. 17, some components of the block diagram of FIG. 12 are omitted.

First, let's look at the basic controller 100.

When the first power unit 1190 supplies appropriate power to each component of the basic controller 100, it can supply power to the expansion module 200 through the first interface unit 1160.

The wireless communication unit 1110 is used for the portable controller 10 to communicate with the master devices 20 and 30, and may include at least one of a wireless Internet module and a short-range communication module.

The first user input unit 1123 is for receiving information from the user. When information or a user command is input through the first user input unit 1123, the first processor 1180 responds to the input information or user command. The operation of the portable controller 10 can be controlled. The first user input unit 1123 includes a mechanical input means (or a mechanical key, for example, a button located on the front/rear or side of the basic controller 100, a dome switch, jog wheel, jog switch, etc.) and touch input means.

The first user input unit 1123 may include at least one of the trackball 121, the jog stick 122, the function key 123, and the trigger button 125.

The first sensor 1140 may include at least one of an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, and a motion sensor. You can. The first processor 1180 may track the movement of at least 3-Degrees of Freedom (DoF) of the portable controller 10 through the first sensor 1140.

The first haptic module 1153 generates vibration as various tactile effects that the user can feel. The intensity and pattern of vibration generated by the first haptic module 1153 may be controlled by user selection or settings of the first processor 1180. For example, the first haptic module 1153 may synthesize and output different vibrations or output them sequentially.

The first interface unit 1160 serves as a passageway for all external devices (eg, expansion module 200) connected to the basic controller 100. The first interface unit 1160 receives data from an external device, supplies power to an external device, or transmits data inside the basic controller 100 to an external device. For example, a wired/wireless data port, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, etc. are connected to the first interface unit 160. may be included.

The first processor 1180 can control the overall operation of the portable controller 10. FIG. 16 illustrates that the first processor 1180 includes a Micro Controller Unit (MCU) and a Bluetooth Low Energy (BLE) control chip for short-distance communication.

The first processor 1180 may transmit a vibration-related control signal to the expansion module 200 through the first interface unit 1160. The vibration-related control signal may be stored in a memory within the basic controller 100 or may be received from the master devices 20 and 30.

Hereinafter, we will look at the expansion module 200.

The second power unit 2190 may receive power from the basic controller 100 through the second interface 2160 and supply power to each component within the expansion module 200.

The second user input unit 2123 is for receiving information from the user. When information or a user command is input through the second user input unit 2123, the second processor 2180 is configured to respond to the input information or user command. The operation of the expansion module 200 can be controlled.

The second haptic module 2153 generates various tactile effects that the user can feel. A representative example of a tactile effect generated by the second haptic module 2153 may be vibration. The intensity and pattern of vibration generated by the second haptic module 2153 may be controlled by the user's selection or settings of the second processor 2180.

The second haptic module 2153 may include at least one impact actuator to output a larger vibration than the first haptic module 1153. 17 illustrates that the second haptic module 2153 is provided with a first impact actuator 2153 and a second impact actuator 2154. For example, the second haptic module 2153 may synthesize and output different vibrations through each of the first impact actuator 2153 and the second impact actuator 2154, or output them sequentially.

The second processor 2180 may control at least some of the components of the expansion module 200 in order to control the overall operation of the expansion module 200. In FIG. 17, it is illustrated that the second processor 2180 is configured as an impact actuator driver IC for driving the second haptic module 2156. That is, the impact actuator driver IC can serve as the second processor 2180.

The impact actuator driver IC may include a memory 2181.

The memory 2181 may store information about vibration patterns and frequencies in addition to the expansion module identifier (ID).

The second processor 2180 controls the second haptic module 2153 to output vibration according to the vibration pattern and frequency corresponding to the vibration-related control signal received from the basic controller 100 with reference to the memory 2181. can do.

The vibration may be provided to the user along with audio (eg, sound effect) through the second sound output unit 2152.

In the above, it has been described that the LED marker 224 is provided in the expansion module 200 so that the master devices 20 and 30 can track the 6-DoF movement of the portable controller 10. However, in order to track the 6-DoF movement of the portable controller 10, the expansion module 200 may be equipped with a plurality of cameras instead of the LED marker 224. This will be explained further with reference to FIG. 18. Figure 18 is a top view, left side view, and right side view of the expansion module of the portable controller according to one aspect of the present invention.

The expansion module 200 may be provided with a plurality of cameras 2121 instead of the LED marking 220.

The camera 2121 may include at least one of a camera sensor (eg, CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

For example, a first camera 2121A is provided on the upper side of the expansion module 200 to photograph the upper outer direction, and a second camera 2121A is provided on the left side to photograph the left outer direction of the expansion module 200. A camera 2121B may be provided, and a third camera 2121C may be provided on the right side of the expansion module 200 to capture images of the right outer direction.

The first camera 2121A is mounted on the expansion module 200 to have an upward view angle of at least 120 degrees, and the second camera 2121B is mounted on the expansion module 200 to have a left and bottom view angle of at least 120 degrees. , the third camera 2121C may be mounted on the expansion module 200 to have a lower-right viewing angle of at least 120 degrees.

The first to third cameras 2121A, 2121B, and 2121C can secure a 360-degree field of view.

The expansion module 200 may be equipped with fewer or more cameras. If two cameras are provided, each camera can have a field of view of at least 180 degrees.

Hereinafter, with reference to FIG. 19, we will look at the interaction between the portable controller 10 according to FIG. 18 and the master device linked thereto in more detail. Figure 19 shows an interactive operation process between a portable controller and a master device linked thereto according to an embodiment of the present invention.

The expansion module 200 is mounted on the basic controller 100, so that the basic controller 100 and the expansion module 200 can be interconnected [S191]. The basic controller 100 and the expansion module 200 may be wired to each other through a first interface unit 1160 and a second interface unit 2160.

When the expansion module 200 is connected to the basic controller 100, the basic (or when the portable controller 10 is turned on after the expansion module 200 and the basic controller 100 are interconnected), The expansion module 200 can be recognized by the basic controller 100. Then, the expansion module 200 can transmit its expansion module identifier (ID) to the basic controller 100 [S192]. Alternatively, the basic controller 100 may read the expansion module identifier (ID) from the expansion module 200. The expansion module identifier (ID) is used to identify the type of the expansion module 200 described above. At this time, the expansion module 200 may transmit parameter information of the first to third cameras 2121A, 2121B, and 2121C to the basic controller 100. The parameter information may include at least one camera characteristic information among manufacturer information, angle of view information, and physical placement information on the expansion module 200 for each of the first to third cameras for SLAM operation described below. .

The basic controller 100 may transmit the expansion module identifier (ID) to the master devices 20 and 30 through short-distance communication (eg, Bluetooth communication) [S193].

If the short-range communication is connected between the portable controller 10 and the master devices 20 and 30 before step S192, step S193 can be performed immediately after step S192.

If the short-range communication is not connected between the portable controller 10 and the master devices 20 and 30 before step S192, step S193 may be performed after step S192 and establishing the short-range communication connection.

The master devices 20 and 30 may identify the expansion module 200 based on the expansion module identifier (ID) [S194].

Meanwhile, the expansion module 200 can capture the first to third images in real time through the first to third cameras 2121A, 2121B, and 2121C [S195].

The expansion module 200 can provide the first to third images to the basic controller 100 in real time [S196].

Apart from the expansion module 200 shooting the first to third images in real time, the basic controller 100 can acquire 3-DoF real-time motion information through the first sensor 1140 [S197 ].

The basic controller 100 may calculate the 6-DoF real-time movement of the portable controller 10 based on the 3-DoF real-time movement information and the first to third images captured in real time [S198]. .

That is, the basic controller 100 uses SLAMS (SLAMS) for calculating real-time motion information of the expansion module 200 or the portable controller 10 based on the motion information of a feature point or a specific object in the first to third images. Simultaneous Localization and Mapping) calculations can be performed. The basic controller 100 calculates the 6-DoF real-time movement of the portable controller 10 based on the 3-DoF real-time movement information and the real-time movement information of the expansion module 200 or the portable controller 10. can

The basic controller 100 performs the 6- DoF real-time motion information can be transmitted to the master devices 20 and 30 through short-distance communication [S199].

The master devices 20 and 30 may reflect the 6-DoF real-time movement of the portable controller 10 on the display screen [S200]. At this time, the master devices 20 and 30 may further reflect the user input when reflecting the 6-DoF real-time movement of the portable controller 10 on the display screen.

Hereinafter, with reference to FIG. 20, we will look at the operation of the camera 2121 of the portable controller 10 in more detail. Figure 20 is a detailed block diagram for explaining the operation of a portable controller according to an aspect of the present invention. In the detailed block diagram of FIG. 20, some components of the block diagram of FIG. 12 are omitted.

First, let's look at the basic controller 100.

When the first power unit 1190 supplies appropriate power to each component of the basic controller 100, it can supply power to the expansion module 200 through the first interface unit 1160.

The wireless communication unit 1110 is used for the portable controller 10 to communicate with the master devices 20 and 30, and may include at least one of a wireless Internet module and a short-range communication module.

The first user input unit 1123 is for receiving information from the user. When information or a user command is input through the first user input unit 1123, the first processor 1180 processes the input information or user command to correspond to the input information or user command. The operation of the portable controller 10 can be controlled. The first user input unit 1123 includes a mechanical input means (or a mechanical key, for example, a button located on the front/rear or side of the basic controller 100, a dome switch, jog wheel, jog switch, etc.) and touch input means.

The first user input unit 1123 may include at least one of the trackball 121, the jog stick 122, the function key 123, and the trigger button 125.

The first sensor 1140 may include at least one of an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, and a motion sensor. You can. The first processor 1180 may track the movement of at least 3-Degrees of Freedom (DoF) of the portable controller 10 through the first sensor 1140.

The first haptic module 1153 generates vibration as various tactile effects that the user can feel. The intensity and pattern of vibration generated by the first haptic module 1153 may be controlled by user selection or settings of the first processor 1180. For example, the first haptic module 1153 may synthesize and output different vibrations or output them sequentially.

The first interface unit 1160 serves as a passageway for all external devices (eg, expansion module 200) connected to the basic controller 100. The first interface unit 1160 receives data from an external device, supplies power to an external device, or transmits data inside the basic controller 100 to an external device. For example, a wired/wireless data port, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, etc. are connected to the first interface unit 160. may be included.

The first processor 1180 can control the overall operation of the portable controller 10. FIG. 16 illustrates that the first processor 1180 includes an application processor (AP) and a Bluetooth Low Energy (BLE) control chip for short-distance communication.

The first processor 1180 may receive the first to third images in real time from the expansion module 200 through the first interface unit 1160.

The first processor 1180 may calculate the 6-DoF real-time movement of the portable controller 10 based on the 3-DoF real-time movement information and the first to third images captured in real time.

That is, the first processor 1180 uses SLAMS (SLAMS) for calculating real-time movement information of the expansion module 200 or the portable controller 10 based on the movement information of a feature point or a specific object in the first to third images. Simultaneous Localization and Mapping) calculations can be performed. The first processor 1180 calculates the 6-DoF real-time movement of the portable controller 10 based on the 3-DoF real-time movement information and the real-time movement information of the expansion module 200 or the portable controller 10. can

The first processor 1180 performs the 6- DoF real-time motion information can be transmitted to the master devices 20 and 30 through the wireless communication unit 1110.

Hereinafter, we will look at the expansion module 200.

The second power unit 2190 may receive power from the basic controller 100 through the second interface 2160 and supply power to each component within the expansion module 200.

The second user input unit 2123 is for receiving information from the user. When information or a user command is input through the second user input unit 2123, the second processor 2180 responds to the input information or user command. The operation of the expansion module 200 can be controlled.

The camera 2121 is equipped with the above-described first to third cameras 2121A, 2121B, and 2121C and can capture the first to third images in real time.

The second processor 2180 may control at least some of the components of the expansion module 200 in order to control the overall operation of the expansion module 200. FIG. 20 illustrates that the second processor 2180 is configured as a Field Programmable Gate Array (FPGA). That is, the FPGA can function as a second processor (2180).

The FPGA may include memory 2181.

The memory 2181 may store an expansion module identifier (ID) and parameter information of the first to third cameras 2121A, 2121B, and 2121C. The parameter information may include camera characteristic information such as angle of view information for each of the first to third cameras, physical placement information on the expansion module 200, etc.

When the expansion module 200 is connected to the basic controller 100 (or when the portable controller 10 is turned on after the expansion module 200 and the basic controller 100 are interconnected), the second The processor 2180 can transmit the expansion module identifier (ID) and parameter information of the first to second cameras 2121A, 2121B, and 2121C to the basic controller 100 through the second interface unit 2160. there is.

In addition, the second processor 2180 processes the first to third images captured in real time through the first to third cameras 2121A, 2121B, and 2121C through the first interface unit 2160 to the basic controller ( 100) can be provided.

below. With reference to FIG. 21 , we will look in more detail at the case where the second sensor 2140 of the expansion module 200 is a temperature sensor. Of course, it goes without saying that the second sensor 2140 may be an environmental sensor or a chemical sensor other than a temperature sensor. Figure 21 is a plan view and a front view of an expansion module of a portable controller according to an aspect of the present invention.

The temperature sensor 2140 may be provided as a second sensor on one side of the expansion module 200. In FIG. 21, the temperature sensor 2140 is illustrated as being provided on the front of the expansion module 200, but it may also be provided on the other side. The temperature sensor 2140 may be an infrared-type non-contact temperature sensor.

The portable controller 10 configured in this way can also be used for remote or non-face-to-face medical treatment together with the master devices 20 and 30 linked thereto.

Hereinafter, with reference to FIG. 22, we will look at the operation of the temperature sensor 2140 of the portable controller 10 in more detail. Figure 22 is a detailed block diagram for explaining the operation of a portable controller according to one aspect of the present invention. In the detailed block diagram of FIG. 22, some components of the block diagram of FIG. 12 are omitted.

First, let's look at the basic controller 100.

When the first power unit 1190 supplies appropriate power to each component of the basic controller 100, it can supply power to the expansion module 200 through the first interface unit 1160.

The wireless communication unit 1110 is used for the portable controller 10 to communicate with the master devices 20 and 30, and may include at least one of a wireless Internet module and a short-range communication module.

The first user input unit 1123 is for receiving information from the user. When information or a user command is input through the first user input unit 1123, the first processor 1180 processes the input information or user command to correspond to the input information or user command. The operation of the portable controller 10 can be controlled. The first user input unit 1123 may include a mechanical input means (or a mechanical key, for example, a button located on the front/rear or side of the basic controller 100, a dome switch, jog wheel, jog switch, etc.) and touch input means.

The first user input unit 1123 may include at least one of the trackball 121, the jog stick 122, the function key 123, and the trigger button 125.

The first sensor 1140 may include at least one of an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, and a motion sensor. You can. The first processor 1180 may track the movement of at least 3-Degrees of Freedom (DoF) of the portable controller 10 through the first sensor 1140.

The first haptic module 1153 generates vibration as various tactile effects that the user can feel. The intensity and pattern of vibration generated by the first haptic module 1153 may be controlled by user selection or settings of the first processor 1180. For example, the first haptic module 1153 may synthesize and output different vibrations or output them sequentially.

The first interface unit 1160 serves as a passageway for all external devices (eg, expansion module 200) connected to the basic controller 100. The first interface unit 1160 receives data from an external device, supplies power to an external device, or transmits data inside the basic controller 100 to an external device. For example, a wired/wireless data port, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, etc. are connected to the first interface unit 160. may be included.

The first processor 1180 can control the overall operation of the portable controller 10. FIG. 21 illustrates that the first processor 1180 includes a Micro Controller Unit (MCU) and a Bluetooth Low Energy (BLE) control chip for short-distance communication.

The first processor 1180 may transmit a control signal related to temperature measurement to the expansion module 200 through the first interface unit 1160. The temperature measurement-related control signal may be stored in a memory within the basic controller 100, or may be received from the master device 20, 30 through the wireless communication unit 1110.

The first processor 1180 may transmit temperature data received from the expansion module 200 through the first interface unit 1160 to the master devices 20 and 30 through the wireless communication unit 1110. At this time, the first processor 1180 transmits the characteristic information of the temperature sensor received from the expansion module 200 through the first interface unit 1160 to the master devices 20 and 30 through the wireless communication unit 1110. You can also send it together.

Hereinafter, we will look at the expansion module 200.

The second power unit 2190 may receive power from the basic controller 100 through the second interface 2160 and supply power to each component within the expansion module 200.

The second user input unit 2123 is for receiving information from the user. When information or a user command is input through the second user input unit 2123, the second processor 2180 is configured to respond to the input information or user command. The operation of the expansion module 200 can be controlled.

The second processor 2180 may control at least some of the components of the expansion module 200 in order to control the overall operation of the expansion module 200. FIG. 20 illustrates that the second processor 2180 is configured as a temperature sensor driver IC for driving the temperature sensor 2140. That is, the temperature sensor driver IC can function as the second processor 2180.

In FIG. 22, the temperature sensor 2140 and the temperature sensor driver IC 2180 are shown as separate and independent components, but they may be implemented as a single component.

The temperature sensor driver IC may include memory 2181.

The memory 2181 may include an expansion module identifier (ID) and temperature sensor characteristic information. The temperature sensor characteristic information may include at least one of manufacturer information and temperature sensing calibration information.

When the expansion module 200 is connected to the basic controller 100 (or when the portable controller 10 is turned on after the expansion module 200 and the basic controller 100 are interconnected), the second The processor 2180 may transmit the expansion module identifier (ID) and the temperature sensor characteristic information to the basic controller 100 through the second interface unit 2160.

Additionally, the second processor 2180 may transmit temperature data sensed through the temperature sensor 2140 to the basic controller 100 through the second interface unit 2160.

As discussed above, according to at least one embodiment of the present invention, the function can be expanded and changed by easily attaching and detaching an expansion module to the portable controller 10.

The above detailed description should not be construed as restrictive in any respect and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (20)

1. default controller; and

   Includes an expansion module detachable from the basic controller,

   The basic controller is,

   First interface unit;

   a wireless communication unit for communicating with the master device; and

   A first processor that receives the identifier of the expansion module from the expansion module through a first interface unit and controls the basic controller to operate to match the type of the expansion module,

   The expansion module is,

   second interface unit; and

   A portable controller comprising a second processor that controls to provide the identifier of the expansion module to the basic controller through a second interface unit.
2. The method of claim 1, wherein the first processor:

   When the expansion module is mounted on the basic controller and the portable controller is turned on, the portable controller is controlled to receive the identifier from the expansion module.
3. The method of claim 1, wherein the expansion module:

   Further comprising an optical output unit including a plurality of optical element markers,

   The second processor is,

   A portable controller, characterized in that control to provide arrangement information of the plurality of optical element markers to the basic controller through a second interface unit.
4. The method of claim 3, wherein the plurality of optical element markers are:

   A portable controller comprising a plurality of infrared LED markers.
5. The method of claim 3, wherein the first processor:

   A portable controller, characterized in that control to transmit arrangement information of the plurality of optical element markers to the master device.
6. The method of claim 5, wherein the master device:

   A portable controller comprising a wearable XR device and an external display device.
7. The method of claim 5, wherein the arrangement information of the plurality of optical element markers is:

   A portable controller, wherein the plurality of optical element markers include three-dimensional spatial coordinate information arranged in the expansion module.
8. The method of claim 5, wherein the basic controller is:

   It further includes a first sensor for sensing real-time movement of the 3-DoF (Degrees of Freedom) of the basic controller,

   The first processor is:

   A portable controller characterized in that it controls to transmit the sensed 3-DoF real-time movement to the master device.
9. The method of claim 8, wherein the first sensor:

   A portable controller comprising at least one of an acceleration sensor, a magnetic sensor, a gravity sensor, a gyroscope sensor, and a motion sensor.
10. The method of claim 1, wherein the expansion module:

    Further comprising a plurality of cameras,

    The second processor is,

    A portable controller, characterized in that it controls to provide characteristic information of the plurality of cameras to the basic controller.
11. The method of claim 10, wherein the basic controller is:

    It further includes a first sensor for sensing real-time movement of the 3-DoF (Degrees of Freedom) of the basic controller,

    The first processor is:

    Calculating 6-DoF real-time movement of the portable controller based on the 3-DoF real-time movement and a plurality of camera images captured by the plurality of cameras received from the expansion module,

    A portable controller, characterized in that control to transmit the calculated 6-DoF real-time movement to the master device.
12. 12. The method of claim 11, wherein the first processor:

    A portable controller, characterized in that control to calculate real-time movement information of the portable controller based on movement information of feature points in the plurality of camera images.
13. The method of claim 10, wherein the plurality of cameras

    A portable controller comprising three cameras with a field of view of at least 120 degrees.
14. The method of claim 1, wherein the expansion module:

    Includes an impact actuator,

    The second processor is,

    A portable controller comprising an impact actuator driver IC.
15. 15. The method of claim 14, wherein the impact actuator driver IC is:

    Includes a memory to contain information about the identifier and vibration pattern and frequency of the expansion module,

    A portable controller, characterized in that the impact actuator is controlled to output vibration according to a vibration pattern and frequency corresponding to a vibration-related control signal received from the basic controller.
16. The method of claim 1, wherein the expansion module:

    Includes a temperature sensor,

    The second processor is,

    A portable controller comprising a temperature sensor driver IC.
17. 15. The method of claim 14, wherein the temperature sensor driver IC is:

    Includes a memory to contain the identifier and temperature sensor characteristic information of the expansion module,

    A portable controller, characterized in that control to provide the temperature sensor characteristic information and temperature data sensed through the temperature sensor to the basic controller.
18. 18. The method of claim 17, wherein the first processor:

    A portable controller, characterized in that it controls to transmit the temperature center characteristic information and the temperature data to the master device.
19. According to claim 1,

    The basic controller is

    A handle part that extends in the longitudinal direction and is held by the user's hand;

    An operating body located in front of the handle portion;

    a first user input unit located on the operation body;

    a first connector for a first interface formed on the operating body; and

    It includes a first coupling rail formed on the operating body,

    The expansion module is,

    a module body including a coupling portion in contact with the manipulation body;

    a second connector for a second interface unit formed on the coupling unit and coupled to the first connector; and

    A portable controller including a second coupling rail that is slide-fastened in the longitudinal direction of the first coupling rail.
20. According to clause 19,

    The first connector is located in the front of the manipulation body,

    The first coupling rail is located in the left and right directions of the operating body,

    A portable controller, characterized in that the coupling portion of the expansion module has a U-shape surrounding the front and left and right sides of the operation body.

Images