KR20230042285A - Route guidance device and its route guidance method - Google Patents

Abstract

The present invention provides a route guidance device mounted on a vehicle and implementing augmented reality and a route guidance method thereof. A route guidance device according to an embodiment of the present invention includes a communication unit receiving a first image captured by a camera, performing calibration on the first image, and guiding vehicle driving to a calibrated second image and a processor that outputs overlapping graphic objects.

Description

Route guidance device and its route guidance method

The present invention relates to a route guidance device mounted on a vehicle and implementing augmented reality and a route guidance method thereof.

A vehicle refers to a means of transportation capable of moving people or loads by using kinetic energy. Representative examples of vehicles include automobiles and motorcycles.

For the safety and convenience of users who use vehicles, vehicles are equipped with various sensors and devices, and the functions of vehicles are diversifying.

Vehicle functions may be divided into a convenience function for promoting driver's convenience and a safety function for promoting driver's and/or pedestrian's safety.

First, convenience functions have development motives related to driver convenience, such as giving infotainment (information + entertainment) functions to vehicles, supporting partial autonomous driving functions, or helping to secure the driver's vision such as night vision or blind spots. . For example, active cruise control (ACC), smart parking assist system (SPAS), night vision (NV), head up display (HUD), around view monitor (around view monitor, AVM) and adaptive headlight system (AHS) functions.

Safety features are technologies that ensure driver safety and/or pedestrian safety, such as lane departure warning system (LDWS), lane keeping assist system (LKAS), and autonomous emergency braking. braking, AEB) functions, etc.

Vehicle-specific communication technology is being developed so that the above convenience and safety functions can be further improved. For example, V2I (Vehicle to Infrastructure), which enables communication between vehicles and infrastructure, V2V (Vehicle to Vehicle), which enables communication between vehicles, and V2X (Vehicle to Everything), which enables communication between vehicles and objects, etc. there is

Vehicles are provided with route guidance devices for visually providing various information to passengers. The route guidance device includes a head-up display (HUD) that outputs information to a windshield of a vehicle or a separately provided transparent screen and/or various displays that output information through a panel.

The route guidance device provides route guidance information to a destination and information on a point of interest (POI), and is evolving toward effectively providing a variety of information. In particular, research on a route guidance device that directly and effectively provides necessary information to a driver who needs to concentrate on driving within a range that does not interfere with driving is ongoing.

The present invention aims to solve the foregoing and other problems.

One object of the present invention is to provide a route guidance device and a route guidance method capable of providing a route for a vehicle to travel in augmented reality in an optimized manner.

Another object of the present invention is to provide a route guidance device capable of providing passengers with various types of information in augmented reality and a route guidance method thereof.

Another object of the present invention is to provide a route guidance device capable of guiding a driving route of a vehicle in augmented reality optimized according to a situation in which the vehicle is driving.

The present invention provides a route guidance device for providing a route along which a vehicle must travel, and a route guidance method thereof.

A route guidance device according to an embodiment of the present invention includes a communication unit receiving a first image captured by a camera, performing calibration on the first image, and guiding vehicle driving to a calibrated second image and a processor that outputs overlapping graphic objects.

In an embodiment, the processor outputs a first graphic object related to driving to the first image before the calibration is performed, and outputs the first graphic object and a second graphic related to driving after the calibration is performed. It is characterized in that the object is overlapped with the second image and outputted.

In an embodiment, the first graphic object includes turn-by-turn information indicating a road to be entered in front of a predetermined distance, and the second graphic object includes the second image It is characterized in that it includes a carpet image overlapping the road included in and guiding a planned driving path of the vehicle.

In an embodiment, the processor overlaps and outputs the second graphic object on a lane in which the vehicle is driving among a plurality of lanes included in the second image, and when it is determined that the vehicle leaves the lane in which the vehicle is driving, It is characterized in that the second graphic object is not output.

In an embodiment, the first graphic object is characterized in that an output is maintained regardless of whether the vehicle leaves the driving lane.

In an embodiment, the processor may perform calibration so that a road area included in the first image has a predetermined size or more.

In an embodiment, the processor may, when the viewing angle of the second image is changed so that a road included in the second image and a graphic object overlapping the road are rotated by driving of the vehicle, the second image having a changed viewing angle Characterized in that the graphic object is changed to match the lane based on .

In an embodiment, the processor outputs a carpet image overlapping a road included in the second image and guiding a planned driving path of the vehicle to the second image, and the carpet image indicates that the vehicle in the second image It is characterized in that the wall image overlaps with the driving lane, and the processor outputs a wall image guiding a driving direction to a lane adjacent to the driving lane in the second image.

In an embodiment, the processor outputs the carpet image to a lane on which the vehicle is traveling before a predetermined distance prior to entering an intersection where the vehicle is to change direction, and the vehicle is driven by the predetermined distance based on the intersection. When entering within the vehicle, the carpet image is changed to the wall image, and the wall image is overlapped with a road adjacent to the road on which the vehicle is driving and output.

In an embodiment, the processor is characterized in that, when the vehicle performs a direction change at the intersection, the wall image is changed to the carpet image, and the carpet image is overlapped and output to the lane on which the vehicle is traveling. .

In an embodiment, the processor may enlarge the output size of the wall image as the distance between the vehicle and the intersection becomes closer.

In an embodiment, the processor may overlap and output a compass image including a compass object indicating a direction from which the front of the vehicle is viewed to the second image.

In an embodiment, the compass image may include a fixed carpet image guiding a direction in which the vehicle should travel from the current location.

In an embodiment, the processor may vary a display position of the fixed carpet image along an edge of the compass object.

In an embodiment, the processor may, based on the distance between the vehicle and an intersection where the vehicle should change direction, use a carpet image for guiding a planned driving path of the vehicle or a compass that informs the direction of the front of the vehicle. It is characterized in that any one of the images is output.

In an embodiment, the processor determines the slope of the road on which the vehicle travels, and determines the slope of the first graphic object to overlap and output the first image based on the determined slope of the road. to be

A route guidance method of a route guidance device according to another embodiment of the present invention includes the steps of receiving a first image captured through a camera, performing calibration on the first image, and a vehicle in the calibrated second image. and overlapping and outputting a graphic object that guides the driving of the vehicle.

In an embodiment, the outputting may include outputting a first graphic object related to driving to the first image before the calibration is performed, and outputting the first graphic object and a first graphic object related to driving to the first image after the calibration is performed. It is characterized in that 2 graphic objects are overlapped with the second image and output.

In an embodiment, the first graphic object includes turn-by-turn information indicating a road to be entered in front of a predetermined distance, and the second graphic object includes the second image It is characterized in that it includes a carpet image overlapping the lane included in and guiding a planned driving path of the vehicle.

In an embodiment, the outputting may include overlapping and outputting the second graphic object on a lane in which the vehicle is driving among a plurality of lanes included in the image, and when it is determined that the vehicle leaves the lane in which the vehicle is driving, It is characterized in that the second graphic object is not output.

Effects of the route guidance device and the route guidance method according to the present invention are described as follows.

According to the present invention, the passenger may be provided with information on a route on which the vehicle autonomously drives or on which the driver should drive in units of lanes through a carpet image.

In addition, according to the present invention, a passenger can determine a path on which a vehicle should travel in an optimized way through images of various types of carpets.

In addition, according to the present invention, the present invention can provide a new route guidance interface capable of guiding a driving route of a vehicle through a compass image.

1 is a view showing the exterior of a vehicle according to an embodiment of the present invention.
2 is a view of a vehicle according to an embodiment of the present invention viewed from various external angles.
3 and 4 are views illustrating the interior of a vehicle according to an embodiment of the present invention.
5 and 6 are referenced diagrams for explaining objects according to an embodiment of the present invention.
7 is a block diagram referred to describe a vehicle according to an embodiment of the present invention.
8 is a block diagram for explaining a mobile terminal related to the present invention.
9 is a block diagram for explaining a route guidance device according to the present invention.
10a, 10b, 10c, 10d, 10e, 10f, 11a, 11b, 11c, 11d, 11e, 11f and 11g show augmented reality according to an embodiment of the present invention. It is a conceptual diagram to explain the applied route guidance method.
12, 13A and 13B are flowcharts for explaining a method of determining a type of an image output to augmented reality according to a driving state of a vehicle according to an embodiment of the present invention.
14, 15, 16, 17, 18, 19, 20, 21, 22, and 23 are conceptual diagrams for explaining a compass image output in augmented reality according to an embodiment of the present invention. am.
24a, 24b, 25, 26, 27a and 27b are conceptual diagrams for explaining a route guidance method according to another embodiment of the present invention.
28 is a conceptual diagram illustrating a method of outputting turn-by-turn information according to an embodiment of the present invention.
29a, 29b, 29c, and 29d are conceptual views illustrating various methods of outputting a compass image according to an embodiment of the present invention.
30, 31 and 32 are conceptual diagrams for explaining a route guidance method to which augmented reality is applied according to the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

A vehicle described in this specification may be a concept including a car and a motorcycle. Hereinafter, vehicles will be mainly described with respect to vehicles.

The vehicle described in this specification may be a concept including all of an internal combustion engine vehicle having an engine as a power source, a hybrid vehicle having an engine and an electric motor as a power source, and an electric vehicle having an electric motor as a power source.

In the following description, the left side of the vehicle means the left side of the driving direction of the vehicle, and the right side of the vehicle means the right side of the driving direction of the vehicle.

1 is a view showing the exterior of a vehicle according to an embodiment of the present invention.

2 is a view of a vehicle according to an embodiment of the present invention viewed from various external angles.

3 and 4 are views illustrating the interior of a vehicle according to an embodiment of the present invention.

5 and 6 are referenced diagrams for explaining objects according to an embodiment of the present invention.

7 is a block diagram referred to describe a vehicle according to an embodiment of the present invention.

Referring to FIGS. 1 to 7 , the vehicle 100 may include wheels rotated by a power source and a steering input device 510 for adjusting the driving direction of the vehicle 100 .

Vehicle 100 may be an autonomous vehicle.

The vehicle 100 may switch to an autonomous driving mode or a manual mode based on a user input.

For example, the vehicle 100 may switch from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode based on a user input received through the user interface device 200 .

The vehicle 100 may switch to an autonomous driving mode or a manual mode based on driving situation information. Driving situation information may be generated based on object information provided by the object detection device 300 .

For example, the vehicle 100 may switch from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode based on driving situation information generated by the object detection device 300 .

For example, the vehicle 100 may switch from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode based on driving situation information received through the communication device 400 .

The vehicle 100 may switch from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode based on information, data, and signals provided from an external device.

When the vehicle 100 is operated in the autonomous driving mode, the autonomous vehicle 100 may be operated based on the driving system 700 .

For example, the autonomous vehicle 100 may operate based on information, data, or signals generated by the driving system 710 , the vehicle exit system 740 , and the parking system 750 .

When the vehicle 100 is operated in the manual mode, the autonomous vehicle 100 may receive a user input for driving through the driving control device 500 . Based on the user input received through the driving control device 500, the vehicle 100 may be driven.

The overall length means the length from the front part to the rear part of the vehicle 100, the width means the width of the vehicle 100, and the height means the length from the lower part of the wheel to the roof. In the following description, the overall length direction (L) is the standard direction for measuring the overall length of the vehicle 100, the overall width direction (W) is the standard direction for measuring the overall width of the vehicle 100, and the overall height direction (H) is the vehicle It may mean a direction that is a standard for measuring the total height of (100).

As illustrated in FIG. 7 , the vehicle 100 includes a user interface device 200, an object detection device 300, a communication device 400, a driving manipulation device 500, a vehicle driving device 600, and a driving system. 700 , a navigation system 770 , a sensing unit 120 , a vehicle interface unit 130 , a memory 140 , a control unit 170 and a power supply unit 190 .

Depending on embodiments, the vehicle 100 may further include components other than the components described herein, or may not include some of the components described herein.

The user interface device 200 is a device for communication between the vehicle 100 and a user. The user interface device 200 may receive a user input and provide information generated in the vehicle 100 to the user. The vehicle 100 may implement UI (User Interfaces) or UX (User Experience) through the user interface device 200 .

The user interface device 200 may include an input unit 210, an internal camera 220, a biological sensor 230, an output unit 250, and a processor 270.

Depending on embodiments, the user interface device 200 may further include components other than the described components or may not include some of the described components.

The input unit 200 is for receiving information from a user, and the data collected by the input unit 120 is analyzed by the processor 270 and processed as a user's control command.

The input unit 200 may be disposed inside a vehicle. For example, the input unit 200 may include one area of a steering wheel, one area of an instrument panel, one area of a seat, one area of each pillar, and a door One area of the door, one area of the center console, one area of the head lining, one area of the sun visor, one area of the windshield or window It may be placed in one area or the like.

The input unit 200 may include a voice input unit 211 , a gesture input unit 212 , a touch input unit 213 , and a mechanical input unit 214 .

The voice input unit 211 may convert a user's voice input into an electrical signal. The converted electrical signal may be provided to the processor 270 or the controller 170 .

The voice input unit 211 may include one or more microphones.

The gesture input unit 212 may convert a user's gesture input into an electrical signal. The converted electrical signal may be provided to the processor 270 or the controller 170 .

The gesture input unit 212 may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input.

Depending on the embodiment, the gesture input unit 212 may detect a user's 3D gesture input. To this end, the gesture input unit 212 may include a light output unit outputting a plurality of infrared lights or a plurality of image sensors.

The gesture input unit 212 may detect a user's 3D gesture input through a time of flight (TOF) method, a structured light method, or a disparity method.

The touch input unit 213 may convert a user's touch input into an electrical signal. The converted electrical signal may be provided to the processor 270 or the controller 170 .

The touch input unit 213 may include a touch sensor for sensing a user's touch input.

Depending on the embodiment, the touch input unit 213 may be integrally formed with the display unit 251 to implement a touch screen. Such a touch screen may provide both an input interface and an output interface between the vehicle 100 and the user.

The mechanical input unit 214 may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit 214 may be provided to the processor 270 or the controller 170 .

The mechanical input unit 214 may be disposed on a steering wheel, a center fascia, a center console, a cockpit module, or a door.

The internal camera 220 may acquire an image inside the vehicle. The processor 270 may detect the user's state based on the vehicle interior image. The processor 270 may obtain gaze information of the user from the vehicle interior image. The processor 270 may detect a user's gesture from the vehicle interior image.

The biometric sensor 230 may obtain user's biometric information. The biometric sensor 230 may include a sensor capable of obtaining user's biometric information, and may obtain user's fingerprint information, heart rate information, and the like, using the sensor. Biometric information may be used for user authentication.

The output unit 250 is for generating an output related to sight, hearing or touch.

The output unit 250 may include at least one of a display unit 251 , an audio output unit 252 , and a haptic output unit 253 .

The display unit 251 may display graphic objects corresponding to various pieces of information.

The display unit 251 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. display), a 3D display, and an e-ink display.

The display unit 251 and the touch input unit 213 may form a mutual layer structure or be integrally formed to implement a touch screen.

The display unit 251 may be implemented as a Head Up Display (HUD). When the display unit 251 is implemented as a HUD, the display unit 251 may include a projection module to output information through an image projected on a windshield or window.

The display unit 251 may include a transparent display. The transparent display may be attached to a windshield or window.

The transparent display may display a predetermined screen while having a predetermined transparency. In order to have transparency, transparent displays are transparent TFEL (Thin Film Electroluminescent), transparent OLED (Organic Light-Emitting Diode), transparent LCD (Liquid Crystal Display), transmissive transparent display, transparent LED (Light Emitting Diode) display may include at least one of them. Transparency of the transparent display can be adjusted.

Meanwhile, the user interface device 200 may include a plurality of display units 251a to 251g.

The display unit 251 includes one area of the steering wheel, one area 521a, 251b, and 251e of the instrument panel, one area 251d of the seat, one area 251f of each pillar, and one area of the door ( 251g), one area of the center console, one area of the headlining, one area of the sun visor, or one area 251c of the windshield and one area 251h of the window.

The audio output unit 252 converts an electrical signal provided from the processor 270 or the controller 170 into an audio signal and outputs the converted audio signal. To this end, the sound output unit 252 may include one or more speakers.

The haptic output unit 253 generates a tactile output. For example, the haptic output unit 253 may vibrate the steering wheel, seat belt, and seats 110FL, 110FR, 110RL, and 110RR so that the user can recognize the output.

The processor 270 may control overall operations of each unit of the user interface device 200 .

Depending on embodiments, the user interface device 200 may include a plurality of processors 270 or may not include a plurality of processors 270 .

When the processor 270 is not included in the user interface device 200, the user interface device 200 may be operated under the control of a processor of another device in the vehicle 100 or the control unit 170.

Meanwhile, the user interface device 200 may be referred to as a vehicle display device.

The user interface device 200 may be operated under the control of the controller 170 .

The object detection device 300 is a device for detecting an object located outside the vehicle 100 .

Objects may be various objects related to driving of the vehicle 100 .

Referring to FIGS. 5 and 6 , objects O include lanes OB10, other vehicles OB11, pedestrians OB12, two-wheeled vehicles OB13, traffic signals OB14 and OB15, lights, roads, structures, May include speed bumps, terrain objects, animals, and the like.

The lane OB10 may be a driving lane, a lane adjacent to the driving lane, or a lane in which an opposing vehicle travels. The lane OB10 may be a concept including left and right lines forming the lane.

The other vehicle OB11 may be a vehicle running around the vehicle 100 . The other vehicle may be a vehicle located within a predetermined distance from the vehicle 100 . For example, the other vehicle OB11 may be a vehicle preceding or following the vehicle 100 .

The pedestrian OB12 may be a person located around the vehicle 100 . The pedestrian OB12 may be a person located within a predetermined distance from the vehicle 100 . For example, the pedestrian OB12 may be a person located on a sidewalk or road.

The two-wheeled vehicle OB12 may refer to a vehicle that is located around the vehicle 100 and moves using two wheels. The two-wheeled vehicle OB12 may be a vehicle having two wheels located within a predetermined distance from the vehicle 100 . For example, the two-wheeled vehicle OB13 may be a motorcycle or bicycle located on a sidewalk or road.

The traffic signal may include a traffic light OB15, a traffic sign OB14, and a pattern or text drawn on a road surface.

The light may be light generated from a lamp provided in another vehicle. The light may be light generated from a street lamp. The light may be sunlight.

The road may include a road surface, a curve, an incline such as an uphill or a downhill.

The structure may be an object located near a road and fixed to the ground. For example, structures may include streetlights, roadside trees, buildings, telephone poles, traffic lights, and bridges.

A feature may include a mountain, a hill, and the like.

Meanwhile, objects may be classified into moving objects and fixed objects. For example, the moving object may be a concept including other vehicles and pedestrians. For example, a fixed object may be a concept including traffic signals, roads, and structures.

The object detection device 300 may include a camera 310, a radar 320, a lidar 330, an ultrasonic sensor 340, an infrared sensor 350, and a processor 370.

Depending on embodiments, the object detection apparatus 300 may further include components other than the described components or may not include some of the described components.

The camera 310 may be located at an appropriate location outside the vehicle in order to obtain an external image of the vehicle. The camera 310 may be a mono camera, a stereo camera 310a, an around view monitoring (AVM) camera 310b, or a 360-degree camera.

For example, the camera 310 may be disposed in the interior of the vehicle and proximate to the front windshield to obtain an image of the front of the vehicle. Alternatively, the camera 310 may be disposed around a front bumper or a radiator grill.

For example, the camera 310 may be disposed in the interior of the vehicle and proximate to the rear glass to obtain an image behind the vehicle. Alternatively, the camera 310 may be disposed around a rear bumper, a trunk, or a tailgate.

For example, the camera 310 may be disposed close to at least one of the side windows in the interior of the vehicle to obtain an image of the side of the vehicle. Alternatively, the camera 310 may be disposed around side mirrors, fenders, or doors.

The camera 310 may provide the acquired image to the processor 370 .

The radar 320 may include an electromagnetic wave transmitter and a receiver. The radar 320 may be implemented in a pulse radar method or a continuous wave radar method in terms of radio wave emission principles. The radar 320 may be implemented in a frequency modulated continuous wave (FMCW) method or a frequency shift keyong (FSK) method according to a signal waveform among continuous wave radar methods.

The radar 320 detects an object based on a Time of Flight (TOF) method or a phase-shift method through electromagnetic waves, and detects the position of the detected object, the distance to the detected object, and the relative speed. can be detected.

The radar 320 may be disposed at an appropriate location outside the vehicle to detect an object located in front, rear or side of the vehicle.

LiDAR 330 may include a laser transmitter and a receiver. The LIDAR 330 may be implemented in a Time of Flight (TOF) method or a phase-shift method.

LiDAR 330 may be implemented as a driven or non-driven type.

When implemented in a driving manner, the lidar 330 is rotated by a motor and may detect an object around the vehicle 100 .

When implemented as a non-driving type, the lidar 330 may detect an object located within a predetermined range with respect to the vehicle 100 by light steering. The vehicle 100 may include a plurality of non-powered lidars 330 .

The LIDAR 330 detects an object based on a Time of Flight (TOF) method or a phase-shift method through a laser light medium, and determines the location of the detected object, the distance to the detected object, and the Relative speed can be detected.

The lidar 330 may be disposed at an appropriate location outside the vehicle in order to detect an object located in front, rear or side of the vehicle.

The ultrasonic sensor 340 may include an ultrasonic transmitter and receiver. The ultrasonic sensor 340 may detect an object based on ultrasonic waves, and detect a position of the detected object, a distance to the detected object, and a relative speed.

The ultrasonic sensor 340 may be disposed at an appropriate location outside the vehicle to detect an object located in front, rear or side of the vehicle.

The infrared sensor 350 may include an infrared transmitter and a receiver. The infrared sensor 340 may detect an object based on infrared light and detect a position of the detected object, a distance to the detected object, and a relative speed.

The infrared sensor 350 may be disposed at an appropriate location outside the vehicle in order to detect an object located in front, rear or side of the vehicle.

The processor 370 may control overall operations of each unit of the object detection device 300 .

The processor 370 may detect and track an object based on the obtained image. The processor 370 may perform operations such as calculating a distance to an object and calculating a relative speed with an object through an image processing algorithm.

The processor 370 may detect and track the object based on the reflected electromagnetic wave that is transmitted and reflected by the object and returned. The processor 370 may perform operations such as calculating a distance to an object and calculating a relative speed with an object based on electromagnetic waves.

The processor 370 may detect and track the object based on reflected laser light that is transmitted and reflected by the object. The processor 370 may perform operations such as calculating a distance to an object and calculating a relative speed to an object based on laser light.

The processor 370 may detect and track the object based on the reflected ultrasonic wave after the transmitted ultrasonic wave is reflected on the object. The processor 370 may perform operations such as calculating a distance with an object and calculating a relative speed with an object based on ultrasonic waves.

The processor 370 may detect and track the object based on the reflected infrared light that is transmitted and reflected by the object and returned. The processor 370 may perform operations such as calculating a distance to an object and calculating a relative speed with an object based on infrared light.

Depending on embodiments, the object detection device 300 may include a plurality of processors 370 or may not include the processor 370 . For example, each of the camera 310, the radar 320, the lidar 330, the ultrasonic sensor 340, and the infrared sensor 350 may individually include a processor.

When the processor 370 is not included in the object detection device 300, the object detection device 300 may be operated according to the control of the processor or the control unit 170 of the device in the vehicle 100.

The object detection device 400 may be operated according to the control of the controller 170 .

The communication device 400 is a device for communicating with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.

The communication device 400 may include at least one of a transmission antenna, a reception antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

The communication device 400 may include a short-distance communication unit 410, a location information unit 420, a V2X communication unit 430, an optical communication unit 440, a broadcast transmission/reception unit 450, and a processor 470.

Depending on embodiments, the communication device 400 may further include components other than the described components, or may not include some of the described components.

The short-range communication unit 410 is a unit for short-range communication. The short-range communication unit 410 includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), and Wireless (Wi-Fi). -Fidelity), Wi-Fi Direct, and wireless USB (Wireless Universal Serial Bus) technologies may be used to support short-distance communication.

The short-range communication unit 410 may perform short-range communication between the vehicle 100 and at least one external device by forming wireless area networks.

The location information unit 420 is a unit for acquiring location information of the vehicle 100 . For example, the location information unit 420 may include a Global Positioning System (GPS) module or a Differential Global Positioning System (DGPS) module.

The V2X communication unit 430 is a unit for performing wireless communication with a server (V2I: Vehicle to Infrastructure), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). The V2X communication unit 430 may include an RF circuit capable of implementing communication with infrastructure (V2I), vehicle-to-vehicle communication (V2V), and pedestrian communication (V2P) protocols.

The optical communication unit 440 is a unit for communicating with an external device via light. The optical communication unit 440 may include an optical transmitter that converts an electrical signal into an optical signal and transmits the optical signal to the outside and an optical receiver that converts the received optical signal into an electrical signal.

Depending on the embodiment, the light emitting unit may be integrally formed with a lamp included in the vehicle 100 .

The broadcast transceiver 450 is a unit for receiving a broadcast signal from an external broadcast management server or transmitting a broadcast signal to the broadcast management server through a broadcast channel. Broadcast channels may include satellite channels and terrestrial channels. The broadcasting signal may include a TV broadcasting signal, a radio broadcasting signal, and a data broadcasting signal.

The processor 470 may control the overall operation of each unit of the communication device 400.

Depending on embodiments, the communication device 400 may include a plurality of processors 470 or may not include a plurality of processors 470 .

When the processor 470 is not included in the communication device 400, the communication device 400 may be operated under the control of a processor of another device in the vehicle 100 or the control unit 170.

Meanwhile, the communication device 400 may implement a vehicle display device together with the user interface device 200 . In this case, the vehicle display device may be referred to as a telematics device or an audio video navigation (AVN) device.

The communication device 400 may operate under the control of the control unit 170 .

The driving control device 500 is a device that receives a user input for driving.

In the case of the manual mode, the vehicle 100 may be operated based on a signal provided by the driving control device 500 .

The driving control device 500 may include a steering input device 510 , an acceleration input device 530 and a brake input device 570 .

The steering input device 510 may receive an input of a driving direction of the vehicle 100 from a user. The steering input device 510 is preferably formed in a wheel shape so that steering input is possible by rotation. Depending on the embodiment, the steering input device may be formed in the form of a touch screen, touch pad, or button.

The acceleration input device 530 may receive an input for acceleration of the vehicle 100 from a user. The brake input device 570 may receive an input for decelerating the vehicle 100 from a user. The acceleration input device 530 and the brake input device 570 are preferably formed in the form of pedals. Depending on the embodiment, the acceleration input device or brake input device may be formed in the form of a touch screen, touch pad, or button.

The driving control device 500 may be operated according to the control of the control unit 170 .

The vehicle driving device 600 is a device that electrically controls driving of various devices in the vehicle 100 .

The vehicle driving device 600 may include a power train driving unit 610, a chassis driving unit 620, a door/window driving unit 630, a safety device driving unit 640, a lamp driving unit 650, and an air conditioning driving unit 660. can

Depending on embodiments, the vehicle driving apparatus 600 may further include components other than the described components or may not include some of the described components.

Meanwhile, the vehicle driving device 600 may include a processor. Each unit of the vehicle driving device 600 may individually include a processor.

The power train driver 610 may control the operation of the power train device.

The power train driving unit 610 may include a power source driving unit 611 and a transmission driving unit 612 .

The power source driver 611 may control the power source of the vehicle 100 .

For example, when a fossil fuel-based engine is a power source, the power source driver 610 may perform electronic control of the engine. This makes it possible to control the output torque of the engine and the like. The power source driver 611 may adjust the engine output torque according to the control of the control unit 170 .

For example, when a motor based on electric energy is a power source, the power source driver 610 may control the motor. The power source driver 610 may adjust the rotational speed and torque of the motor according to the control of the control unit 170 .

The transmission driving unit 612 may control the transmission.

The transmission driving unit 612 can adjust the state of the transmission. The transmission driving unit 612 may adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).

Meanwhile, when the engine is the power source, the transmission driving unit 612 may adjust the engagement state of the gear in the forward (D) state.

The chassis driving unit 620 may control the operation of the chassis device.

The chassis driving unit 620 may include a steering driving unit 621 , a brake driving unit 622 and a suspension driving unit 623 .

The steering driver 621 may perform electronic control of a steering apparatus in the vehicle 100 . The steering drive unit 621 can change the traveling direction of the vehicle.

The brake driver 622 may perform electronic control of a brake apparatus in the vehicle 100 . For example, the speed of the vehicle 100 may be reduced by controlling the operation of brakes disposed on wheels.

Meanwhile, the brake driving unit 622 may individually control each of a plurality of brakes. The brake driving unit 622 may differently control braking force applied to a plurality of wheels.

The suspension driver 623 may perform electronic control of a suspension apparatus in the vehicle 100 . For example, the suspension driving unit 623 may control the vibration of the vehicle 100 to be reduced by controlling the suspension device when there is a curve on the road surface.

Meanwhile, the suspension driving unit 623 may individually control each of the plurality of suspensions.

The door/window driver 630 may perform electronic control of a door apparatus or a window apparatus in the vehicle 100 .

The door/window driver 630 may include a door driver 631 and a window driver 632 .

The door driver 631 may control the door device. The door driving unit 631 may control opening and closing of a plurality of doors included in the vehicle 100 . The door driver 631 may control opening or closing of a trunk or a tail gate. The door driver 631 may control opening or closing of the sunroof.

The window driving unit 632 may perform electronic control of a window apparatus. Opening or closing of a plurality of windows included in the vehicle 100 may be controlled.

The safety apparatus driver 640 may perform electronic control of various safety apparatuses in the vehicle 100 .

The safety device driving unit 640 may include an airbag driving unit 641 , a seat belt driving unit 642 , and a pedestrian protection device driving unit 643 .

The airbag driver 641 may perform electronic control of an airbag apparatus in the vehicle 100 . For example, the airbag driver 641 may control the airbag to be deployed when danger is detected.

The seat belt driver 642 may perform electronic control of a seat belt appartus in the vehicle 100 . For example, the seat belt driver 642 may control the occupants to be fixed to the seats 110FL, 110FR, 110RL, and 110RR using the seat belt when danger is detected.

The pedestrian protection device driving unit 643 may perform electronic control of the hood lift and the pedestrian airbag. For example, when detecting a collision with a pedestrian, the pedestrian protection device driver 643 may control the hood to be lifted up and the pedestrian airbag to be deployed.

The lamp driver 650 may perform electronic control of various lamp apparatuses in the vehicle 100 .

The air conditioning driving unit 660 may perform electronic control of an air cinditioner in the vehicle 100 . For example, when the temperature inside the vehicle is high, the air conditioning driving unit 660 may operate the air conditioner and control cool air to be supplied to the inside of the vehicle.

The vehicle driving apparatus 600 may include a processor. Each unit of the vehicle driving device 600 may individually include a processor.

The vehicle driving device 600 may be operated according to the control of the controller 170 .

The operating system 700 is a system that controls various operations of the vehicle 100 . The driving system 700 may be operated in an autonomous driving mode.

The driving system 700 may include a driving system 710 , a vehicle exit system 740 and a parking system 750 .

Depending on embodiments, the navigation system 700 may further include other components in addition to the described components, or may not include some of the described components.

Meanwhile, the driving system 700 may include a processor. Each unit of the driving system 700 may individually include a processor.

Meanwhile, according to embodiments, when the driving system 700 is implemented in software, it may be a sub-concept of the control unit 170.

Meanwhile, according to an embodiment, the driving system 700 uses at least one of the user interface device 200, the object detection device 300, the communication device 400, the vehicle driving device 600, and the control unit 170. It may be a concept that includes

The driving system 710 may perform driving of the vehicle 100 .

The driving system 710 may perform driving of the vehicle 100 by receiving navigation information from the navigation system 770 and providing a control signal to the vehicle driving device 600 .

The driving system 710 may receive object information from the object detection device 300 and provide a control signal to the vehicle driving device 600 to drive the vehicle 100 .

The driving system 710 may receive a signal from an external device through the communication device 400 and provide a control signal to the vehicle driving device 600 to drive the vehicle 100 .

The vehicle extraction system 740 may perform vehicle 100 extraction.

The vehicle exit system 740 may receive navigation information from the navigation system 770 and provide a control signal to the vehicle driving device 600 to exit the vehicle 100 .

The vehicle extraction system 740 may receive object information from the object detection device 300 and provide a control signal to the vehicle driving device 600 to extract the vehicle 100 .

The vehicle extraction system 740 may receive a signal from an external device through the communication device 400 and provide a control signal to the vehicle driving apparatus 600 to extract the vehicle 100 .

The parking system 750 may perform parking of the vehicle 100 .

The parking system 750 may receive navigation information from the navigation system 770 and provide a control signal to the vehicle driving device 600 to park the vehicle 100 .

The parking system 750 may perform parking of the vehicle 100 by receiving object information from the object detection device 300 and providing a control signal to the vehicle driving device 600 .

The parking system 750 may receive a signal from an external device through the communication device 400 and provide a control signal to the vehicle driving device 600 to park the vehicle 100 .

The navigation system 770 may provide navigation information. The navigation information may include at least one of map information, set destination information, route information according to the destination setting, information on various objects on the route, lane information, and current location information of the vehicle.

The navigation system 770 may include a memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system 770 .

According to an embodiment, the navigation system 770 may receive information from an external device through the communication device 400 and update pre-stored information.

According to embodiments, the navigation system 770 may be classified as a sub-component of the user interface device 200 .

The sensing unit 120 may sense the state of the vehicle. The sensing unit 120 may include a posture sensor (eg, a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, and an inclination sensor. Sensor, weight detection sensor, heading sensor, yaw sensor, gyro sensor, position module, vehicle forward/reverse sensor, battery sensor, fuel sensor, tire sensor, steering wheel A rotational steering sensor, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, and the like may be included.

The sensing unit 120 includes vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward/backward information, battery Acquire sensing signals for information, fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, vehicle external illuminance, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc. can do.

In addition, the sensing unit 120 includes an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), and a throttle position sensor. (TPS), a TDC sensor, a crank angle sensor (CAS), and the like may be further included.

The vehicle interface unit 130 may serve as a passage for various types of external devices connected to the vehicle 100 . For example, the vehicle interface unit 130 may have a port connectable to a mobile terminal, and may be connected to the mobile terminal through the port. In this case, the vehicle interface unit 130 may exchange data with the mobile terminal.

Meanwhile, the vehicle interface unit 130 may serve as a passage through which electrical energy is supplied to the connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit 130, under the control of the controller 170, the vehicle interface unit 130 may provide electrical energy supplied from the power supply unit 190 to the mobile terminal. .

The memory 140 is electrically connected to the controller 170 . The memory 140 may store basic data for the unit, control data for controlling the operation of the unit, and input/output data. The memory 140 may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware. The memory 140 may store various data for overall operation of the vehicle 100, such as a program for processing or control by the control unit 170.

Depending on the embodiment, the memory 140 may be integrally formed with the controller 170 or may be implemented as a sub-component of the controller 170 .

The controller 170 may control overall operations of each unit in the vehicle 100 . The controller 170 may be referred to as an Electronic Control Unit (ECU).

The power supply unit 190 may supply power required for operation of each component according to the control of the control unit 170 . In particular, the power supply unit 190 may receive power from a battery inside the vehicle.

One or more processors and controllers 170 included in the vehicle 100 include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs ( It may be implemented using at least one of field programmable gate arrays, processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

Meanwhile, the vehicle 100 related to the present invention may include a route guidance device 790.

The route guidance device 790 can control at least one of the components described in FIG. 7 . From this point of view, the route guidance device 790 may be the control unit 170.

The route guidance device 790 is not limited thereto, and may be a separate component independent of the control unit 170 . That is, the route guidance device 790 may be implemented as a device or module independent of the vehicle 100. In this case, the route guidance device 790 may be implemented as a vehicle 100 through a part of the vehicle 100. Can communicate/interact with one component of

When the route guidance device 790 is implemented as an independent component separate from the control unit 170, the route guidance device 790 may be provided/connected/mounted to a part of the vehicle 100.

Hereinafter, for convenience of description, the path guidance device 790 will be described as a separate component independent of the control unit 170.

In addition, the function (operation) and control method described for the path guidance device 790 in this specification may be performed by the control unit 170 of the vehicle. That is, all contents described in relation to the route guidance device 790 may be similarly/equally applied to the control unit 170 as well.

In addition, the path guidance device 790 described herein may include some of the components described in FIG. 7 and various components provided in a vehicle. In this specification, for convenience of description, separate names and reference numerals will be given to the components described in FIG. 7 and various components provided in the vehicle to be described.

Meanwhile, the route guidance device 790 of the present invention may be the mobile terminal 800 shown in FIG. 8 or implemented as the mobile terminal 800 . The mobile terminal 800 can communicate with the vehicle 100 and may be a component independent of the vehicle 100 .

Hereinafter, the mobile terminal 800, which is an embodiment of the route guidance device 790, will be described in detail.

Mobile terminals described in this specification include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, and slate PCs. , tablet PC, ultrabook, wearable device (eg, watch type terminal (smartwatch), glass type terminal (smart glass), HMD (head mounted display)), etc. may be included there is.

However, those skilled in the art can easily recognize that the configuration according to the embodiments described in this specification may be applied to fixed terminals such as digital TVs, desktop computers, digital signage, etc., except when applicable only to mobile terminals. will be.

8 is a block diagram for explaining a mobile terminal related to the present invention.

The mobile terminal 800 includes a wireless communication unit 810, an input unit 820, a sensing unit 840, an output unit 850, an interface unit 860, a memory 870, a control unit 880, and a power supply unit 890. ) and the like. The components shown in FIG. 8 are not essential for implementing a mobile terminal, so a mobile terminal described herein may have more or fewer components than those listed above.

More specifically, among the components, the wireless communication unit 810 is between the mobile terminal 800 and the wireless communication system, between the mobile terminal 800 and another mobile terminal 800, or between the mobile terminal 800 and an external server. It may include one or more modules enabling wireless communication between Also, the wireless communication unit 810 may include one or more modules that connect the mobile terminal 800 to one or more networks.

The wireless communication unit 810 may include at least one of a broadcast reception module 811, a mobile communication module 812, a wireless Internet module 813, a short-distance communication module 814, and a location information module 815. .

The input unit 820 includes a camera 821 or video input unit for inputting a video signal, a microphone 822 for inputting an audio signal, or a user input unit 823 for receiving information from a user, for example , a touch key, a push key (mechanical key, etc.). Voice data or image data collected by the input unit 820 may be analyzed and processed as a user's control command.

The sensing unit 840 may include one or more sensors for sensing at least one of information within the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information. For example, the sensing unit 840 may include a proximity sensor 841, an illumination sensor 842, a touch sensor, an acceleration sensor, a magnetic sensor, and gravity. Sensor (G-sensor), gyroscope sensor (gyroscope sensor), motion sensor (motion sensor), RGB sensor, infrared sensor (IR sensor), finger scan sensor, ultrasonic sensor , optical sensor (e.g., camera (see 821)), microphone (microphone, see 822), battery gauge, environmental sensor (e.g., barometer, hygrometer, thermometer, radiation detection sensor, It may include at least one of a heat detection sensor, a gas detection sensor, etc.), a chemical sensor (eg, an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the mobile terminal disclosed in this specification may combine and utilize information sensed by at least two or more of these sensors.

The output unit 850 is for generating an output related to sight, hearing, or touch, and includes at least one of a display unit 851, a sound output unit 852, a haptic module 853, and an optical output unit 854. can do. The display unit 851 may implement a touch screen by forming a mutual layer structure or integrally with the touch sensor. Such a touch screen may function as a user input unit 823 providing an input interface between the mobile terminal 800 and the user and provide an output interface between the mobile terminal 800 and the user.

The interface unit 860 serves as a passage for various types of external devices connected to the mobile terminal 800 . The interface unit 860 connects a device equipped with a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. It may include at least one of a port, an audio input/output (I/O) port, a video input/output (I/O) port, and an earphone port. In response to an external device being connected to the interface unit 860, the mobile terminal 800 may perform appropriate control related to the connected external device.

In addition, the memory 870 stores data supporting various functions of the mobile terminal 800 . The memory 870 may store a plurality of application programs (applications) running in the mobile terminal 800, data for operation of the mobile terminal 800, and commands. At least some of these application programs may be downloaded from an external server through wireless communication. In addition, at least some of these application programs may exist on the mobile terminal 800 from the time of shipment for the basic functions of the mobile terminal 800 (eg, incoming call, outgoing function, message reception, and outgoing function). Meanwhile, the application program may be stored in the memory 870, installed on the mobile terminal 800, and driven by the controller 880 to perform an operation (or function) of the mobile terminal.

The controller 880 controls general operations of the mobile terminal 800 in addition to operations related to the application program. The control unit 880 may provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components described above or by running an application program stored in the memory 870.

In addition, the controller 880 may control at least some of the components discussed in conjunction with FIG. 8 in order to drive an application program stored in the memory 870 . Furthermore, the controller 880 may combine and operate at least two or more of the components included in the mobile terminal 800 to drive the application program.

The power supply unit 890 receives external power and internal power under the control of the controller 880 and supplies power to each component included in the mobile terminal 800 . The power supply unit 890 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the above components may operate in cooperation with each other to implement an operation, control, or control method of a mobile terminal according to various embodiments described below. Also, the operation, control, or control method of the mobile terminal may be implemented on the mobile terminal by driving at least one application program stored in the memory 870 .

Hereinafter, prior to examining various embodiments implemented through the mobile terminal 800 discussed above, the above-listed components will be examined in more detail with reference to FIG. 8 .

First, looking at the wireless communication unit 810, the broadcast reception module 811 of the wireless communication unit 810 receives a broadcast signal and/or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more broadcast reception modules may be provided in the mobile terminal 800 to receive simultaneous broadcasts of at least two broadcast channels or to switch broadcast channels.

The mobile communication module 812 complies with technical standards or communication methods for mobile communication (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), EV) -DO(Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA(Wideband CDMA), HSDPA(High Speed Downlink Packet Access), HSUPA(High Speed Uplink Packet Access), LTE(Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.) to transmit and receive radio signals with at least one of a base station, an external terminal, and a server on a mobile communication network.

The wireless signal may include a voice call signal, a video call signal, or various types of data according to text/multimedia message transmission/reception.

The wireless Internet module 813 refers to a module for wireless Internet access, and may be built into or external to the mobile terminal 800 . The wireless Internet module 813 is configured to transmit and receive radio signals in a communication network based on wireless Internet technologies.

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc., and the wireless Internet module ( 813) transmits and receives data according to at least one wireless Internet technology within a range including Internet technologies not listed above.

From the viewpoint that wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, etc. is made through a mobile communication network, the wireless Internet module (813 ) may be understood as a kind of the mobile communication module 812.

The short range communication module 814 is for short range communication, and includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and NFC. Near Field Communication (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and wireless USB (Wireless Universal Serial Bus) technology may be used to support short-distance communication. Such a short-range communication module 814 may be used between the mobile terminal 800 and a wireless communication system, between the mobile terminal 800 and another mobile terminal 800, or between the mobile terminal 800 through wireless area networks. ) and a network where another mobile terminal (800 or external server) is located. The local area wireless communication network may be a local area wireless personal area network (Wireless Personal Area Networks).

Here, the other mobile terminal 800 is a wearable device capable of exchanging (or interlocking) data with the mobile terminal 800 according to the present invention (for example, a smartwatch, smart glasses) (smart glass) or HMD (head mounted display). The short-distance communication module 814 may detect (or recognize) a wearable device capable of communicating with the mobile terminal 800 around the mobile terminal 800 . Furthermore, if the detected wearable device is a device authorized to communicate with the mobile terminal 800 according to the present invention, the controller 880 transmits at least a portion of data processed by the mobile terminal 800 to the short-range communication module ( 814), it can be transmitted to the wearable device. Accordingly, the user of the wearable device may use data processed by the mobile terminal 800 through the wearable device. For example, according to this, when a call is received in the mobile terminal 800, the user makes a phone call through the wearable device, or when a message is received in the mobile terminal 800, the user receives the received message through the wearable device. It is possible to check the message.

The location information module 815 is a module for acquiring the location (or current location) of the mobile terminal, and representative examples thereof include a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, when a mobile terminal utilizes a GPS module, the location of the mobile terminal can be obtained using signals transmitted from GPS satellites. As another example, when the mobile terminal utilizes the Wi-Fi module, the location of the mobile terminal may be obtained based on information of the Wi-Fi module and a wireless access point (AP) that transmits or receives a wireless signal. If necessary, the location information module 815 may perform any function among other modules of the wireless communication unit 810 in order to obtain data on the location of the mobile terminal in substitution or addition. The location information module 815 is a module used to obtain the location (or current location) of the mobile terminal, and is not limited to a module that directly calculates or obtains the location of the mobile terminal.

Next, the input unit 820 is for inputting video information (or signals), audio information (or signals), data, or information input from a user. For inputting video information, the mobile terminal 800 has one Alternatively, a plurality of cameras 821 may be provided. The camera 821 processes an image frame such as a still image or a moving image obtained by an image sensor in a video call mode or a photographing mode. The processed image frame may be displayed on the display unit 851 or stored in the memory 870 . Meanwhile, the plurality of cameras 821 provided in the mobile terminal 800 may be arranged to form a matrix structure, and through the cameras 821 constituting the matrix structure, various angles or focal points may be provided to the mobile terminal 800. A plurality of image information having may be input. Also, the plurality of cameras 821 may be arranged in a stereo structure to acquire left and right images for realizing a stereoscopic image.

The microphone 822 processes external sound signals into electrical voice data. The processed voice data may be used in various ways according to the function (or application program being executed) being performed in the mobile terminal 800 . Meanwhile, various noise cancellation algorithms for removing noise generated in the process of receiving an external sound signal may be implemented in the microphone 822 .

The user input unit 823 is for receiving information from a user. When information is input through the user input unit 823, the control unit 880 can control the operation of the mobile terminal 800 to correspond to the input information. . The user input unit 823 is a mechanical input means (or a mechanical key, for example, a button located on the front, rear or side of the mobile terminal 800, a dome switch, a jog wheel, jog switch, etc.) and a touch input means. As an example, the touch input means consists of a virtual key, soft key, or visual key displayed on a touch screen through software processing, or a part other than the touch screen. It can be made of a touch key (touch key) disposed on. On the other hand, the virtual key or visual key can be displayed on the touch screen while having various forms, for example, graphic (graphic), text (text), icon (icon), video (video) or these may be made of a combination of

Meanwhile, the sensing unit 840 senses at least one of information within the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information, and generates a sensing signal corresponding thereto. Based on these sensing signals, the controller 880 may control driving or operation of the mobile terminal 800 or perform data processing, functions, or operations related to applications installed in the mobile terminal 800 . Representative sensors among various sensors that may be included in the sensing unit 840 will be described in more detail.

First, the proximity sensor 841 refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing nearby without mechanical contact by using the force of an electromagnetic field or infrared rays. The proximity sensor 841 may be disposed in an inner area of the mobile terminal surrounded by the touch screen described above or in the vicinity of the touch screen.

Examples of the proximity sensor 841 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitance type proximity sensor, a magnetic type proximity sensor, an infrared proximity sensor, and the like. If the touch screen is a capacitive type, the proximity sensor 841 may be configured to detect the proximity of a conductive object as a change in electric field according to the proximity of the object. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of description, an action of approaching a touch screen without contacting an object to recognize that the object is located on the touch screen is referred to as a "proximity touch", and the touch An act of actually touching an object on a screen is called a "contact touch". The location at which an object is proximity-touched on the touch screen means a location at which the object corresponds vertically to the touch screen when the object is proximity-touched. The proximity sensor 841 may detect a proximity touch and a proximity touch pattern (eg, proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.) there is. Meanwhile, as described above, the controller 880 processes data (or information) corresponding to the proximity touch operation and proximity touch pattern detected through the proximity sensor 841, and furthermore, visual information corresponding to the processed data. It can be output on the touch screen. Furthermore, the controller 880 may control the mobile terminal 800 to process different operations or data (or information) depending on whether a touch to the same point on the touch screen is a proximity touch or a contact touch. .

The touch sensor detects a touch (or touch input) applied to the touch screen (or the display unit 851) using at least one of various touch methods such as a resistive film method, a capacitive method, an infrared method, an ultrasonic method, and a magnetic field method. detect

As an example, the touch sensor may be configured to convert a change in pressure applied to a specific part of the touch screen or capacitance generated in a specific part into an electrical input signal. The touch sensor may be configured to detect a location, area, pressure upon touch, capacitance upon touch, and the like, at which a touch object that applies a touch to the touch screen is touched on the touch sensor. Here, the touch object is an object that applies a touch to the touch sensor, and may be, for example, a finger, a touch pen, a stylus pen, or a pointer.

As such, when there is a touch input to the touch sensor, the corresponding signal(s) is sent to the touch controller. The touch controller processes the signal(s) and then transmits corresponding data to controller 880. Accordingly, the controller 880 can know which area of the display unit 851 has been touched. Here, the touch controller may be a component separate from the controller 880 or may be the controller 880 itself.

Meanwhile, the controller 880 may perform different controls or the same control according to the type of the touch object that touches the touch screen (or a touch key provided other than the touch screen). Whether different or the same control is to be performed according to the type of the touch object may be determined according to an operating state of the mobile terminal 800 or an application program being executed.

On the other hand, the touch sensor and proximity sensor described above are independently or combined, short (or tap) touch, long touch, multi-touch, drag touch on the touch screen ), flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, etc. Touch can be sensed.

The ultrasonic sensor may recognize location information of a sensing object by using ultrasonic waves. Meanwhile, the controller 880 may calculate the location of the wave generating source through information detected by the optical sensor and the plurality of ultrasonic sensors. The position of the wave generating source can be calculated using the property that light is much faster than ultrasonic waves, that is, the time for light to reach the optical sensor is much faster than the time for ultrasonic waves to reach the ultrasonic sensor. More specifically, the location of the wave generating source may be calculated by using light as a reference signal and a time difference between the arrival time of ultrasonic waves.

Meanwhile, looking at the configuration of the input unit 820, the camera 821 includes at least one of a camera sensor (eg, CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

The camera 821 and the laser sensor may be combined with each other to sense a touch of a sensing target for a 3D stereoscopic image. A photo sensor may be stacked on a display device, and the photo sensor is configured to scan a motion of a sensing target close to the touch screen. More specifically, the photo sensor scans the content placed on the photo sensor by mounting photo diodes and transistors (TRs) in rows/columns and using electrical signals that change according to the amount of light applied to the photo diodes. That is, the photo sensor calculates the coordinates of the sensing object according to the change in light, and through this, the location information of the sensing object can be obtained.

The display unit 851 displays (outputs) information processed by the mobile terminal 800 . For example, the display unit 851 may display execution screen information of an application program driven in the mobile terminal 800 or UI (User Interface) and GUI (Graphic User Interface) information according to such execution screen information. .

Also, the display unit 851 may be configured as a 3D display unit that displays a 3D image.

A 3D display method such as a stereoscopic method (glasses method), an auto stereoscopic method (non-glasses method), or a projection method (holographic method) may be applied to the 3D display unit.

In general, a 3D stereoscopic image is composed of a left image (image for the left eye) and a right image (image for the right eye). According to the method of combining the left and right images into a 3D stereoscopic image, top-down method in which the left and right images are arranged up and down in one frame, left and right images in one frame L-to-R (left-to-right, side by side) method for arranging left and right images, checker board method for arranging pieces of left and right images in tile form, and left and right images in units of columns Alternatively, it is divided into an interlaced method that alternately arranges rows, and a time-sequential, frame-by-frame method that alternately displays a left image and a right image by time.

Also, the 3D thumbnail image may be generated as a single image by generating left and right image thumbnails from the left and right images of the original image frame, respectively, and combining them. In general, a thumbnail refers to a reduced image or a reduced still image. The generated left and right image thumbnails are displayed with a left and right distance difference on the screen by a depth corresponding to the parallax between the left and right images, thereby representing a three-dimensional sense of space.

A left image and a right image necessary for implementing a 3D stereoscopic image may be displayed on a stereoscopic display unit by a stereoscopic processing unit. The stereoscopic processing unit receives 3D images (images at a reference point and images at an extended point of view) and sets left and right images therefrom, or receives 2D images and converts them into left and right images.

The audio output unit 852 may output audio data received from the wireless communication unit 810 or stored in the memory 870 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, or a broadcast reception mode. The sound output unit 852 also outputs sound signals related to functions performed by the mobile terminal 800 (eg, call signal reception sound, message reception sound, etc.). The sound output unit 852 may include a receiver, a speaker, a buzzer, and the like.

A haptic module 853 generates various tactile effects that a user can feel. A representative example of the tactile effect generated by the haptic module 853 may be vibration. The intensity and pattern of the vibration generated by the haptic module 853 may be controlled by a user's selection or a controller's settings. For example, the haptic module 853 may synthesize and output different vibrations or sequentially output them.

In addition to vibration, the haptic module 853 responds to stimuli such as an array of pins that move vertically with respect to the contacted skin surface, blowing force or suction force of air through a nozzle or suction port, rubbing against the skin surface, contact with an electrode, and electrostatic force. It is possible to generate various tactile effects, such as the effect of heat absorption and the effect of reproducing the feeling of coolness and warmth using an element capable of absorbing heat or generating heat.

The haptic module 853 can deliver tactile effects through direct contact, and can also be implemented so that the user can feel the tactile effects through muscle sensations such as fingers or arms. Two or more haptic modules 853 may be provided according to the configuration of the mobile terminal 800 .

The light output unit 854 outputs a signal for notifying occurrence of an event using light from a light source of the mobile terminal 800 . Examples of events occurring in the mobile terminal 800 may include message reception, call signal reception, missed calls, alarms, schedule notifications, e-mail reception, and information reception through applications.

A signal output from the light output unit 854 is implemented as the mobile terminal emits light of a single color or multiple colors toward the front or back side. The signal output may be terminated when the mobile terminal detects the user's confirmation of the event.

The interface unit 860 serves as a passage for all external devices connected to the mobile terminal 800 . The interface unit 860 receives data from an external device or receives power and transmits it to each component inside the mobile terminal 800 or transmits data inside the mobile terminal 800 to an external device. For example, a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and a port for connecting a device having an identification module. A port, an audio input/output (I/O) port, a video input/output (I/O) port, an earphone port, and the like may be included in the interface unit 860 .

On the other hand, the identification module is a chip that stores various types of information for authenticating the right to use the mobile terminal 800, and includes a user identification module (UIM), a subscriber identity module (SIM), and universal user authentication. module (universal subscriber identity module; USIM) and the like. A device equipped with an identification module (hereinafter referred to as 'identification device') may be manufactured in the form of a smart card. Accordingly, the identification device may be connected to the terminal 800 through the interface unit 860 .

In addition, the interface unit 860 becomes a passage through which power from the cradle is supplied to the mobile terminal 800 when the mobile terminal 800 is connected to an external cradle, or a user inputs power from the cradle. It can be a passage through which various command signals are transmitted to the mobile terminal 800 . Various command signals or the power input from the cradle may operate as a signal for recognizing that the mobile terminal 800 is correctly mounted on the cradle.

The memory 870 may store programs for operation of the controller 880 and may temporarily store input/output data (eg, phonebook, message, still image, video, etc.). The memory 870 may store data related to vibration and sound of various patterns output when a touch is input on the touch screen.

The memory 870 may be a flash memory type, a hard disk type, a solid state disk type, a silicon disk drive type, or a multimedia card micro type. ), card-type memory (eg SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read -only memory), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The mobile terminal 800 may be operated in relation to a web storage that performs a storage function of the memory 870 on the Internet.

Meanwhile, as described above, the controller 880 controls operations related to applications and general operations of the mobile terminal 800 . For example, if the state of the mobile terminal satisfies a set condition, the controller 880 may execute or release a locked state that restricts the user's control command input to applications.

In addition, the controller 880 may perform control and processing related to voice calls, data communications, video calls, etc., or perform pattern recognition processing capable of recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively. can Furthermore, the control unit 880 may control any one or a combination of the components described above in order to implement various embodiments described below on the mobile terminal 800 according to the present invention.

The power supply unit 890 receives external power and internal power under the control of the control unit 880 and supplies power necessary for the operation of each component. The power supply unit 890 includes a battery, and the battery may be a built-in battery capable of being charged, or may be detachably coupled to the terminal body for charging.

In addition, the power supply unit 890 may have a connection port, and the connection port may be configured as an example of an interface 860 to which an external charger that supplies power for battery charging is electrically connected.

As another example, the power supply unit 890 may charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 890 uses at least one of an inductive coupling method based on magnetic induction from an external wireless power transmitter or a magnetic resonance coupling method based on electromagnetic resonance. power can be delivered.

Meanwhile, various embodiments hereinafter may be embodied in a recording medium readable by a computer or a similar device using, for example, software, hardware, or a combination thereof.

Hereinafter, a route guidance device and a route guidance method for guiding a route to be driven by a vehicle in an optimized manner through augmented reality according to an embodiment of the present invention will be described.

9 is a block diagram for explaining a route guidance device according to the present invention.

Referring to FIG. 9 , a route guidance device 790 according to an embodiment of the present invention includes a communication unit 910, a processor 920 capable of controlling it, and a display unit outputting information processed by the processor 920. (930).

The communication unit 910 may be formed so that the path guidance device 790 communicates with an external device or an external component, and may perform wired/wireless communication.

The communication unit 910 is any one of the communication device 400 of the vehicle 100 described above, the wireless communication unit 810 of the mobile terminal 800, or the interface unit 860 of the mobile terminal 800, or The same/similar content can be inferred and applied.

For example, when the route guidance device 790 is implemented as the controller 170 of the vehicle 100, the communication unit 910 may be the communication device 400 of the vehicle.

As another example, when the route guidance device 790 is implemented as the mobile terminal 100, the communication unit 910 may be the wireless communication unit 810 of the mobile terminal 800 or the interface unit 860 of the mobile terminal 800. can

As another example, when the route guidance device 790 is a separate and independent device implemented to enable communication with the vehicle 100 and/or the mobile terminal 800, the above-described communication device 400 and the wireless communication unit 810 and/or the contents of the interface unit 860 may be similarly/identically applied.

The processor 920 may be configured to control the communication unit 910 and may control operations and functions performed by the route guidance device 790 .

The processor 920 may be the control unit 170 of the vehicle 100 or the control unit 880 of the mobile terminal 800 described above, and the same/similar contents thereof may be analogously applied.

For example, when the route guidance device 790 is implemented as the control unit 170 of the vehicle 100, the processor 920 may be the control unit 170 of the vehicle 100.

As another example, when the route guidance device 790 is implemented in the mobile terminal 100, the processor 920 may be the controller 880 of the mobile terminal 800.

As another example, if the route guidance device 790 is a separate and independent device implemented to be able to communicate with the vehicle 100 and/or the mobile terminal 800, the processor 920 may Alternatively, it may communicate with the mobile terminal and control at least one of the vehicle and/or the mobile terminal.

The communication unit 910 may receive the first image photographed through the camera.

The camera may be a camera installed in the route guidance device 790, a camera 310 installed in a vehicle, or a camera 821 installed in a mobile terminal.

The processor 920 may receive an image processed through a camera (eg, a real-time image, a preview image, or a live image) through the communication unit 910 .

For example, when the route guidance device 790 is implemented as the mobile terminal 800, the processor 920 transmits the first image captured through the camera 821 to the communication unit 810 (or the interface unit 860). can be received through

The processor 920 may perform calibration for implementing augmented reality on the received first image.

In this specification, an image before calibration is performed is referred to as a first image, and an image after calibration is referred to as a second image.

The processor 920 may perform calibration on the first image.

Also, the processor 920 may overlap and output a graphic object guiding vehicle driving to the calibrated second image.

Overlapping and outputting a graphic object for guiding driving of a vehicle on an image captured by a camera may mean implementing augmented reality or performing route guidance with augmented reality.

The display unit 930 may be formed to output information processed by the processor 920, and may be configured to display the display unit 251 provided in the vehicle 100 and/or the display unit 851 of the mobile terminal 800. can be

For example, the processor 920 may overlap an image received through a camera and a graphic object guiding driving of a vehicle to the image and output the overlapping image to the display unit 930 .

The operation and control method of the route guidance device 790 described herein may be performed by the control unit 170 of the vehicle 100 . That is, the operation and/or control method performed by the processor 920 of the route guidance device 790 may be performed by the controller 170 of the vehicle 100 or the controller 880 of the mobile terminal 800. there is.

The communication unit 910 is configured to perform communication with various components described in FIGS. 7 and 8 . For example, the communication unit 910 may receive various types of information provided through a controller are network (CAN). As another example, the communication unit 910 may perform communication with all communicable devices such as vehicles, mobile terminals and servers, and other vehicles. This may be referred to as V2X (Vehicle to everything) communication. V2X communication can be defined as a technology that exchanges or shares information such as traffic conditions while communicating with road infrastructure and other vehicles while driving.

The communication unit 910 is configured to perform communication with one or more devices provided in the vehicle 100 . The communication unit 810 may include a beam former and a radio frequency IC (RFIC) that controls the beam former to implement 5G communication in a frequency band of 6 GHz or higher. However, when the communication unit 810 implements 5G communication in a frequency band of 6 GHz or less, the beamformer and RFIC are not essential.

Furthermore, the communication unit 910 may receive information related to driving of the vehicle from most devices included in the vehicle 100 . The information transmitted from the vehicle 100 to the route guidance device 790 is referred to as 'vehicle driving information'.

Vehicle driving information includes vehicle information and surrounding information of the vehicle. Based on the frame of the vehicle 100, information related to the inside of the vehicle may be defined as vehicle information, and information related to the outside of the vehicle may be defined as surrounding information.

The vehicle information refers to information about the vehicle itself. For example, vehicle information includes vehicle driving speed, driving direction, acceleration, angular velocity, location (GPS), weight, vehicle occupants, vehicle braking force, vehicle maximum braking force, air pressure of each wheel, and centrifugal force applied to the vehicle. , driving mode of the vehicle (whether autonomous driving mode or manual driving), parking mode of the vehicle (autonomous parking mode, automatic parking mode, manual parking mode), whether the user is in the vehicle and information related to the user, etc. can include

The surrounding information refers to information about other objects located within a predetermined range around the vehicle and information related to the outside of the vehicle. For example, the condition of the road surface on which the vehicle is driving (friction), weather, distance to the front (or rear) vehicle, relative speed of the front (or rear) vehicle, curvature of the curve when the driving lane is a curve, vehicle Ambient brightness, information related to an object existing in a reference area (region) based on the vehicle, whether an object enters/exits the predetermined area, whether a user exists around the vehicle, and information related to the user (for example, For example, whether the user is an authenticated user), and the like.

In addition, the surrounding information includes ambient brightness, temperature, sun position, object information (person, other vehicle, sign, etc.) located around, type of road surface being driven, terrain features, lane information, driving lane (Lane) ) information, and information necessary for autonomous driving/autonomous parking/automatic parking/manual parking modes.

In addition, the surrounding information may include a distance between an object (object) existing around the vehicle and the vehicle 100, a possibility of collision, a type of the object, a parking space in which the vehicle can be parked, and an object for identifying the parking space (for example, , parking lines, strings, other vehicles, walls, etc.) and the like may be further included.

The vehicle travel information is not limited to the above-described examples, and may include all information generated from components included in the vehicle 100 .

The display unit 930 outputs various visual information under the control of the processor 920 . The display unit 930 may output visual information to a windshield of a vehicle or a separately provided screen, or output visual information through a panel. The display unit 930 may correspond to the display unit 251 described with reference to FIG. 7 .

For example, the visual information output by the display unit 930 is reflected from the windshield or the screen, so that the visual information is displayed on the windshield or the screen. The passenger simultaneously checks the real world located outside the vehicle 100 and the virtual object displayed on the windshield or the screen, and augmented reality is implemented by the display unit 930 .

As another example, in the display unit 930, under the control of the processor 920, augmented reality may be implemented by overlapping various visual information with an image received in real time through a camera.

The processor 920 may control one or more devices included in the vehicle 100 using the communication unit 910 .

Specifically, the processor 920 may determine whether at least one condition among a plurality of preset conditions is satisfied based on vehicle driving information received through the communication unit 910 .

Depending on the conditions satisfied, the processor 920 may control the one or more displays in different ways.

Regarding a preset condition, the processor 90 may detect that an event has occurred in an electronic device and/or an application provided in the vehicle 100 and determine whether the detected event satisfies a preset condition. At this time, the processor 920 may detect that an event has occurred from information received through the communication unit 910 .

The application is a concept including a widget or a home launcher, and refers to all types of programs that can be driven in the vehicle 100 . Accordingly, the application may be a program that performs functions of web browser, video playback, message transmission/reception, schedule management, and application update.

Furthermore, the applications include Forward Collision Warning (FCW), Blind Spot Detection (BSD), Lane Departure Warning (LDW), Pedestrian Detection (PD), and Curve Speed Warning (Curve Speed Warning, CSW) and turn by turn navigation (TBT).

For example, event occurrence is when there is a missed call, when there is an application to be updated, when a message arrives, start on, start off, autonomous driving on/off, display activation key pressed (LCD awake key), alarm, incoming call, missed notification, and the like.

As another example, the occurrence of an event may be a case in which a warning set in an advanced driver assistance system (ADAS) occurs or a function set in the ADAS is performed. For example, when a forward collision warning occurs, when blind spot detection occurs, when a lane departure warning occurs, and when a driving steering assist warning occurs When an assist warning occurs, an event may be regarded as occurring when an autonomous emergency braking function is performed.

As another example, when changing from a forward gear to a reverse gear, when an acceleration greater than a predetermined value occurs, when a deceleration greater than a predetermined value occurs, when a power unit is changed from an internal combustion engine to a motor, or when a motor An event can be regarded as occurring even when the engine is changed to an internal combustion engine.

In addition, an event may be regarded as occurring even when various ECUs provided in the vehicle 100 perform a specific function.

When an event that has occurred satisfies a preset condition, the processor 920 controls the communication unit 910 to display information corresponding to the satisfied condition on the one or more displays.

The processor 920 receives a front image obtained by photographing the front of the vehicle 100 through a camera. The forward image may be received through the communication unit 910 and may be composed of one or more images.

Next, the processor 920 searches for one or more lanes in which the vehicle 100 is scheduled to travel from the forward image.

For convenience of description, one or more lanes in which the vehicle 100 is scheduled to drive are referred to as 'driving lanes'.

The scheduled driving lane refers to a lane in which the vehicle 100 is scheduled to travel until time t, which is a positive real number, based on the current time point. The t may vary according to the speed of the vehicle 100, the characteristics of the road on which the vehicle 100 is driving, and the speed limit set for the road on which the vehicle 100 is driving.

When the vehicle 100 is autonomously driving, the driving scheduled lane means a driving scheduled lane in autonomous driving. When the vehicle 100 is driving manually, the driving scheduled lane means a lane recommended to the driver.

In order to search for the scheduled driving lane, the processor 920 may receive a high-precision map from a path providing device and/or a server, and receive vehicle driving information capable of specifying the driving scheduled lane.

More specifically, the processor 920 may receive forward path information for guiding the road located in front of the vehicle 100 in units of lanes.

The forward route information is configured to provide a driving route to a destination for each lane marked on the road, and may be route information conforming to ADASIS standards.

The forward path information may be provided by segmenting a path along which the vehicle should or may be driven on a per-lane basis. The forward path information may be information for guiding a driving path to a destination in units of lanes. When the forward route information is displayed on a display mounted in the vehicle 100, a guide line guiding a drivable lane may be displayed on a map. Furthermore, a graphic object indicating the location of the vehicle 100 may be included on at least one lane in which the vehicle 100 is located among a plurality of lanes included in the map.

For example, the road located in front of the vehicle 100 may have 8 lanes, and the scheduled driving lane may have 2 lanes. In this case, the processor 920 may search for the second lane in the forward image.

For another example, the road located in front of the vehicle 100 has 8 lanes, driving in 2 lanes is scheduled from the current point to 50 m ahead, and lane change to 3 lanes at 50 m in front may be scheduled. there is. In this case, the processor 920 may search for a second lane up to 50m ahead and a third lane after 50m ahead in the forward image.

Here, searching for a lane means searching for a partial area including the driving schedule lane among the entire area of the forward image. This is to allow the occupants of the vehicle 100 to intuitively recognize the driving schedule lane by displaying the carpet image guiding the driving schedule lane so as to overlap with the searched partial area.

Next, the processor 920 outputs, through the display unit 930 , a carpet image guiding the searched one or more lanes on a lane-by-lane basis.

When the display unit 930 is a windshield or a window of a vehicle, the processor 920 may set an image display area to output visual information based on the eye position and/or line of sight of a passenger.

Furthermore, the processor 920 determines at least one of the location, size, and shape of the main carpet image based on the eye position and/or line of sight of the passenger. At least one of the position, size, and shape of the main carpet image output to the windshield or the screen may vary according to the position and/or line of sight of the occupant. This is to provide augmented reality in which the real world and the virtual image exactly match.

The main carpet image guides the driving scheduled lane, and may be a transparent image overlapping the driving scheduled lane and having a predetermined color.

The predetermined color may vary according to reference conditions. For example, in the case of a general road, it is a first color, but in the case of snow on the road, it may be a second color different from the first color.

Passengers may be provided with route information on lanes on which the vehicle will drive autonomously or on which lanes the driver is expected to drive through the main carpet image in units of lanes.

Meanwhile, the processor 920 may provide the main carpet image and one or more sub-carpet images selectable by the passenger together with the main carpet image to the passenger.

Meanwhile, the processor 920 controls the communication unit to receive an image captured from another vehicle existing on a route on which the vehicle is scheduled to travel. Specifically, an image captured from another vehicle may be encoded and then transmitted to the vehicle. Accordingly, the image captured in the other vehicle received through the communication unit needs to undergo a separate decoding process. A decoder for decoding may be built into the route guidance device 800 or the vehicle 100.

Hereinafter, a method of guiding (guiding) a path on which a vehicle must travel by the route guidance device 790 of the present invention through augmented reality will be described in more detail with reference to the accompanying drawings.

10a, 10b, 10c, 10d, 10e, 10f, 11a, 11b, 11c, 11d, 11e, 11f and 11g show augmented reality according to an embodiment of the present invention. It is a conceptual diagram to explain the applied route guidance method.

In this specification, outputting certain information by the processor 920 may mean outputting certain information to the display unit 930 or controlling the display unit 930 to output certain information.

As shown in FIG. 10 , the processor 920 executes an application for road guidance (eg, a navigation app) 1000 and, based on a preset condition being satisfied, selects an augmented reality (AR) mode. can enter

Preset conditions for entering the AR mode may include various conditions, such as when a user request is input, when communication is smooth, when a vehicle enters a highway, and within a predetermined distance from entering/exiting an intersection. , can be added/changed/set by the user.

Upon entering the AR mode, the processor 920 may receive a first image captured through a camera. The first image may be an image captured in real time.

First, the processor 920 may receive a first image captured by a camera and perform calibration on the first image. The calibration may mean correcting or correcting an image to be suitable for implementing augmented reality.

The processor 920 may perform calibration on the first image, and output a graphic object guiding the driving of the vehicle by overlapping the calibrated second image.

In this specification, an image obtained by performing calibration on the first image is referred to as a second image.

As the first image is received in real time, calibration may be continuously performed in real time, and accordingly, the second image may also be captured in real time.

The processor 920 may output a first graphic object related to driving to the first image 1010 before calibration is performed.

After the calibration is performed, the processor 920 may overlap the first graphic object and the second graphic object related to driving on the second image 1020 and output the same.

The first graphic object or the second graphic object related to the driving is, for example, information generated by a navigation application (or navigation system 770) or through communication with a navigation application (or navigation system 770). It may be generated by the processor 920 based on the received path information to the destination.

The first graphic object includes turn-by-turn information 931 indicating a road to be entered in front of a predetermined distance, and the second graphic object is included in the second image. It may include a carpet image 932 that overlaps the designated lane and guides a planned driving path of the vehicle.

A road is a place that is open to anyone or a vehicle or horse to pass through, and a place where safe and smooth communication can occur.

A portion of a road divided into lanes 1031 so that horses run in one line on such a road is called a lane 1032.

The lane may refer to a line marked with a safety mark at a boundary point between lanes.

As shown in FIG. 10A , the processor 920 provides, in the first image 1010 before calibration, turn-by-turn information (first Only the graphic object) 931 may be overlapped with the first image 1010 and output.

Thereafter, when calibration is performed, the processor 920 displays a carpet image (second graphic) guiding a planned driving path of the vehicle along with the first graphic object 931 in the calibrated second image 1020 . The object) 932 may be overlapped with the road 1032 and output.

The lane 1032 overlapping the carpet image may be a lane in which a vehicle is currently driving.

As shown in FIG. 10A , the processor 920 overlaps and outputs a second graphic object 932 on a lane on which the vehicle is driving among a plurality of lanes included in the second image 1020, and outputs the overlapped second graphic object 932 while the vehicle is driving. When it is determined to deviate from the in-lane, the second graphic object may not be output (Off Route).

Also, as shown in FIG. 10A , the output of the first graphic object 931 may be maintained regardless of whether the vehicle leaves the driving lane.

Meanwhile, the processor 920 may perform calibration so that the road area 1022 included in the first image has a predetermined size or larger. For example, the processor 920 may calibrate the image so that the area of the road occupied by the image is more than half.

Referring to FIG. 10B , when the AR mode is entered again after the execution screen of another application 1000 is displayed while the second image 1020 for which calibration has been performed is output, the processor 920 displays the first image. After re-outputting, calibration may be performed to output the second image to the display unit 930 .

Referring to FIG. 10C, the processor 920, even when the route guidance device 790 is placed in the landscape mode, outputs a first image in the same way, performs calibration on the first image, and then performs calibration The first and second graphic objects may be overlapped and output on the second image.

Meanwhile, when the viewing angle of the second image is changed so that the road included in the second image and the graphic object 932a overlapping the road are rotated by the driving of the vehicle, the processor 920 may change the viewing angle of the second image. Based on 2 images, the graphic object may be changed to match the lane (932a is changed to 932b).

For example, as shown in FIG. 10D , when the vehicle drives near a point where the inclination of the slope changes, the angle of view of the second image captured by the camera with respect to the ground is changed (ie, relative to the ground The direction (angle) the camera shoots will change).

Accordingly, the size of the area occupied by the road in the image captured by the camera is changed, and a graphic object overlapping with the road and being displayed may be distorted.

When the viewing angle of the second image is changed, the processor 920 may change the graphic object 932b (carpet image) to match the road based on the second image 1020b having the changed viewing angle.

In addition, as shown in FIG. 10E, even when the viewing angle of the camera (ie, the viewing angle of the second image) is changed by the user's manipulation, as described in FIG. 10D, the road and the graphic object overlapping the road are distorted. do.

Even in this case, the processor 920 may change the graphic object 932b (carpet image) to match the road based on the second image 1020b having a changed viewing angle.

In addition, the processor 920 may end the AR mode, as shown in FIG. 10F , if a condition for turning off the AR mode is satisfied while entering the AR mode.

The condition for turning off the AR mode is when there is no transmission of GPS data for a predetermined time (for example, no transmission for 2 seconds or more) or when a delay of a predetermined time or more occurs a specific number of times within a predetermined time (1.5 seconds in 10 seconds). When transmission delay of more than 3 seconds occurs three times), when there is no transmission of gyro data, when there is no transmission of accelerator data, when transmission of SD MAP does not occur for a predetermined period of time, when the image quality of the camera is lower than a certain Hz. This may include falling and the like.

In this case, the processor 920 may end the AR mode and output the execution screen 1000 of the navigation application, as shown in FIG. 10F .

Meanwhile, the processor 920 may output a carpet image overlapping the road included in the second image and guiding a planned driving path of the vehicle to the second image.

The carpet image may overlap the lane on which the vehicle is traveling in the second image.

The processor 920 may output a wall image guiding a driving direction to a lane adjacent to the lane in which the vehicle is driving in the second image.

Referring to FIG. 11A , the processor 920 determines a predetermined distance (eg, a point where a vehicle needs to change direction, such as a right turn, a left turn, or a U-turn rather than going straight at an intersection) where a vehicle needs to change direction. For example, before 100 m), a carpet image 932 may be output on the lane in which the vehicle is running. In this case, the turn-by-turn information 931 may be continuously output to the calibrated second image 1020 .

When the vehicle enters within a predetermined distance based on the intersection in a state where the carpet image 932 overlaps the lane on which the vehicle is traveling, the processor 920 further generates a wall image 933 for guiding the driving direction of the vehicle. can be printed out.

The processor 920 may change the carpet image 932 to the wall image 933 when the vehicle enters within a predetermined distance (eg, 50 m) based on the intersection.

The processor 920 may overlap and output the wall image 933 to a lane adjacent to the lane in which the vehicle is driving.

In other words, the processor 920 outputs the carpet image 932 when the vehicle enters within a predetermined distance (50 m) based on the intersection while the carpet image 932 and the wall image 933 are being output together. may be stopped, and only the wall image 933 may be output.

The processor 920 may overlap and output the wall image 933 to a lane adjacent to the lane in which the vehicle is driving.

The carpet image 932 may be output so as to overlap the lane on which the vehicle is driving, and the wall image (or wall arrow) 933 may be displayed on a lane adjacent to the lane on which the vehicle is driving.

For example, when the wall image 933 is a wall image pointing in a first direction (right), the wall image 933 is adjacent to a lane in which a vehicle is driving, as shown in the second and third drawings of FIG. 11A . Among the left and right lanes, the output may be overlapped with a lane located in a second direction (left) opposite to the first direction.

As shown in the third drawing of FIG. 11A, the processor 930 outputs only the wall image 933 on a road adjacent to the road on which the vehicle is traveling when the vehicle enters within a predetermined distance (eg, 50 m) from the intersection, and , the carpet image 932 may not be output.

When the vehicle changes direction at the intersection (for example, when turning right at the intersection), the processor 920 performs the carpet image (for example, 30 m from the point at which the direction change was performed). 932) may not be output, and a state of outputting the wall image 933 may be maintained.

Thereafter, when the vehicle changes direction at the intersection, the processor 920 changes the wall image 933 to a carpet image 932, as shown in the fourth and fifth drawings of FIG. 11A, and the vehicle is driving. The carpet image 932 may be overlapped and output on the in-road.

As shown in FIG. 11B, when the processor 920 is set to output a wall image 933 from a predetermined distance (eg, 100 m) from an intersection where a direction change is to be performed, the same as shown in the second drawing. Likewise, the wall image 933 may be output to a lane adjacent to the lane in which the vehicle is driving, starting from a location (eg, 200 m before) where the predetermined distance (eg, 100 m) is identified in the image. At this time, as shown in the second drawing of FIG. 11B, the wall image 933 may be output to overlap from the predetermined distance (100 m).

Referring to FIG. 11C , a conversion process between a carpet image 932 and a wall image 933 will be described in more detail.

The processor 920 may output the turn-by-turn information 931 and the carpet image 932 overlapping the lane on which the vehicle is traveling when the vehicle is at least a predetermined distance from the intersection where the vehicle is to change direction. .

When the vehicle enters within a predetermined distance (eg, 100 m) from the intersection, the processor 920 additionally outputs a wall image 933 guiding the driving direction of the vehicle to a lane adjacent to the lane in which the vehicle is traveling. Alternatively, the carpet image 932 may be changed to the wall image 932 .

As shown in FIG. 11C , when the vehicle enters within a predetermined distance from an intersection, the processor 920 outputs a carpet image 932 overlapping the road on which the vehicle is traveling at a location where a wall image 933 is to be output. After conversion, the converted carpet image 932-1 may be output at a position where the wall image is to be output.

Thereafter, the processor 920 may change the converted carpet image 932-1 into a wall image 933.

After the vehicle changes direction at an intersection, the processor 920 converts the wall image 933 to be output at the location where the carpet image 932 is to be output, and converts the converted wall image 933-1 to the carpet image. It can be output to the position to be output (the lane in which the vehicle is driving).

Thereafter, the processor 920 may change the converted wall image 933 - 1 to a carpet image 932 .

As shown in FIGS. 11D and 11E , the processor 920 may enlarge the output size of the wall image 933 as the distance between the vehicle and the intersection increases.

Thereafter, when the direction change of the vehicle is completed, the processor 920 may not output the wall image 933 guiding the direction change.

As shown in FIG. 11F , the processor 920 stops outputting a carpet image when the vehicle cannot receive the GPS signal or enters a tunnel where the strength of the GPS signal is less than the reference level, and the POI (Point of Interest) ) The graphic object 1100 guiding the position of the image (eg, the second image) may be overlapped and output.

In addition, as shown in FIG. 11F, in a state where the carpet image 932 is being output on the lane in which the vehicle is driving, when the vehicle deviate from the planned driving route (Off-route), as shown above, turn-by -At least one of the turn information 931, the wall image 933, and the POI location guide graphic object 1100 may be overlapped with the image and output.

At this time, output of the carpet image may be stopped, and turn-by-turn information 931 may not be output.

12, 13A and 13B are flowcharts for explaining a method of determining a type of an image output to augmented reality according to a driving state of a vehicle according to an embodiment of the present invention.

Referring to FIG. 12 , the processor 920 may determine whether altitude information exists in an on route (S1202). The altitude information may be received from a satellite or an external server through the communication unit 910 .

The processor 920 may output a carpet image 932 when altitude information exists in the route. If it is determined that the altitude information does not exist in the route, a new user interface (UX) may be implemented as augmented reality by overlapping the image.

Referring to FIG. 13A , the processor 920 of the present invention may include an AR adapter 922 and an AR engine 924.

The AR adapter 922 may play a role of converting various information required for route guidance received from a navigation application or navigation system 770 into information (or data) required for implementing augmented reality.

The AR engine 924 may implement route guidance in augmented reality in an image captured through a camera using information converted by the AR adapter 922 .

Referring to FIG. 13A , the navigation application (or system) 770 may transmit the type (or class) of the current road, the location of the current vehicle (GPS information), and curve data of the current road to the AR adapter. (S1302, S1304, S1306, S1308).

The AR adapter 922 may determine whether the current road is a curve (S1310), and if it is not a curve, it may determine an altitude change of the previous predetermined distance (10 m) (S1312).

Altitude change of the last predetermined distance is not more than a specific distance (3m) (S1314), the current road is not local (S1316), there is no next fork (S1318), and the lane (or lane) for the past predetermined time (5 seconds) If there is no (S1320), the AR adapter 922 may request a carpet image from the AR engine for a predetermined time (10 seconds) (S1322).

The requested AR engine 924 may output the carpet image by overlapping it with the image (S1324).

On the other hand, if yes in steps S1310, S1314, S1316, S1318, and S1320, the AR adapter 922 may request the AR engine 924 to output a fixed carpet image or compass image (S1326).

The requested AR engine 924 may output a fixed carpet image or a compass image overlapped with an image (S1328).

Another embodiment of outputting a compass image, a carpet image, or a fixed carpet image may be the same as the flowchart shown in FIG. 13B.

The navigation application (or system) 770 may transmit current location (current GPS) information to the AR adapter 922 .

When the AR adapter 922 has a guide point (eg, an intersection at which a direction change is required) is on the road (S1332) and is within a predetermined distance (300 m) to the next guide point (S1334), there is no guide point. The AR engine 924 may request a compass image for the next road.

In response to the request, the AR engine 924 may overlap the compass image with the image and output the image in augmented reality (S1338).

The AR adapter 922 does not have a guide point (S1332, S1340), calculates an altitude change for the last predetermined distance (10 m) (S1342), and if the altitude change is greater than a specific distance (3 m), a fixed carpet image is displayed at a predetermined distance. It is possible to request the AR engine 924 to output during the time (S1348).

In response to the request, the AR engine 924 may overlap and output the fixed carpet image to the image (S1350).

The fixed carpet image may mean a carpet image with a fixed shape, rather than a carpet image whose shape is varied according to the shape of a road identified in the image.

On the other hand, the AR adapter 922, if there is a guide point on the road, but it is not within a predetermined distance to the next guide point (S1332, 1334), the change in altitude for the last predetermined distance (10 m) can be calculated (S1352).

When the altitude change is within a specific distance (3 m), the AR adapter 922 may request the AR engine 924 to output a carpet image for a predetermined time (10 seconds) (S1346 and S1354).

Based on the request, the AR engine 924 may overlap and output the carpet image 932 to be matched with the lane on which the vehicle is driving (S1358).

If there is no guide point on the road, and it is not a road without a guide point (for example, when it is impossible to determine whether a guide point exists) (S1332, 1340), the AR adapter 922 determines that the vehicle is on a route ( or the road) may be determined to be off the road (S1360).

Operations/functions/control methods performed by the AR adapter 922 and the AR engine 924 may be understood as being performed by the processor 920.

14, 15, 16, 17, 18, 19, 20, 21, 22, and 23 are conceptual diagrams for explaining a compass image output in augmented reality according to an embodiment of the present invention. am.

As reviewed in the flowcharts of FIGS. 12 to 13B, the processor 920 performs calibration on an image obtained through a camera, and then displays a carpet image 932 guiding a planned driving path to a lane in which the vehicle is driving. Augmented reality can be implemented by overlapping to correspond.

On the other hand, as shown in the right figure of FIG. 14, the processor 920 converts the compass image 1400 including the compass object 1410 indicating the direction toward which the front of the vehicle is facing to a calibrated second image. (1020).

The compass image 1400 may include a fixed carpet image 1420 guiding a direction in which the vehicle should drive from the current location.

In addition, the compass image 1400 has a compass object 1410 indicating which direction the front of the vehicle (ie, corresponding to the direction taken by the camera) points and a display position along the edge of the compass object 1410. A fixed carpet image 1420 that is changed may be included.

The processor 920 may vary the display position of the fixed carpet image 1420 along the edge of the compass object 1410 .

The fixed carpet image 1420 may be fixed in a specific shape (for example, a triangular shape) without varying according to the shape of the road, and the direction in which the vehicle should travel from the current location of the vehicle is displayed as a compass object ( 1410) may be guided while moving along.

15, the AR adapter 922 of the processor 920 receives map information (SD MAP) and information on the direction in which the vehicle is driving (GPS heading) from a navigation application (or navigation system) 770 can do.

The AR adapter 922 may convert data applicable to a compass image based on the received information, and then transmit the data to the AR engine 924.

The AR engine 924 may generate a compass image 1400 to be output in augmented reality based on the converted data, and output the compass image 1400 overlapped with the image in augmented reality.

Referring to FIG. 16, a compass image 1400 includes a compass object 1410, direction information 1430 indicating in which direction the vehicle is currently traveling based on north (N), and the vehicle to the next guide point (eg For example, a fixed carpet image 1430 guiding a direction in which a vehicle should travel from a current location in order to drive to an intersection where a direction change is to be performed may be included.

The output position of the fixed carpet image 1430 may be determined and varied based on an angle at which the next guide point is located with respect to the front of the vehicle.

Referring to FIG. 17 , the processor 920 may output a compass image 1400 in various ways.

As described above, the fixed carpet image 1420 may guide the direction in which the vehicle should drive from the current location (ie, the vehicle guides the location of the next guide point from the current location), and the compass object 1410 The display position can be varied along the border of the.

Meanwhile, as shown in FIG. 18 , the processor 920 may output the fixed carpet image 1832 in various ways.

For example, the carpet image 932 described above means a dynamic carpet image, and is generated/varied to correspond to the lane on which the vehicle is driving, overlapped with the lane on which the vehicle is driving, and output as augmented reality. It can be.

On the other hand, the static carpet image 1832 may be displayed bent using the moving direction (or inertia) of the vehicle.

This fixed carpet image is useful for a one-lane curved road, does not use the lane recognition result, is fixed on the screen, and may be formed so that it does not move while driving and only bends.

The processor 920 generates a carpet image 932 for guiding a planned driving path of the vehicle or a compass image for indicating the direction the vehicle faces, based on the distance between the vehicle and the intersection at which the vehicle has to change direction. Any one (or at least one) of 1400 may be output.

In addition, the processor 920 outputs one of a carpet image 932 for guiding a planned driving route of the vehicle or a compass image 1400 for informing the direction of the front of the vehicle, based on the type of road on which the vehicle is traveling. You can output one (or at least one).

Referring to FIG. 19 , the processor 920 displays a carpet image 932 overlapping a lane on which the vehicle is traveling when driving on a preset type of road where straight driving mainly consists of highways/arterial roads/collector roads. It can be output by overlapping the image captured by the camera.

On the other hand, when the road on which the vehicle is driving is a local road (eg, downtown, city center, country road, etc.), the processor 920 may output the compass image 1400 instead of the carpet image 932 .

When the vehicle enters the preset type of road again, the carpet image 932 may be output so as to overlap the road on which the vehicle is driving.

Referring to FIG. 20 , when the location of the vehicle is not a local road (ie, a preset type of road), the processor 920 implements augmented reality by overlapping a carpet image with an image (S2000 and S2010). ), when the location of the vehicle is a local road, augmented reality may be implemented by overlapping the compass image with the image (S2000, S2020).

Referring to FIG. 21 , the processor 920 may change the carpet image 932 to a compass image 1400 based on the vehicle entering a road with a slope.

For example, the processor 920 overlaps the carpet image 932 with the lane on which the vehicle included in the image is driving when the GPS altitude change for the past predetermined time (5 seconds) is within a specific distance (10 m) and outputs the carpet image 932. can do.

As another example, the processor 920 may overlap and output the compass image 1400 to an image captured by a camera when the GPS altitude change during the last predetermined time (5 seconds) is equal to or greater than a specific distance (10 m).

As shown in FIG. 22 , the processor 920 may, based on the distance between the vehicle and the intersection where the vehicle should change direction, use a carpet image guiding a planned driving path of the vehicle or a view from the front of the vehicle. One of the compass images indicating the direction may be overlapped with the image and output.

For example, as shown in FIG. 22 , the processor 920 outputs a carpet image when a distance between the vehicle and an intersection (eg, a guide point) at which a direction change is to be performed is greater than or equal to a predetermined distance, and the vehicle When the distance between the intersection and the intersection is within a predetermined distance (eg, 200 m), the carpet image 932 may not be output and the compass image 1400 may be overlapped with the image and output.

Thereafter, the processor 920 maintains the output of the compass image 1400 up to a predetermined distance (eg, 30 m) after the vehicle passes the intersection, and displays the compass image 1400 when the vehicle passes the predetermined distance (30 m). The output may be stopped, and the carpet image 932 may be overlapped with the image and then output.

As shown in FIG. 23, the processor 920, when a plurality of intersections exist and the distance between the plurality of intersections is within a predetermined distance, for example (within 300 m), the processor 920 determines a first intersection among the plurality of intersections. When a vehicle enters within a predetermined distance before entering, the carpet image 932 may be changed to the compass image 1400 .

Thereafter, the processor 920 may maintain the output of the compass image 1400 until the vehicle passes through the last intersection among the plurality of intersections.

Thereafter, the processor 920 maintains the compass image 1400 until the vehicle travels up to a predetermined distance (eg, 30 m) after passing through the last intersection, and then, when the predetermined distance is passed, the compass image 1400 ) is stopped, and augmented reality may be implemented so that the carpet image 932 overlaps the lane on which the vehicle included in the image is traveling.

24a, 24b, 25, 26, 27a and 27b are conceptual diagrams for explaining a route guidance method according to another embodiment of the present invention.

Referring to FIGS. 24A and 24B , the present invention provides a graphic object output in augmented reality to an image captured by a camera according to a distance between an intersection (or guide point) at which a direction change is to be performed and a vehicle. It can be set in various ways.

Referring to FIG. 24A , the processor 920 may output a carpet image, a wall image, and/or a fixed carpet image when the vehicle enters within a predetermined distance based on the guide point.

When the vehicle enters within a specific distance from the guide point, the processor 920 may stop outputting at least one of a carpet image, a wall image, and a fixed carpet image.

When the vehicle passes the guide point, the processor 920 may output a different type of carpet image or maintain the output of an object being displayed within the specific distance for a predetermined distance.

As shown in FIG. 24B, the processor 920 determines whether a guide point (an intersection or a point of an intersection requiring a change of direction) exists within a predetermined distance and a case where the guide point exists outside a predetermined distance (Road without guide point), it is possible to determine the type of graphic object implemented in augmented reality.

Referring to FIG. 25 , the processor 920 overlaps a fusion carpet image 251, a road carpet image 2502, a fixed carpet image 2503, a compass image 1400, and the like on an image captured by a camera. It can be output in augmented reality.

The fusion carpet image 2501 is a carpet image (e.g., 932) that is mapped and output to a lane on which a vehicle is driving based on a map (SD map), and the road carpet image 2502 is an image captured by a camera. It may be a carpet image output only within the identified lane.

The fixed carpet image 2503 may be a carpet image output based on a driving direction (or movement direction) of the vehicle, rather than a carpet image generated/variable based on a map or image. As an example, the fixed carpet image 2503 may output a carpet image bent to the left when the vehicle is driving to the left.

Compass image 1400, as described above, is the GPS direction received from the navigation application or system 770 (i.e., the direction my vehicle is facing) and the SD MAP direction (i.e., in the direction the vehicle is currently facing). angle at which the next guide point is located).

The processor 920 may output a fusion carpet image if the GPS direction and the SD MAP direction coincide with each other, and may output a carpet image 2502 of a vehicle if they do not match.

Referring to FIG. 26 , a flowchart for explaining an embodiment of outputting various augmented reality graphic objects described above is shown.

The processor 920 may hide the carpet image and output turn-by-turn information (S2602 and S2612) when it is a tunnel or a route (or road) has been deviated.

The processor 920 determines the type of the graphic object overlapped with the image and being output if there is no tunnel or route deviation, no guide point exists, and the user is driving on a road without a guide point (S2602, S2604, S2606).

When the processor 920 is outputting the carpet image of the road to , the processor 920 may output the carpet image of the road to the next road including the guide point (S2608 and S2610).

When the fixed carpet image is being output, the processor 920 may output a carpet image of the road up to the next road including the guide point without sending a lane boundary message (S2632 and S2634).

When the compass image is being output, the processor 920 may hide the carpet image without sending a lane boundary message, and output the carpet image to the next road including the guide point (S2636, S2638, S2640). ).

When the fusion carpet image or the auto carpet image is being output, the processor 920 may determine whether the driving road is sloped or local (S2642, S2644, S2646).

The processor 920 may enter step S2632 when the driving road is sloped or local.

The processor 920 outputs a fusion carpet image up to the next road including a guide point if the image being output is a fusion carpet image when the road being driven is not a slope or a local one, and outputs a guide point if the image being output is an auto carpet image. An auto carpet image may be output up to the next road including the road (S2648, 2650).

The processor 920 outputs a fusion carpet image 932, a wall image 933, a guide point If it is within a specific distance from , the fusion carpet image is not output, and after passing the guide point, the carpet image may be output with a difference of 0 (S2604, S2614, S2616, S2618, S2620, S2622).

If the processor 920 is driving on a road including guide points but deviated from a tunnel or a route, the processor 920 may not transmit a lane warning message, hide a carpet image, and output a compass image overlapped with an image (S2604). S2614, S2624, S2626, S2628).

DS_EGO, DS_ROUTE, and DS_MERGED are terms used by the AR engine and mean Ego Lane, Fusion, and Auto respectively.

In addition, according to the contents of the flowchart shown in the drawing, the processor 920 may output various graphic objects to the image in augmented reality.

27A and 27B illustrate examples of graphic objects output for each situation based on the description of FIG. 26 .

For each situation of FIGS. 27A and 27B , the processor 920 may implement a graphic object that performs route guidance in augmented reality in an image captured through a camera in accordance with each situation.

28 is a conceptual diagram illustrating a method of outputting turn-by-turn information according to an embodiment of the present invention.

The processor 920 determines the slope of the road on which the vehicle travels, and determines the slope of the first graphic object 931 overlapping and outputting the first image (and the second image) based on the determined slope of the road. can

Referring to FIG. 28 , the processor 920 may receive slope information of a road on which the vehicle is currently driving from a vehicle through a communication unit or may receive slope information of a road on which the vehicle is currently driving through a camera.

Then, as shown in FIG. 28 , the processor 920 tilts the first graphic object 931 corresponding to the turn-by-turn information to correspond to the tilt information, and the tilted first graphic object may be overlapped with an image and output in augmented reality.

Through this configuration, the present invention can improve the quality of augmented reality by tilting and overlapping augmented reality graphic objects to correspond to the inclined road.

29a, 29b, 29c, and 29d are conceptual views illustrating various methods of outputting a compass image according to an embodiment of the present invention.

Referring to FIG. 29A , a compass image 1400 including a compass object 1410 indicating a direction the front of the vehicle is facing is output to a calibrated second image 1020. can

The compass image 1400 may include a fixed carpet image 1420 guiding a direction in which the vehicle should drive from the current location.

In addition, the compass image 1400 has a compass object 1410 indicating which direction the front of the vehicle (ie, corresponding to the direction taken by the camera) points and a display position along the edge of the compass object 1410. A fixed carpet image 1420 that is changed may be included.

The processor 920 may vary the display position of the fixed carpet image 1420 along the edge of the compass object 1410 .

The fixed carpet image 1420 may be fixed in a specific shape (for example, a triangular shape) without varying according to the shape of the road, and the direction in which the vehicle should travel from the current location of the vehicle is displayed as a compass object ( 1410) may be guided while moving along.

As shown in FIG. 29B , the processor 920 may output a compass image 1400 overlapped with an image captured by a camera.

The processor 920 may output the compass image in various ways.

For example, as shown in FIGS. 29C and 29D , the processor 920 determines the difference between a direction (azimuth) that the vehicle is currently facing and a direction (angle) in which the next guide point is located relative to the front of the vehicle. If the angle (θ) exceeds 90 degrees, either reduce the size of the compass image (Option 1), or use a fixed carpet image 1430 to guide the direction (angle) of the next guide point based on the front of the vehicle. The output can be maintained at a position of 90 degrees based on the front of the (option 2).

30, 31 and 32 are conceptual diagrams for explaining a route guidance method to which augmented reality is applied according to the present invention.

Referring to FIG. 30 , the processor 920 determines whether the gyroscope direction (eg, the direction in which the vehicle travels by the steering wheel of the vehicle) and the center line do not coincide (eg, the gyroscope direction is bent If there is), a short fixed carpet image 3000 may be output, and if they match (eg, when the gyroscope direction is a straight line), a long fixed carpet image 3020 may be output.

Referring to FIG. 31 , the processor 920 may output various types of wall images according to circumstances.

In order to improve the low visibility and ambiguous direction of the wall image (wall arrow), the route guidance device of the present invention, as shown in FIG. It may be output to the lane adjacent to (type1), output to maintain the lane and direction (type2), or output to indicate crossing the intersection (type3).

The route guidance device of the present invention may output a carpet image and a wall image to an image captured through a camera in augmented reality, and perform route guidance optimized for a user.

Effects of the route guidance device and the route guidance method according to the present invention are described as follows.

According to the present invention, the passenger may be provided with information on a route on which the vehicle autonomously drives or on which the driver should drive in units of lanes through a carpet image.

In addition, according to the present invention, a passenger can determine a path on which a vehicle should travel in an optimized way through images of various types of carpets.

In addition, according to the present invention, the present invention can provide a new route guidance interface capable of guiding a driving route of a vehicle through a compass image.

The above-described present invention can be implemented as computer readable code (or application or software) on a medium on which a program is recorded. The control method of the self-driving vehicle described above may be realized by a code stored in a memory or the like.

The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. , and also includes those implemented in the form of a carrier wave (eg, transmission over the Internet). Also, the computer may include a processor or a control unit. Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (20)

a communication unit for receiving a first image captured by a camera; and
A route guidance device including a processor that performs calibration on the first image and outputs a graphic object guiding vehicle driving by overlapping the calibrated second image. According to claim 1,
the processor,
Before the calibration is performed, a first graphic object related to driving is output to the first image,
After the calibration is performed, the first graphic object and the second graphic object related to driving are overlapped with the second image and outputted. According to claim 2,
The first graphic object includes turn-by-turn information indicating a road to be entered in front of a predetermined distance,
The second graphic object includes a carpet image overlapping a road included in the second image and guiding a planned driving path of the vehicle. According to claim 3,
the processor,
overlapping and outputting the second graphic object on a lane on which a vehicle is traveling among a plurality of lanes included in the second image;
When it is determined that the vehicle departs from the driving lane, the route guidance device characterized in that the second graphic object is not output. According to claim 4,
The route guidance device, characterized in that the output of the first graphic object is maintained regardless of whether the vehicle leaves the driving lane. According to claim 1,
the processor,
The route guidance device characterized in that performing calibration so that the road area included in the first image has a predetermined size or more. According to claim 1,
the processor,
When the viewing angle of the second image is changed so that a road included in the second image and a graphic object overlapping the road are displaced by driving of the vehicle, the viewing angle is matched to the lane based on the second image having a changed viewing angle. A route guidance device characterized by changing a graphic object. According to claim 1,
the processor,
Outputting a carpet image overlapping a road included in the second image and guiding a planned driving path of the vehicle to the second image;
The carpet image overlaps the lane on which the vehicle is running in the second image,
the processor,
A path guidance device characterized in that outputting a wall image for guiding a driving direction to a lane adjacent to a lane in which the vehicle is driving in the second image. According to claim 8,
the processor,
Outputting the carpet image on a lane in which the vehicle is traveling before a predetermined distance before entering an intersection where the vehicle needs to change direction;
When the vehicle enters within the predetermined distance based on the intersection, the carpet image is changed to the wall image;
The route guidance device according to claim 1 , wherein the wall image is overlapped and output to a lane adjacent to a lane in which the vehicle is running. According to claim 9,
the processor,
When the vehicle changes direction at the intersection, the wall image is changed to the carpet image;
A route guidance device characterized in that outputting an overlapping carpet image on a lane in which the vehicle is traveling. According to claim 8,
The route guidance device according to claim 1 , wherein the processor enlarges an output size of the wall image as the distance between the vehicle and the intersection decreases. According to claim 1,
the processor,
A route guidance device characterized in that outputting a compass image including a compass object indicating a direction from which the front of the vehicle is viewed overlapped with the second image. According to claim 12,
The compass image,
A route guidance device comprising a fixed carpet image for guiding a direction in which a vehicle should drive from a current location. According to claim 13,
the processor,
Route guidance device, characterized in that for varying the display position of the fixed carpet image along the edge of the compass object. According to claim 1,
the processor,
Based on the distance between the vehicle and the intersection where the vehicle must change direction, either a carpet image guiding the vehicle's scheduled driving route or a compass image indicating the direction the vehicle is facing is output. A route guidance device with According to claim 2,
the processor,
Determine the slope of the road on which the vehicle is traveling,
Based on the determined slope of the road, a slope of the first graphic object to overlap and output to the first image is determined. Receiving a first image captured by a camera; and
and performing calibration on the first image and overlapping and outputting a graphic object guiding driving of the vehicle on the calibrated second image. According to claim 1,
The outputting step is
Before the calibration is performed, a first graphic object related to driving is output to the first image,
After the calibration is performed, the first graphic object and the second graphic object related to driving are overlapped with the second image and outputted. According to claim 18,
The first graphic object includes turn-by-turn information indicating a road to be entered in front of a predetermined distance,
The route guidance method of claim 1 , wherein the second graphic object includes a carpet image overlapping a road included in the second image and guiding a scheduled driving route of the vehicle. According to claim 19,
The outputting step is
overlapping and outputting the second graphic object on a lane on which a vehicle is driving among a plurality of lanes included in the image;
and not outputting the second graphic object when it is determined that the vehicle deviates from the driving lane.

Images