KR20220029930A - Vehicle management apparatus - Google Patents

Abstract

The present invention relates to a vehicle management apparatus to increase accuracy of vehicle damage determination through a plurality of images of the external of a vehicle. According to one embodiment of the present invention, the vehicle management apparatus comprises: an output unit outputting a damage image and a control unit controlling the output unit to output the damage image on the basis of a first image captured by a camera at a first viewpoint and a second image captured by a camera at a second viewpoint. The first image includes a first area of the exterior of a vehicle and at least one fixed area and a variable area are included inside the first area. The second image includes a second area of the exterior of the vehicle and at least a part of the fixed area and at least a part of the variable area are included in the second area. The damage image is an image in which the variable area is removed from the first image and the second image, wherein the fixed area is an area for damage to the vehicle, the variable area is an area not related to the damage to the vehicle, and the first viewpoint is different from the second viewpoint.

Description

Vehicle Management Device {VEHICLE MANAGEMENT APPARATUS}

The present invention relates to a vehicle management apparatus, and more particularly, to a vehicle management apparatus for improving the accuracy of vehicle damage determination through a plurality of images related to the exterior of a vehicle.

As the economy develops and society becomes more sophisticated, the sales volume of vehicles continues to increase. Accordingly, vehicle-related industries such as parking lot, used car transaction, car insurance, rental car, and shared car management are also gradually diversifying, and the size of the market is also increasing day by day.

As the size of the relevant market increases, the necessity of evaluating the condition or value of a vehicle and continuously managing the vehicle according to the needs of various consumers is also increasing. In particular, if quality control fails due to damage to the vehicle, etc., there is a possibility that this may lead to not only a decrease in the value of the vehicle but also an accident. However, it is still common to input manpower by mobilizing experts for vehicle quality control, so access is insufficient for general consumers.

Accordingly, there is a need to develop a technology that allows general consumers to easily manage the state of a vehicle in various industrial environments.

An object of the present invention is to provide a vehicle management apparatus for managing a damaged state of a vehicle.

Another object of the present invention is to provide a vehicle management apparatus for improving the accuracy of vehicle damage determination through a plurality of images related to the exterior of the vehicle.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

According to an aspect of the present invention, the damaged image is output based on a first image taken at a first time point by an output unit and a camera for outputting a damaged image and a second image taken at a second time point by the camera and a control unit for controlling the output unit to In the second image, a second area related to the exterior of the vehicle is photographed, and at least some of the fixed area and at least some of the variable area are included in the second area, and the damaged image is An image in which the variable region is removed from the first image and the second image, wherein the fixed region is a region for vehicle damage, the variable region is a region not for vehicle damage, and the first time point The vehicle management device and the second time point may be different from each other.

The solutions to the problems of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. will be able

According to the present invention, a vehicle management apparatus for managing a damaged state of a vehicle can be provided.

According to the present invention, a vehicle management apparatus for improving the accuracy of vehicle damage determination through a plurality of images related to the exterior of the vehicle may be obtained.

Effects of the present invention are not limited to the above-described effects, and effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention pertains from the present specification and accompanying drawings.

1 is a diagram for describing a vehicle management method according to an exemplary embodiment.
2 is a diagram for describing a vehicle image according to an exemplary embodiment.
3 is a diagram for illustrating image data according to an embodiment.
4 is a diagram illustrating image data according to an embodiment.
5 is a diagram illustrating image data according to another embodiment.
6 is a diagram illustrating image data according to an embodiment.
7 is a diagram for explaining noise removal processing.
8 is a diagram for illustrating noise removal processing using a plurality of images.
9 is a diagram for illustrating noise removal processing using a plurality of images.
10 is a diagram for explaining a neural network model for acquiring vehicle information according to an embodiment.
11 is a diagram for explaining a neural network model for acquiring vehicle information according to an embodiment.
12 is a diagram for explaining a neural network model for acquiring vehicle information according to an embodiment.
13 is a diagram illustrating a neural network model according to an embodiment.
14 is a diagram for describing part area information according to an exemplary embodiment.
15 is a diagram for describing a vehicle image and part information.
16 is a diagram for describing a vehicle image and part information.
17 is a diagram illustrating parts learning data according to an embodiment.
18 is a diagram illustrating parts learning data according to another embodiment.
19 is a view for explaining damage information according to an embodiment.
20 is a diagram for explaining damaged area information.
21 is a diagram for illustrating impairment learning data.
22 is a diagram for explaining a device for acquiring part-damage information.
23 is a diagram for explaining acquisition of vehicle information according to an exemplary embodiment.
24 is a diagram for explaining a device for acquiring part-damage information.
25 is a diagram for explaining a device for acquiring part-damage information.
26 is a view for explaining new damage information according to an embodiment.
27 is a view for explaining damage history management.
28 is a diagram for explaining an output screen according to an exemplary embodiment.
29 is a view for explaining the display of damaged area information according to an embodiment.
30 is a diagram for describing a vehicle management assistance method according to an exemplary embodiment.
31 is a view for explaining a vehicle damage management method according to an embodiment.
32 is a view for explaining a vehicle damage management method according to an embodiment.
33 is a view for explaining a vehicle damage management method according to an embodiment.
34 is a diagram for explaining the configuration of an on-site vehicle management system.
35 is a diagram for explaining a series of steps performed by the on-site vehicle management process.
36 is a diagram for explaining a step of acquiring vehicle data required for vehicle management in an on-site vehicle management process.
37 is a diagram for explaining that image data is captured by the camera module at a specific point in time when image data is sensed by a sensing means according to an embodiment.
38 is a diagram for explaining that image data is captured by a camera module at a specific point in time when image data is sensed by a sensor, according to another embodiment.
FIG. 39 is a diagram for explaining that image data is captured by a camera module at a specific point in time when image data is sensed by a sensor, according to another embodiment.
40 is a view for explaining that image data is captured by the camera module at a time within a predetermined range from a time when a circuit breaker operates according to another embodiment.
41 is a diagram for describing a method of extracting image data from image data according to an exemplary embodiment.
42 is a diagram for illustrating noise removal processing using a plurality of images.
43 is a diagram for illustrating noise removal processing using a plurality of images.
44 is a view for explaining a screen for outputting damage information among the output screens of the management device.
45 is a view for explaining a screen for selecting whether a user or an administrator agrees to damage information among the output screens of the management device.
46 is a view for explaining a screen including detailed information regarding damage information among the output screens of the management device.
47 is a view for explaining an output screen including information about a damage history among the output screens of the management device.
48 is a view for explaining an output screen including information about a damage history among the output screens of the management device.
49 is a diagram for explaining the configuration of a mobile-based vehicle management system.
50 is a diagram for explaining a series of steps performed by the mobile-based vehicle management process.
51 is a diagram for explaining a request for re-photographing when vehicle damage information included in image data is not sufficient, according to an exemplary embodiment.
52 is a diagram for explaining that image data is captured by a camera module at a specific time point according to an embodiment.
53 is a view for explaining a photographing guide provided based on information on a target vehicle when photographing a vehicle with a camera module.
54 is a view for explaining a photographing map provided when photographing a vehicle with a camera module.
55 is a diagram for explaining a screen for outputting vehicle information and/or auxiliary information in a terminal according to an embodiment.
56 is a diagram schematically illustrating a process in which a plurality of shared vehicle users use the shared vehicle and end the use according to time sequence.
57 is a diagram for explaining a vehicle return process in which a shared vehicle user takes a picture of the vehicle exterior and checks additional damage information after using the shared vehicle.
58 is a diagram for explaining a vehicle use process in which a shared vehicle user displays whether or not to agree to the existing damage information of the shared vehicle before using the shared vehicle, and takes a picture of the exterior of the vehicle when the user does not agree.
59 is a view for explaining a process of re-photographing the appearance of the shared vehicle when the shared vehicle user does not agree with the existing damage information of the shared vehicle.

The above-mentioned objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. However, since the present invention may have various changes and may have various embodiments, specific embodiments will be exemplified in the drawings and described in detail below.

In the drawings, the thicknesses of layers and regions are exaggerated for clarity, and an element or layer is also referred to as “on” or “on” another component or layer. It includes all cases where another layer or other component is interposed in the middle as well as directly on top of another component or layer. Throughout the specification, like reference numerals refer to like elements in principle. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

If it is determined that a detailed description of a known function or configuration related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of the present specification are merely identification symbols for distinguishing one component from other components.

In addition, the suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

According to an example, there is provided a vehicle management assistance apparatus for obtaining vehicle information related to the outside of a vehicle based on a vehicle image, comprising: an image obtaining unit which obtains the vehicle image obtained by photographing the outside of the vehicle; an additional information acquisition unit configured to acquire additional information related to the characteristics of the vehicle; and a processing unit configured to obtain the vehicle information using a neural network model trained to obtain the vehicle information based on the vehicle image. wherein the neural network model is trained to obtain a first feature set based on the vehicle image and a first part comprising at least one convolutional neural network layer and a first part based on the first feature set and the additional information. a second part learned to acquire two feature sets, wherein the processing unit acquires a target vehicle image obtained by photographing the exterior of the target vehicle through the image acquisition unit, and the characteristics of the target vehicle through the additional information acquisition unit A vehicle management assistance apparatus may be provided that obtains additional target information related to , and acquires target vehicle information on the target using the neural network model based on the target vehicle image.

The neural network model may be trained to obtain damage information related to damage outside the vehicle, and the second feature set may include at least one feature value related to whether or not the vehicle is damaged outside.

The neural network model may be trained to acquire part information related to parts constituting the exterior of the vehicle, and the second feature set may include at least one feature value related to parts constituting the exterior of the vehicle.

The additional information related to the characteristics of the vehicle may be determined based on at least one of a manufacturer, color information, year, and vehicle model of the vehicle.

The additional information related to the characteristic of the vehicle may be obtained based on the vehicle interior image obtained by photographing the interior of the vehicle.

the second part acquires the second feature set related to at least one part constituting the exterior of the vehicle based on the first feature set and the additional information, wherein the second feature set comprises the first part a first set of domain features representing the area of distribution and a second set of domain features representing areas over which the second part is distributed.

the second part obtains, based on the first set of features and the additional information, the second set of features associated with at least one damaged area located outside of the target vehicle, wherein the second set of features includes: It may include a set of damaged area features indicating the location of the area.

According to an example, vehicle management comprising an image acquisition unit acquiring a vehicle image, an additional information acquiring unit acquiring additional information related to characteristics of a vehicle, and a processing unit acquiring vehicle information based on the vehicle image and the additional information A vehicle management assistance method for acquiring information related to the exterior of the vehicle based on the vehicle image by using an auxiliary device, the method comprising: acquiring a target vehicle image of the target vehicle through the image acquisition unit; acquiring additional target information related to the characteristics of the target vehicle through the additional information acquiring unit; and obtaining, by the processing unit, target vehicle information on the target vehicle based on the target additional information and the target vehicle image; Including, wherein the obtaining of the vehicle information on the target vehicle includes: Based on the target vehicle image, using a neural network model trained to obtain the vehicle information based on the vehicle image, to the target vehicle and obtaining the target vehicle information for the first part and the first feature, wherein the neural network model is trained to obtain a first feature set based on the vehicle image and includes at least one convolutional neural network layer and a second part learned to obtain a second feature set based on the set and the additional information.

The neural network model may be trained to obtain damage information related to damage outside the vehicle, and the second feature set may include at least one feature value related to whether or not the vehicle is damaged outside.

The neural network model may be trained to acquire part information related to parts constituting the exterior of the vehicle, and the second feature set may include at least one feature value related to parts constituting the exterior of the vehicle.

The additional information related to the characteristics of the vehicle may be determined based on at least one of a manufacturer, color information, year, and vehicle model of the vehicle.

The additional information related to the characteristic of the vehicle may be obtained based on the vehicle interior image obtained by photographing the interior of the vehicle.

The second part acquires the second feature set related to at least one part constituting the exterior of the target vehicle, wherein the second feature set is the first feature set based on the first feature set and the additional information and a first set of region features representative of an area in which the part is distributed and a second set of region features representative of an region in which the second part is distributed.

the second part obtains, based on the first set of features and the additional information, the second set of features associated with at least one damaged area located outside of the target vehicle, wherein the second set of features includes: It may include a set of damaged area features indicating the location of the area.

A computer-readable recording medium storing a program for performing the vehicle management assistance method may be provided.

According to an example, an output unit for outputting a damaged image; and a control unit controlling the output unit to output the damaged image based on a first image photographed at a first time point by the camera and a second image photographed at a second time point by the camera, wherein the first In the image, a first area related to the exterior of the vehicle is photographed, and the first area includes at least one or more fixed areas and variable areas, which are at least one area, and the second image is a second area related to the exterior of the vehicle. At least a partial region of the fixed region and at least a partial region of the variable region are included in the inside of the second region, and the variable region is removed from the first image and the second image in the damaged image. image, wherein the fixed area is an area for damage to the vehicle, the variable area is an area not for damage to the vehicle, and the first time point and the second time point are different points of view. can

The first image and the second image include a first feature point among a plurality of feature points related to a vehicle, the first feature point of the first image has a first coordinate, and the first feature point of the second image is a first 2 coordinates, the coordinates representing the fixed region included in the first image are third coordinates, the coordinates representing the fixed region included in the second image are fourth coordinates, and the first coordinate and The first distance between the third coordinates and the second distance between the second coordinates and the fourth coordinates may correspond to each other.

The coordinates representing the variable region included in the first image are fifth coordinates, the coordinates representing the variable region included in the second image are the sixth coordinates, and the coordinates between the first coordinates and the fifth coordinates are The third distance may provide a vehicle management device different from a fourth distance between the second coordinate and the sixth coordinate.

A difference between the first distance and the second distance may be smaller than a difference between the third distance and the fourth distance.

The control unit classifies a fixed region and a variable region in the abnormal region, outputs the damaged image from which the variable region is removed, and the first image and the second image include a first feature point among a plurality of feature points related to the vehicle, , the first feature point of the first image has a first coordinate, the first feature point of the second image has a second coordinate, and the coordinate representing the abnormal region of the first image is a seventh coordinate, The coordinates representing the abnormal region of the second image are eighth coordinates, and the controller is configured to measure a fifth distance between the first coordinate and the seventh coordinate and a sixth distance between the second coordinate and the eighth coordinate. It is possible to provide a vehicle management apparatus for classifying the fixed region and the variable region based on the above.

When the difference between the fifth distance and the sixth distance is equal to or greater than a predetermined value, the control unit determines as a variable region, and when the difference between the fifth distance and the sixth distance is less than a predetermined value, the vehicle management apparatus determines as a fixed region. can provide

The variable region may provide a vehicle management device that is a background region outside the vehicle.

The variable region may provide a vehicle management apparatus that is a region included in an image by reflecting an external object by external light.

The damaged image may be an image in which the variable region is removed from a first image, and the first image may be an image in which the vehicle is located closer to a central region of the image compared to the second image.

Further comprising a communication unit for receiving a signal from the sensor unit, wherein the sensor unit includes a first sensor and a second sensor, the first sensor is located in an area close to the vehicle's entrance direction, the second sensor is the vehicle's entrance It is possible to provide a vehicle management device located in an area remote from the direction.

The first time point and the second time point are time points within an effective period between the start time point and the end time point, the start time point is a time point when the communication unit receives the first signal from the first sensor, and the end time point is the time point when the communication unit receives the first signal from the first sensor It is the time when the second signal is received from the second sensor. The first image and the second image may provide a vehicle management device included in a plurality of images acquired within the validity period.

The start time may be a time when the vehicle is detected by the first sensor and then the detection of the vehicle is finished, and the end time may be a time when the vehicle is first detected by the second sensor.

The start time may be a time when the vehicle is first detected by the first sensor, and the end time may be a time when the detection is terminated after the vehicle is detected by the second sensor.

Further comprising a communication unit for receiving the open information from the breaker, wherein the first time point and the second time point are time points within the effective period between the start time and the end time, the time when the breaker is opened is the end time, the breaker is open It is possible to provide a vehicle management apparatus that is included in a plurality of images acquired within the validity period, wherein a predetermined time point before the predetermined time point is a start time point.

The camera includes a first camera and a second camera, wherein the first camera is located in an area corresponding to the first sensor to photograph one side of the vehicle, and the second camera corresponds to the second sensor It is possible to provide a vehicle management device that is located in the area where the vehicle is used and captures the other side of the vehicle.

The first image and the second image include a first feature point among a plurality of feature points related to a vehicle, and at the first time point, the camera is located at a position having a first distance from the first feature point, and at the second time point The camera may be located at a position having a second distance from the first feature point, and the first distance and the second distance may be different from each other.

The vehicle management apparatus may be a mobile device, and the camera may be a camera built into the mobile device.

The camera may capture a moving picture between a start time and an end time, and the first image and the second image may be images extracted from the moving image.

According to an example, there is provided a vehicle damage management apparatus for acquiring information related to damage to the outside of a vehicle based on a vehicle image, comprising: an image acquisition unit configured to acquire at least one target vehicle image obtained by photographing an exterior of a target vehicle; and a processing unit configured to obtain damage information related to exterior damage of the target vehicle based on the at least one target vehicle image. The at least one target vehicle image includes a first target vehicle image obtained by photographing one surface of the target vehicle, and the first target vehicle image includes a first area corresponding to the target vehicle and the and a second area in which identification information of a target vehicle is located, wherein the first area is larger than the second area, wherein the processing unit includes the vehicle model information and the vehicle model information of the target vehicle determined based on the second area and a vehicle damage management apparatus that is obtained based on the first target vehicle image and obtains the damage information related to the first area.

an output unit for outputting the damage information; Further comprising, the output unit may provide a vehicle damage management apparatus for outputting the damage information of the target vehicle.

The damage information may include damage area information corresponding to damage to the target vehicle, and the output unit may provide a vehicle damage management apparatus that outputs the target vehicle image and the damage area information superimposed on the target vehicle image .

An input unit for obtaining a user input; further comprising, wherein the processing unit, in response to the output of the damage information, obtains a user response to the damage information through the input unit, it is possible to provide a vehicle damage management apparatus.

The processing unit may provide an apparatus for managing vehicle damage to obtain identification information of the target vehicle determined based on the second area.

The processing unit obtains identification information of the target vehicle determined based on the second region, obtains a damage history of the target vehicle corresponding to the identification information of the target vehicle, and the damage history includes: It is possible to provide an apparatus for managing vehicle damage, which is obtained based on a second target vehicle image captured before the vehicle image and includes existing damage information related to the exterior damage of the target vehicle.

The processing unit may provide a vehicle damage management apparatus that acquires new damage information corresponding to a time point at which the first target vehicle image of the target vehicle is captured, based on the existing damage information and the damage information.

The damage information may include first damage information related to the second area included in the first target vehicle image and second damage information stored in advance in association with identification information of the target vehicle obtained based on the first area. It is possible to provide a vehicle damage management device, which is generated based on the

The at least one target vehicle image further includes a second target vehicle image obtained by photographing another surface different from the one surface of the target vehicle, wherein the second target vehicle image includes a third area corresponding to the target vehicle and the processing unit vehicle model information of the target vehicle determined based on the second area of the first target vehicle image and the damage related to the third area obtained based on the vehicle model information and the second target vehicle image It is possible to provide a vehicle damage management apparatus for obtaining information.

The processing unit may be configured to include, based on the first target region image and the vehicle model information, a first part region included in the first region and corresponding to a first part of the target vehicle and a first part region included in the first region and the target vehicle It is possible to provide an apparatus for managing vehicle damage that obtains parts information including a second part area corresponding to the second part of the .

The processing unit, based on the damage information and the parts information, may provide a vehicle damage management apparatus for acquiring parts area information corresponding to the damage.

The processing unit obtains the second area corresponding to the license plate of the target vehicle from the first target vehicle image, and provides a vehicle damage management method for obtaining the vehicle model information based on the second area corresponding to the license plate can do.

The processing unit may provide a vehicle damage management method for obtaining the damage information based on the target vehicle image by using a neural network model trained to obtain the damage information based on the target vehicle image.

The damage information may provide the vehicle damage management apparatus including area information indicating an area corresponding to the damage outside the vehicle among the first area.

According to an example, in a vehicle damage management method using an apparatus including an image acquisition unit for acquiring a vehicle image and a processing unit for acquiring damage information, at least one obtained by photographing the exterior of a target vehicle through the image acquisition unit Acquiring a target vehicle image of - The at least one target vehicle image includes a first target vehicle image obtained by photographing one surface of the target vehicle, and the first target vehicle image corresponds to the target vehicle. comprising a first area and a second area in which identification information of the target vehicle is located; obtaining, through the processing unit, vehicle model information of the target vehicle determined based on the second area; and acquiring, through the processing unit, damage information related to exterior damage of the target vehicle based on vehicle model information and the first target vehicle image. It is possible to provide a vehicle damage management method comprising a.

outputting the damage information; further comprising, wherein the damage information includes damage area information corresponding to damage to the target vehicle, and outputting the damage information includes outputting the damage area information superimposed on the target vehicle image and the target vehicle image It is possible to provide a vehicle damage management method further comprising that.

According to an example, the method may include: acquiring, through the processing unit, identification information of the target vehicle determined based on the second area; Further comprising, it is possible to provide a vehicle damage management method.

Obtaining, through the processing unit, a damage history of the target vehicle corresponding to the identification information of the target vehicle - The damage history is obtained based on a second target vehicle image taken before the first target vehicle image, and It is possible to provide a vehicle damage management method further comprising a-; including existing damage information related to the damage to the exterior of the target vehicle.

acquiring, through the processing unit, new damage information corresponding to a time point at which the first target vehicle image of the target vehicle is captured, based on the existing damage information and the damage information; It is possible to provide a vehicle damage management method further comprising a.

The damage information may be included in the first area and may provide a vehicle damage management method including damage area information corresponding to damage to the target vehicle.

The at least one target vehicle image further includes a second target vehicle image obtained by photographing another surface different from the one surface of the target vehicle, wherein the second target vehicle image includes a third area corresponding to the target vehicle and obtaining the damage information includes: vehicle model information of the target vehicle determined based on the second region of the first target vehicle image, and obtained based on the vehicle model information and the second target vehicle image, and It is possible to provide a vehicle damage management method further comprising obtaining the damage information related to area 3 .

Through the processing unit, based on the image of the first target region and the vehicle model information, a first part region included in the first region and corresponding to a first part of the target vehicle and a first part region included in the first region and the target vehicle obtaining part information including a second part area corresponding to a second part of the vehicle; It is possible to provide a vehicle damage management method further comprising a.

Through the processing unit, based on the damage information and the parts information, obtaining part region information corresponding to the damage; may provide a vehicle damage management method further comprising.

obtaining, by the processing unit, the second area corresponding to the license plate of the target vehicle from the first target vehicle image; The method may further include, wherein the acquiring of the vehicle model information may provide a vehicle damage management method further comprising acquiring the vehicle model information based on the second area corresponding to the license plate.

The obtaining of the damage information includes obtaining the damage information based on the target vehicle image by using a neural network model trained to obtain the damage information based on the target vehicle image. can provide

The vehicle damage management apparatus further includes an input unit for obtaining a user input, and in response to the processing unit outputting the damage information, obtaining a user response to the damage information through the input unit; It is possible to provide a vehicle damage management method further comprising a.

As an example, it is possible to provide a computer-readable recording medium storing a program for performing the vehicle damage management method.

For example, a camera for taking an image of the vehicle; an input unit for receiving information from a user; a communication unit for transmitting and receiving wireless signals; Display unit for displaying a message related to the existing damage information; and a control unit for controlling the camera, the communication unit and the display unit, wherein the user includes a previous user and a current user, and the control unit provides the existing damage information to the current user through the display unit, and the existing damage information When the current user agrees with respect to It is possible to provide a mobile device that displays a first notification window through which the current user can select whether to agree or not.

The first image may be an image taken by the previous user when returning the vehicle, and the existing damage information may mean damage information that occurred until the previous user's vehicle use ends.

The damage information may provide the mobile device which is information updated by the first image sent by the mobile device of the previous user.

The display unit, when the current user does not agree with the existing damage information, displays a second notification window requesting the current user to take a second image, and the control unit displays the second notification window when the second image shooting is completed It is possible to provide a mobile device that transmits an image to a server through the communication unit and changes the vehicle to a usable state.

When the current user does not agree with all of the existing damage information, the display unit displays a second notification window requesting the current user to take a second image of the entire exterior of the vehicle, and the control unit displays the second image When photographing is completed, a mobile device that transmits the second image to the server through the communication unit and changes the vehicle to a usable state may be provided.

When the current user does not partially agree to the existing damage information, the display unit displays a second notification window requesting the current user to take a second image of a portion of the vehicle exterior that the current user does not agree to, , the control unit may provide a mobile device that transmits the second image to the server through the communication unit when capturing the second image is completed and changes the vehicle to a usable state.

The display unit may provide a mobile device that displays a shooting guide, wherein the shooting guide guides the entire exterior of the vehicle or parts of the vehicle to be photographed when the current user shoots the second image there is.

The photographing guide may guide photographing based on vehicle information, and the vehicle information may be obtained through a neural network model trained to obtain information or vehicle information input by the current user. can

When the current user captures the second image, the controller may provide a mobile device that automatically executes a photographing button to store the image when the vehicle is located in the photographing guide by a predetermined ratio or more.

The display unit displays a photographing map, wherein the photographing map guides photographing so that at least some of the main parts of the exterior of the vehicle can be included in the second image when the current user photographs the second image device can be provided.

The control unit receives a comparison result of which the server compares the second image with the existing damage information to determine whether additional damage has occurred to the vehicle through the communication unit, and the control unit transmits the comparison result through the display unit to the present It is provided to the user, and the display unit may provide a mobile device that displays a third notification window through which the current user can select whether to agree to the comparison result.

When the current user agrees to the comparison result, the controller may provide a mobile device that changes the vehicle to a usable state.

When the current user does not agree with the comparison result, the display unit displays a message that can object to the comparison result, and the control unit changes the vehicle to a usable state when the current user completes the objection. device can be provided.

For example, a camera for taking an image of the vehicle; an input unit for receiving information from a user; a communication unit for transmitting and receiving wireless signals; a display unit for displaying a message related to vehicle damage information; and a control unit for controlling the camera, the communication unit and the display unit, wherein the user includes a previous user and a current user, and the display unit requests the current user to take a third image when returning the vehicle of the current user A fourth notification window may be displayed, and when capturing the third image is completed, the controller may provide a mobile device that transmits the third image to the server through the communication unit and processes to complete the return process of the vehicle.

The display unit may provide a mobile device that displays a photographing guide, and the photographing guide guides the overall appearance of the vehicle or parts of the vehicle to be photographed when the current user photographs the third image. there is.

The display unit displays a photographing map, wherein the photographing map guides photographing so that at least some of the main parts of the exterior of the vehicle can be included in the second image when the current user photographs the third image device can be provided.

The control unit receives the additional damage information that the server compares the third image with the existing damage information to determine whether additional damage has occurred to the vehicle through the communication unit, and the control unit receives the additional damage information through the display unit It is provided to the current user, and the display unit may provide a mobile device displaying a fifth notification window through which the current user can select whether to agree to the additional damage information.

The additional damage information may provide a mobile device that means damage information that has occurred from the start time of the current user's vehicle use to the end time of the current user's vehicle use.

When the current user agrees to the additional damage information, the control unit may provide a mobile device that processes the return process of the vehicle to be completed.

The display unit displays a message that can be objected when the current user does not agree to the additional damage information, and the control unit allows the return process of the vehicle to be completed when the current user completes the objection It is possible to provide a mobile device for processing.

Hereinafter, a vehicle management apparatus according to an embodiment will be described with reference to the drawings.

1. Overview

In this specification, an apparatus, method, and system for managing the state of a vehicle will be described. Hereinafter, a device for managing the state of a vehicle, etc. will be described with reference to some embodiments.

2. Vehicle management system

According to an embodiment, a vehicle management system for managing the state of a vehicle using an image captured by the vehicle may be provided.

The vehicle management system may include at least one device. The vehicle management system may include a device having at least one camera module. The vehicle management system may include a device having at least one control unit. The vehicle management system may include at least one device including an input module for obtaining a user input. The vehicle management system may include at least one device having a storage module. The vehicle management system may include at least one device having a communication module.

The vehicle management system may include a user device such as a tablet, a smartphone, a PC, or a laptop computer. The user device may include a display, a user interface, a communication module, a camera module, a memory and/or a controller.

The vehicle management system may include a server device. The server device may store data or programs. The server device may communicate with the user device.

The vehicle management system may include a vehicle photographing device. The vehicle imaging device may communicate with a server device and/or a user device.

2.1 Fleet Management Process

According to an embodiment, a vehicle management process for managing the state of the vehicle using an image captured by the vehicle may be provided. The vehicle management process may be performed by the aforementioned vehicle management system.

Referring to FIG. 1 , the vehicle management process obtains vehicle data related to a vehicle (S110), obtains vehicle information based on the obtained vehicle data (S130), manages vehicle information (S150), and collects vehicle information It may include outputting (S170). Hereinafter, each step will be described in more detail.

2.2 Data Acquisition

The vehicle management process may include acquiring vehicle data required for vehicle management.

The vehicle data may be provided in various forms. For example, the vehicle data may be image data or non-image data. The vehicle data may be pre-stored in the device or obtained from outside. For example, the vehicle data may be obtained through a camera module or may be obtained through a user input module.

The vehicle data and the acquisition of the vehicle data will be described in more detail below with reference to specific examples.

2.2.1 Image data acquisition

According to an embodiment, the vehicle data may be image data including a vehicle image obtained by photographing the vehicle. The vehicle image may be an image including some parts of the vehicle. The vehicle image may be an image obtained by photographing a part of the vehicle to include identification information for identifying the vehicle.

The vehicle data may be vehicle image data obtained by photographing the vehicle. The vehicle data may include image data and image data extracted from the image data. Hereinafter, an embodiment of vehicle data will be described based on image data. However, the vehicle data may be acquired in the form of a vehicle image corresponding to the vehicle image of the form described below.

2 is a diagram for describing a vehicle image according to an exemplary embodiment.

Referring to FIG. 2A , a vehicle image according to an exemplary embodiment may be an image obtained by photographing the vehicle in one direction, for example, from a side. The vehicle image may be an image obtained by photographing so that the entire vehicle enters the frame in one direction.

Referring to FIG. 2B , the vehicle image according to an exemplary embodiment may be an image obtained by photographing the vehicle from the front. The vehicle image may include identification information for identifying the vehicle. The identification information may be a vehicle identification number (VIN) or a vehicle number. Referring to FIG. 2B , the vehicle image according to an exemplary embodiment may include a VIN area in which the VIN of the vehicle is displayed or a license plate area in which a vehicle number is displayed.

Referring to FIG. 2C , a vehicle image according to an exemplary embodiment may be an image captured to include some parts of the vehicle, for example.

According to an embodiment, the vehicle data may include a plurality of vehicle images.

The plurality of vehicle images may include one or more images captured at different viewpoints. For example, the plurality of vehicle images may include images captured at different viewpoints with respect to the same vehicle. The plurality of vehicle images may include images captured in the same direction (eg, from the front) with respect to the same vehicle.

3 is a diagram for illustrating image data according to an embodiment. Referring to FIG. 3 , the image data includes a first image (a) obtained by photographing a vehicle at a first time point, a second image (b) obtained by photographing a vehicle at a second time point after the first time point, and a vehicle at a third time point. The photographed third image (c) may be included. The first image, the second image, and the third image may be images extracted from one image data. The first image, the second image, and the third image may be images taken in the same direction, for example, from the left side with respect to the same vehicle.

The plurality of vehicle images may include one or more images obtained by photographing the vehicle in various directions.

4 is a diagram illustrating image data according to an embodiment. Referring to FIG. 4 , the image data includes a left image obtained by photographing the left side of the vehicle (a), a right image obtained by photographing the right side of the vehicle (b), and a front image obtained by photographing the front of the vehicle (c) And it may include a rear image (d) obtained by photographing the rear of the vehicle.

5 is a diagram illustrating image data according to another embodiment. 5 , the image data is a left front image (a) obtained by photographing the left front of the vehicle, a left and rear image (b) obtained by photographing the left and rear of the vehicle, and the right front image obtained by photographing the vehicle It may include a right-front image (c) and a right-rear image (d) obtained by photographing the right-rear image of the vehicle.

The plurality of vehicle images may include one or more images captured to include different parts of the vehicle.

6 is a diagram illustrating image data according to an embodiment. Referring to FIG. 6 , the image data includes a first image (a) obtained by photographing to include the left front fender of the vehicle, a second image (b) obtained by photographing to include the left front door of the vehicle, and the left side of the vehicle. A third image (c) obtained by photographing to include the rear door and a fourth image (d) obtained by photographing to include the left rear fender of the vehicle may be included.

According to an embodiment, image data obtained by photographing an object other than a vehicle may be obtained. For example, an object image obtained by photographing an object around the vehicle, an image photographing the environment around the vehicle, for example, the sky, an image photographing the driver's identification card or driver's license may be acquired.

Image data or image data may be acquired through a camera module. The image data or image data may be obtained from an external device, for example, an external device having a camera module.

2.2.2 Acquisition of Non-Image Data

According to an embodiment, vehicle data in the form of a non-image other than an image may be obtained. The vehicle data may be non-image data. The vehicle data may be non-image data including information related to the vehicle.

The vehicle data may include vehicle information indicating a manufacturer, model name, year, manufacturing time, manufacturing location, manufacturer, location, registration base, and the like of the vehicle.

The vehicle data may include vehicle model information indicating the type of vehicle. The vehicle type information may indicate a vehicle type of a vehicle, such as a sedan, a coupe, a wagon, a convertible, a hatchback, an SUV, a van, a truck, and a bus.

The vehicle data may include vehicle identification information for identifying the vehicle. The vehicle identification information may include a vehicle number assigned to the vehicle, a vehicle identification number (VIN), and the like.

According to an embodiment, additional information in the form of a non-image may be further obtained. For example, additional information including driver information indicating personal information of the driver or vehicle owner, insurance information indicating whether the vehicle is insured, insurance type, etc., and an accident history of the vehicle may be further acquired.

The non-image data may be pre-stored or obtained. The non-image data may be obtained through an input module that obtains a user input. Non-image data may be obtained through a user interface. Non-image data may be obtained from an external device.

2.2.3 Data processing

Processing may be performed on the obtained data.

Pre-processing (or post-processing) may be performed on the acquired image data. For example, with respect to the acquired image (eg, vehicle image), brightness change, saturation change, color change, color temperature change, luminance change, size change, image cropping, number of pixels adjustment, size change, horizontal/vertical ratio adjustment, pixels Pixel linearization, filtering, dead pixel correction, image sharpening, geometric distortion correction, chroma suppression, image scaling, dithering ( dithering), interpolation, compression (compressing), decompressing (decompressing), etc. may be performed.

As an example, noise removal processing may be performed on the acquired image data. For the vehicle image, noise removal processing may be performed. Noise removal processing for removing noises such as reflected light and stains located in the background area or vehicle area included in the image, ie, image information unnecessary for vehicle management, may be performed. The image may be subjected to processing such as a Gaussian filter, and noise removal processing may be performed. Alternatively, noise removal processing may be performed on the image through a neural network model trained for noise removal.

7 is a diagram for explaining noise removal processing.

According to an embodiment, noise removal processing for removing non-vehicle objects from the vehicle image may be performed. Referring to FIGS. 7A and 7B , the noise removal processing according to an embodiment acquires an original image (a) including a vehicle area corresponding to a vehicle and a background area that is a non-vehicle area, and , it may include acquiring a pre-processed image (b) from which the background area is removed.

The denoising processing may be performed using a neural network model trained to obtain the vehicle region from the original image. For example, the neural network model includes a convolutional neural network (CNN) layer, and based on training data including a plurality of images in which the vehicle region is masked in the original image, it can be learned to extract the vehicle region from the original image. there is. A noise removal process of extracting a vehicle region from the target original image and removing the remaining regions may be performed using the learned neural network model.

According to another exemplary embodiment, noise removal processing may be performed to remove an element that obstructs acquisition of vehicle information from a vehicle image. The noise removal processing for removing a factor that obstructs acquisition of vehicle information may include removing noise included in a vehicle area corresponding to the vehicle. Referring to FIGS. 7C and 7D , the noise removal process according to an embodiment acquires a vehicle image for acquiring vehicle damage information, and corresponds to reflected light independent of damage to the vehicle from the vehicle area. It may include removing the reflected light region.

The noise removal processing may be performed using a neural network model trained to obtain an image from which reflected light has been removed from the original image. For example, a neural network model includes a convolutional neural network layer, and based on training data (or an original image including a specular region and an image in which the specular region is manually removed) including a plurality of images in which the specular region is masked in the original image. , can be learned to obtain an image in which the specular region is removed from the original image. Using the learned neural network model, a noise removal process for obtaining an image in which the reflected light region is removed from the target original image may be performed.

As another example, noise removal processing may be performed on image data including a plurality of images. When the image data includes a plurality of images, noise removal processing with improved accuracy may be performed by performing noise removal processing on each image or using one or more images together.

Based on the plurality of original images, noise removal processing for obtaining an image from which the background region is removed may be performed. The noise removal process acquires a plurality of images, classifies an object whose position changes (eg, an object corresponding to a vehicle) and an object whose position does not change (eg, an object corresponding to a background), and an object whose position changes Alternatively, it may include removing an object whose position does not change. The positional change may mean a relative positional change within the image frame. For example, the noise removal processing may include acquiring a plurality of images, removing an object whose position does not change, and obtaining an image including only an object whose position does not change. Also, for example, the noise removal processing may include acquiring a plurality of images, removing an object whose position changes, and acquiring an image including only an object whose position does not change. Here, an object whose position changes may be defined as a variable region, and an object whose position does not change may be defined as a fixed region.

8 is a diagram for illustrating noise removal processing using a plurality of images. Referring to FIG. 8 , a plurality of original images (a, b, and c) having a vehicle region and a background region are acquired, and an image d from which a background region is removed based on the plurality of original images (a, b and c). ) may be subjected to noise removal processing. Referring to FIG. 8 , based on the plurality of original images (a, b, and c), a region (ie, a background region) corresponding to a background, which is an object having a fixed position in the image, is removed, and An image d including a region (ie, a vehicle region) corresponding to a vehicle, which is an object whose position is changed, may be obtained.

The image d from which the background is removed may be generated based on any one of the plurality of original images a, b, and c. For example, the denoising process obtains a background area and a vehicle area based on the plurality of original images (a, b and c), and removes the background area from any one of the plurality of original images (a, b and c), It may include acquiring a denoised image (d).

Here, noise removal processing may be performed based on at least two of the plurality of original images a, b, and c. The image on which the noise removal process is performed may be defined as a first image and a second image.

For example, (a) of FIG. 8 may be defined as a first image, and (b) of FIG. 8 may be defined as a second image. The first image and the second image may be images taken at different time points. The first image may be an image captured at a first time point, and the second image may be an image captured at a second time point. The first time point and the second time point may be different time points.

The noise removal processing may be performed using a neural network model. The neural network model may include a convolutional neural network layer, and may be trained to obtain an image from which the background region is removed by using training data including a plurality of vehicle images in which the vehicle region and/or the background region are masked. The noise removal processing may include acquiring a vehicle image from which a background region is removed from the target original images by using the learned neural network model.

Based on the plurality of original images, noise removal processing for removing reflected light or the like may be performed. The noise removal processing acquires a plurality of images, and distinguishes an object whose position changes (eg, a reflective region corresponding to reflected light) and an object whose position does not change (eg, a vehicle region or a damaged region corresponding to damage to the vehicle) Thus, it may include removing an object whose position is changed or an object whose position does not change. For example, based on a vehicle image including a damaged area corresponding to a damaged part of the vehicle and a reflection area corresponding to light reflected by the surface of the vehicle, noise removal processing to obtain a vehicle image from which the reflection area is removed may be performed. . Here, an object whose position changes may be defined as a variable region, and an object whose position does not change may be defined as a fixed region. That is, the damaged region may be defined as a fixed region, and the reflective region may be defined as a variable region.

9 is a diagram for illustrating noise removal processing using a plurality of images. Referring to FIG. 9 , a noise removal process of acquiring a plurality of original images (a and b) including a damaged region and a reflective region, and obtaining an image (c) from which the reflective region is removed may be performed. Referring to FIG. 9 , based on a plurality of original images (a and b), noise removal processing is performed to obtain an image (c) from which a reflective region whose position or state is changed within the image or within the vehicle region is removed. can be

The image (c) from which the reflection region is removed may be generated based on any one of the plurality of original images (a and b). For example, the noise removal process obtains a reflection area based on a plurality of original images (a and b), and removes the reflection area from any one of the plurality of original images (a and b) to obtain a denoised image may include

Here, noise removal processing may be performed based on at least two of the plurality of original images a, b, and c. The image on which the noise removal process is performed may be defined as a first image and a second image.

For example, FIG. 9A may be defined as a first image, and FIG. 9B may be defined as a second image. The first image and the second image may be images taken at different time points. The first image may be an image captured at a first time point, and the second image may be an image captured at a second time point. The first time point and the second time point may be different time points.

The noise removal processing may be performed based on a difference between the positions of the fixed region and the variable region in the first image and the positions of the fixed region and the variable region in the second image.

In the noise removal process, the fixed region and the variable region are classified according to the difference in distance between the abnormal region included in the first image and the second image, and the variable region is removed from the classified fixed region and the variable region to create a damaged image. can be processed by The damaged image may be an image in which a variable region is removed from an image in which the vehicle is located close to a central region of the first image and the second image.

For example, the first image includes a plurality of abnormal regions including a first abnormal region and a second abnormal region, and the second image includes a plurality of abnormal regions including a third abnormal region and a fourth abnormal region including, wherein the first abnormal region and the third abnormal region correspond, and the second abnormal region and the fourth abnormal region correspond to each other.

The noise removal processing may classify the fixed region and the variable region based on the position of each abnormal region in the first image and the position of each abnormal region in the second image.

The position of the abnormal region may be a relative position defined based on a partial region of the vehicle. That is, when any one of a plurality of feature points of the vehicle included in the first image and the second image is defined as the first feature point, it may be defined based on the first feature point located in the abnormal region.

The first feature point may have different coordinates in each image.

For example, a first feature point in the first image may have first coordinates, and the first feature point in the second image may have second coordinates.

Since the abnormal region is not a single point but a region having an area, coordinates representing each region must be defined for comparison with the first feature point. The noise removal process can define coordinates representing each area. The noise removal process may be defined as a coordinate representing any one of a plurality of coordinates included in each region. For example, in the noise removal process, the coordinates of the center of each region may be defined as coordinates representing the corresponding region.

Coordinates representing the first abnormal region of the first image may be defined as third coordinates, and coordinates representing the second abnormal region may be defined as fourth coordinates.

Also, in the second image, the coordinates representing the third abnormal region may be defined as fifth coordinates, and the coordinates representing the fourth abnormal region may be defined as the sixth coordinates.

A first distance between the first coordinates and the third coordinates and a second distance between the second coordinates and the fifth coordinates may correspond to each other. A third distance between the first coordinates and the fourth coordinates may be different from a fourth distance between the second coordinates and the sixth coordinates.

A difference between the first distance and the second distance may be smaller than a difference between the third distance and the fourth distance.

The noise removal process may classify whether the abnormal region is a fixed region or a variable region based on a difference value between the first distance and the second distance and a difference value between the third distance and the fourth distance.

In the fixed region, relative coordinates positioned in the vehicle may not change, and in the variable region, state coordinates positioned in the vehicle may change.

The noise removal process may be classified as a fixed region when the difference between the first distance and the second distance is less than a predetermined value, and when the difference between the third distance and the fourth distance is greater than or equal to a predetermined value It can be classified as a variable region.

That is, the first abnormal region and the third abnormal region may be classified as a fixed region, and the second abnormal region and the fourth abnormal region may be classified as a variable region.

The noise removal processing may be performed using a neural network model. The neural network model may include a convolutional neural network layer, and may be trained to obtain an image in which the reflective region is removed by using training data including a plurality of vehicle images in which the reflective region is masked. The noise removal processing may include acquiring a vehicle image in which a reflection region is removed from the target original images by using the learned neural network model.

Meanwhile, in the above embodiments, a case in which an image subjected to noise removal is obtained based on three or two original images has been described as a reference. It may be performed based on

Pre-processing may be performed on the acquired non-image data.

For example, the obtained non-image data may be transformed into a form that is easy to obtain vehicle information. Non-image data can be quantified. Non-image data may be converted into binary data form. Based on the plurality of non-image data, a score for obtaining vehicle information may be calculated.

2.3 Acquisition of vehicle information

The vehicle management process may include obtaining vehicle information based on vehicle data.

The vehicle information may include parts information related to parts constituting the target vehicle, damage information related to damage inside/outside the target vehicle, vehicle model information related to the type of the target vehicle, and/or identification information for identifying the target vehicle. .

Vehicle information may be extracted from vehicle data or obtained by processing vehicle data. Vehicle information may be obtained using vehicle data as input data, an algorithm designed to obtain vehicle information, or a neural network model trained to obtain vehicle information.

In order to obtain vehicle information, a classifier algorithm that classifies or predicts an input image with respect to one or more labels may be used. For classification or prediction, various types of algorithms may be used. For example, k-nearest neighbor, support vector machine, artificial neural network, decision tree, self-organizing map, logical Logistic regression or the like may be used.

The artificial neural network may be a classifier, hybrid classifiers, ensemble classifiers, a linear regression neural network, or the like. An artificial neural network may be a supervised learning, unsupervised learning, or reinforcement learning model.

10 is a diagram for explaining a neural network model for acquiring vehicle information according to an embodiment.

Referring to FIG. 10 , a neural network model according to an embodiment may be provided in the form of a classifier. According to an embodiment, the neural network model includes an input layer (IL), a pooling layer (PL), a convolutional neural network layer (CL), a full-connection layer (FCL), a hidden layer (HL), an output layer (OL), and the like. , a feature vector may be obtained based on the input image. The neural network model may be prepared in the form of a classifier that classifies input images into one or more labels. Alternatively, the neural network model may be prepared in the form of a regression model. The neural network model may be prepared as a regression model that obtains a linear output value for specific vehicle information based on an image as an input.

The neural network model may acquire output information by taking a vehicle image obtained by photographing a part of the vehicle as an input. The output information may indicate whether the input image includes the target object. For example, the output layer of the neural network model may include an output node to which a probability function is assigned. The output layer of the neural network model may include an output node to which a probability function indicating whether the target image includes the target object is assigned. The output layer may include, with respect to one or more target objects, an output node to which one or more probability functions indicating whether an input image includes each target object are assigned.

In order to obtain vehicle information, a neural network model that obtains a segmentation map (or image) including segmentation of one or more objects included in the image may be used, based on an input image. In order to obtain the segmentation map, various types of neural networks may be used. For example, a Fully Convolutional Network (FCN), SegNet, UNet, UNet++, DeConvNet, etc. may be used.

11 is a diagram for explaining a neural network model for acquiring vehicle information according to an embodiment. Referring to FIG. 11 , a neural network model according to an embodiment may be an image segmentation neural network model. The neural network model consists of an input layer (IL), a pooling layer (PL) that performs downsampling, a convolutional neural network layer (CL) that acquires a feature map, a deconvolutional neural network layer (DCL) that acquires a segmentation map, and upsampling. It may include an unpooling layer (UPL), an output layer (OL), and the like.

The pooling layer may perform maximum value pooling, minimum value pooling, average value pooling, and the like. The unpooling layer may correspond to the pooling layer. The deconvolution layer may generate an intermediate segmentation map. The unpooling layer may upsample the intermediate segmentation map. The unpooling layer may perform maximum value unpooling, nearest neighbor unpooling, and the like.

In order to obtain vehicle information, a neural network model that detects one or more objects from an input image may be used. For detection of an object, various types of neural network models may be used. It can be performed using YOLO (You Only Look Once), R-CNN, fast-RNN, faster R-CNN, SAS deep learning, SPPnet, etc.

12 is a diagram for explaining a neural network model for acquiring vehicle information according to an embodiment. According to an embodiment, a fast-RNN model may be used to obtain part information. For example, referring to FIG. 12 , a neural network model according to an embodiment includes an input layer (OL), a convolutional neural network layer (CNL), a feature map (FM), a region proposal neural network (RPN), and a region of interest (ROI). ; Reason of Interest) It may include a deep learning neural network (DNN) including a bounding box (RBB), an ROI pooling layer (RP), and an all-connection layer.

The region proposal neural network may obtain one or more ROIs (or object proposals) or a bounding box of the ROIs based on the feature map of the image. A region proposal neural network may obtain a score for an individual ROI (or object proposal) or its bounding box. The score may indicate a degree to which the target ROI matches the detection target object.

The ROI pooling layer may perform pooling on individual ROIs. The ROI pooling layer may perform pooling on a feature map corresponding to an individual ROI. The ROI pooling layer may perform size adjustment on individual ROIs. The ROI pooling layer may acquire a feature map of a predetermined size. The fully connected neural network may use a value obtained by flattening the feature map obtained through the ROI pooling layer.

The neural network model may acquire an identifier of an object included in an individual ROI or a bounding box of an individual object through the all-connected neural network layer and the hidden layer and the output layer connected to the all-connected neural network layer.

Meanwhile, the neural network models described above are only examples, and the acquisition of vehicle information described herein is not limited thereto. Acquisition of vehicle information may be performed using an algorithm or neural network model not described herein.

The neural network model may acquire vehicle information by using the additional information as an additional input. The neural network model may acquire vehicle information based on the additional input generated based on the additional information and the image of the vehicle. The additional input may be obtained based on additional information such as the type, model name, manufacturer, year, and/or color of the vehicle. The additional input may be an input image or a feature value obtained based on an additional image different from the input image.

13 is a diagram illustrating a neural network model according to an embodiment. Referring to FIG. 13 , a convolutional neural network and a neural network including a pre-connection layer, a hidden layer, and an output layer (eg, an image-based classifier neural network) may further use additional information to acquire vehicle information. For example, the classifier model includes a convolutional neural network and at least one preconnected layer, and the at least one preconnected layer is an additional input value (AI) obtained based on feature values and additional information obtained through the convolutional neural network. It may include one or more corresponding nodes. The classifier model may include at least one node having an additional input value AI as an input value.

Additional information may be similarly used in the case of the neural network model illustrated in FIG. 11 . For example, the region division neural network model according to an embodiment includes an encoding part and a decoding part, and may acquire vehicle information by further using an additional input generated based on the additional information. A region segmentation neural network model may include one or more nodes that use additional inputs generated based on the additional information. The region segmentation neural network model may include one or more layers that use additional inputs generated based on the additional information as parameters.

Additional information may be similarly used in the case of the neural network model illustrated in FIG. 12 . For example, the neural network model according to an embodiment includes an input layer, a convolutional neural network layer, a feature map, a region proposal neural network, a reason of interest (ROI) layer, an ROI pooling layer, and a full-connection neural network, The fully connected neural network may include a layer including at least one node having an additional input generated based on the additional information as an input value.

According to an embodiment, a vehicle management assistance apparatus for obtaining vehicle information based on a vehicle image and additional information may be provided.

The vehicle management assistance device for acquiring vehicle information related to the exterior of the vehicle based on the vehicle image includes an image acquisition unit that acquires a vehicle image obtained by photographing the exterior of the vehicle, and additional information acquisition that acquires additional information related to characteristics of the vehicle and a processing unit configured to acquire vehicle information using a neural network model trained to acquire vehicle information based on the vehicle and vehicle image.

The neural network model is trained to obtain a first feature set based on the vehicle image and is trained to obtain a second feature set based on the first feature set and additional information and a first part comprising at least one convolutional neural network layer. It may include a second part.

The neural network model may be a classifier or a segmentation model. The neural network model may be prepared in the form of CNN, FCN (Fully Convolutional Network), SegNet, UNet, UNet++, DeConvNet, and the like.

The neural network model may be trained to obtain a feature set based on the vehicle image. The neural network model may be trained to obtain part information or damage information of the vehicle based on the vehicle image. The neural network model may be trained using training data including a vehicle image labeled with part information or a vehicle image labeled with damage information.

The neural network model may be trained to obtain damage information or parts information based on the vehicle image and additional information. The neural network model may be trained based on training data comprising additional information and a vehicle image associated with the additional information and labeled with damage information or part information. The additional information may be image or non-image information. The additional information related to the characteristics of the vehicle may be determined based on at least one of a vehicle manufacturer, color information, a year, and a vehicle model. The additional information related to the characteristics of the vehicle may be obtained based on an image inside the vehicle obtained by photographing the interior of the vehicle.

The processing unit acquires the target vehicle image obtained by photographing the exterior of the target vehicle through the image acquisition unit, obtains target additional information related to the characteristics of the target vehicle through the additional information acquisition unit, and uses a neural network model based on the target vehicle image Thus, target vehicle information for the target may be acquired.

The neural network model can be trained to obtain damage information related to damage outside the vehicle. The first feature set may include at least one feature value indicating a part, type, color, configuration, shape, etc. of the vehicle. The second feature set may include at least one feature value related to whether the vehicle is damaged or not. The second feature set may include a feature value indicating whether or not the vehicle is damaged externally. The second feature set may include one or more feature values indicating a type of external damage to the vehicle. A feature value constituting the feature set may be obtained as a value of a probability function.

The neural network model may be trained to acquire part information related to parts constituting the exterior of the vehicle. The first feature set may include at least one feature value indicating a part, type, color, configuration, shape, etc. of the vehicle. The second feature set may include at least one feature value related to a component constituting the exterior of the vehicle. The second feature set may include one or more feature values indicating types of parts constituting the vehicle. Each feature value may indicate whether or not a specific part is included in the input vehicle image or a probability of including it.

The second part may acquire, based on the first feature set and the additional information, a second feature set related to at least one part constituting the exterior of the vehicle. The second set of features may include a first set of regional features representing regions in which the first part is distributed and a second set of regional features indicating regions in which the second part is distributed.

The second part may obtain, based on the first feature set and the additional information, a second feature set related to at least one damaged area located outside of the target vehicle. The second set of features may include a set of damage area features indicating the location of the damage area.

According to another embodiment, a vehicle management assistance method for acquiring vehicle information based on a vehicle image and additional information may be provided. With respect to the vehicle management assistance method, the above-described information regarding the vehicle management assistance apparatus may be similarly applied unless otherwise specified.

30 is a diagram for describing a vehicle management assistance method according to an exemplary embodiment.

Using a vehicle management assistance device including an image acquisition unit that acquires a vehicle image, an additional information acquisition unit that acquires additional information related to characteristics of the vehicle, and a processing unit that acquires vehicle information based on the vehicle image and the additional information, A vehicle management assistance method for acquiring information related to the exterior of the vehicle based on the image may be provided.

Referring to FIG. 30 , the vehicle management assistance method includes the steps of acquiring a target vehicle image (S3010), acquiring additional target information related to characteristics of the target vehicle (S3030), and a target based on the target vehicle image and target additional information The method may further include obtaining vehicle information (S3050).

The vehicle management assistance method may include acquiring a target vehicle image of the target vehicle through the image acquisition unit ( S3010 ). The target vehicle image may be an image obtained by photographing the vehicle in an arbitrary direction. Alternatively, the target vehicle image may be an image photographed from the front, side, or rear of the vehicle.

The vehicle management assistance method may include acquiring additional target information related to the characteristics of the target vehicle through the additional information obtaining unit ( S3030 ).

The vehicle management assistance method may include, by the processor, obtaining target vehicle information on the target vehicle based on the target additional information and the target vehicle image ( S3050 ).

The step of obtaining vehicle information on the target vehicle ( S3050 ) includes, on the basis of the target vehicle image, using a neural network model trained to obtain vehicle information based on the vehicle image to obtain target vehicle information on the target vehicle. may include more.

The neural network model is trained to obtain a first feature set based on the vehicle image and is trained to obtain a second feature set based on the first feature set and additional information and a first part comprising at least one convolutional neural network layer. It may include a second part.

The neural network model is trained to acquire damage information related to damage to the exterior of the vehicle, and the second feature set may include at least one feature value related to whether or not the vehicle is damaged outside.

The neural network model is trained to obtain part information related to parts constituting the exterior of the vehicle, and the second feature set may include at least one feature value related to parts constituting the exterior of the vehicle.

The additional information related to the characteristics of the vehicle may be determined based on at least one of a vehicle manufacturer, color information, a year, and a vehicle model.

The additional information related to the characteristics of the vehicle may be obtained based on an image inside the vehicle obtained by photographing the interior of the vehicle.

The second part may acquire a second feature set related to at least one part constituting the exterior of the target vehicle based on the first feature set and the additional information. The second set of features may include a first set of regional features representing regions in which the first part is distributed and a second set of regional features indicating regions in which the second part is distributed.

The second part may obtain, based on the first feature set and the additional information, a second feature set related to at least one damaged area located outside of the target vehicle. The second set of features may include a set of damage area features indicating the location of the damage area.

The vehicle management assistance method described in this specification may be provided in the form of a computer-readable recording medium storing a program for performing each method.

Meanwhile, the neural network model may be trained using training data. The learning data may be prepared differently according to acquisition target information. The form of specific learning data and learning of the neural network model will be described in more detail below in relation to the acquisition of each information.

Hereinafter, the types and acquisition methods of each vehicle information will be described.

2.3.1 Obtaining part information

Part information related to parts constituting the vehicle may be acquired.

According to an embodiment, the part information may include the number, type, name, and identification information of parts constituting the vehicle.

Part information may be obtained based on non-image data. The part information may be determined by further considering vehicle information such as the type, model name, manufacturer, year, and/or color of the vehicle. The parts information may be obtained from a pre-stored parts information database based on the vehicle information.

The part information may be obtained based on the image data. The parts information may be acquired from a pre-stored parts information database based on vehicle information (eg, type of vehicle, model name, manufacturer, color, etc.) acquired based on image data. The parts information may be acquired from a pre-stored parts information database based on image data and vehicle information acquired using a neural network model.

The parts information may be obtained based on the parts information database stored by matching the number, type, name, identification information, etc. of the parts with classification information such as vehicle model, model name, manufacturer, and year. For example, the parts information database may store the number, type, name, and identification information corresponding to each part constituting a specific year vehicle of a specific model by matching it with the corresponding model year.

According to another exemplary embodiment, the part information is included in the vehicle image and may include part area information corresponding to each part. The part region information may include a boundary of an individual part region included in the vehicle image. The part region information may include a parts region included in the vehicle image and a name or identification number of a part corresponding to each part region.

The part information may include masked part region information by changing the format of regions corresponding to different parts included in the vehicle image. The part information may include part region information masked with different colors for regions corresponding to different parts included in the vehicle image or parts region information labeled with different tags in regions corresponding to different parts.

14 is a diagram for describing part area information according to an exemplary embodiment. Referring to FIG. 14 , the part region information may include part region information corresponding to each part in a vehicle image.

Referring to (a) of FIG. 14 , the part information includes headlight area information HL corresponding to a headlight of the vehicle included in an image photographed from the left front side of the vehicle, and front bumper area information corresponding to the front bumper of the vehicle. (FB), left front wheel area information (LFF) corresponding to the left front fender of the vehicle, left front door area information (LFD) corresponding to the left front door of the vehicle, left rear door area corresponding to the left rear door of the vehicle information (LRD), license plate area information (NP) corresponding to the license plate of the vehicle, left rear fender area information (LRF) corresponding to the rear fender of the vehicle, hood area information (HO) corresponding to the hood of the vehicle, etc. can Each region information may be stored in association with a name or identification information of a part corresponding to each region.

Referring to FIG. 14B , the part region information may include part region information corresponding to each part in a vehicle image captured from the front of the vehicle. Referring to (b) of FIG. 14 , the part information includes headlight area information (LHL, RHL) corresponding to the headlight of the vehicle included in the front image of the vehicle, and front bumper area information (FB) corresponding to the front bumper of the vehicle. ), license plate area information (NP) corresponding to the vehicle's license plate, side mirror area information (LSM, RSM) corresponding to the vehicle's side mirror, windshield area information (FG) corresponding to the vehicle's windshield, vehicle emblem may include emblem area information EM corresponding to , hood area information HO corresponding to the hood of the vehicle, and the like. Each region information may be stored in association with a name or identification information of a part corresponding to each region.

In FIG. 14 , the parts area information has been described based on the images taken from the front left and the front of the vehicle, but this is only an example, and the parts area information may be obtained with respect to the vehicle images taken from various directions.

According to an embodiment, the part information may be obtained using a neural network model. The part information may be obtained using a neural network model that performs object recognition. The part information may be obtained using the neural network model described above with reference to FIGS. 10 to 13 .

The neural network model may acquire part information by taking a vehicle image as an input. The neural network model may acquire the type or name of a part included in the input vehicle image. The neural network model may acquire whether a specific part is included in the input vehicle image.

According to an embodiment, the output layer of the neural network model may include a probability function indicating whether the input image includes a specific part. Also for example, the output layer may include, for one or more target parts, one or more probability functions indicating whether the input image includes each target part.

According to another embodiment, the part information may be obtained using the neural network model described above with reference to FIG. 12 . The part information may include region information that is obtained through a neural network model and indicates a region corresponding to an individual part.

Referring to FIG. 15 , the neural network model according to an embodiment may acquire part information on parts included in the vehicle image by inputting an image (a) of a part of the vehicle as an input. The neural network model may identify a part included in an input image using a neural network model that acquires part information, and may acquire part information corresponding to the identified part. The neural network model may acquire parts information indicating that the left front door is included in the input image using a neural network model that obtains parts information based on the image (a) in which a part of the vehicle is captured.

Referring to FIG. 15 , the neural network model according to an embodiment may obtain part region information corresponding to a part included in the vehicle image by inputting an image (a) of a part of the vehicle as an input. The neural network model may identify a part included in the vehicle image using the neural network model and acquire a region (or segmentation) corresponding to the identified part. Referring to FIG. 16B , the neural network model may acquire region information corresponding to the left front door included in the vehicle image.

Referring to FIG. 16 , the neural network model according to an embodiment may acquire part information on one or more parts included in the vehicle image by inputting an image (a) of one side of the vehicle as an input. The neural network model is a part indicating that the left front door, left rear door, left front fender and left rear fender are included in the input image using a neural network model that is based on the image (a) taken of a part of the vehicle and acquires parts information information can be obtained.

The neural network model may obtain a segmentation map in which regions corresponding to one or more parts are displayed by receiving a vehicle image as an input. For example, the neural network model may obtain a segmentation map in which a first region corresponding to the first part and a second region corresponding to the second part are displayed differently.

Referring to FIG. 16 , the neural network model may obtain part region information (b) corresponding to one or more parts in a part included in the vehicle image by inputting an image (a) of one side of the vehicle as an input. The neural network model may acquire images (b) displayed differently from each other (eg, masked with different colors) for a plurality of regions corresponding to individual parts included in the image. Referring to FIG. 16B , the neural network model may acquire area information corresponding to the front left door, the left rear door, the left front fender, and the left rear fender included in the vehicle image.

According to another embodiment, the part information may be obtained using the neural network model described above with reference to FIGS. 10 to 13 . The part information may include a bounding box that is acquired through a neural network model and displays a region corresponding to an individual part.

The neural network model may acquire an identifier of an object included in an individual ROI or a bounding box of an individual ROI through a hidden layer and an output layer connected to the all-connection layer and the all-connection layer. The neural network model may obtain an ROI bounding box for individual parts included in the vehicle image by taking the vehicle image as an input. The neural network model may obtain a classification result for an individual part region corresponding to an individual ROI included in the vehicle image by receiving a vehicle image as an input.

The neural network model may be trained and prepared to acquire part information. The neural network model may be trained using part learning data including a plurality of vehicle images labeled with part information.

The part learning data may include a plurality of vehicle images. The component learning data may include a plurality of vehicle images photographed in different directions with respect to the vehicle. The component learning data may include a plurality of vehicle images photographed in the same direction with respect to the vehicle. The parts learning data may include a plurality of vehicle images obtained by photographing vehicles of various types, models, or manufacturers. The parts learning data may include a plurality of vehicle images obtained by photographing a vehicle of a single type, model, or manufacturer.

The part learning data may include a vehicle image to which a part label corresponding to one or more parts included in the image is given, respectively. The parts learning data may include vehicle images obtained by photographing various types of vehicles and to which parts labels are assigned. The parts label may be differently assigned to each vehicle type, model, or manufacturer.

The part learning data may include at least one vehicle image masked (or labeled) in different formats for different parts. The vehicle image may include at least one vehicle image that is masked (or labeled) with different colors for different parts or a vehicle image labeled with different tags for regions corresponding to different parts.

The part learning data may include masked vehicle images with respect to various parts. The component learning data may include one or more component areas, for example, information on the left front door corresponding to the left front door and/or the front bumper area information FB corresponding to the front bumper of the vehicle. The component learning data may include one or more pieces of common component area information and may include a plurality of vehicle images photographed in different directions with respect to the vehicle. The part learning data may include one or more vehicle images including at least some different part region information. The parts learning data may include one or more vehicle images masked with each corresponding part information in all parts corresponding regions included in the image.

17 is a diagram illustrating parts learning data according to an embodiment. Referring to FIG. 17 , the component learning data may include a plurality of vehicle images obtained by photographing vehicles of various models and in which a specific component, for example, an area corresponding to a left front door is labeled. The plurality of vehicle images may be obtained by photographing different types of vehicles.

18 is a diagram illustrating parts learning data according to another embodiment. Referring to FIG. 18 , parts learning data is obtained by photographing vehicles of various models, and a plurality of parts, for example, areas corresponding to the left front door, left rear door, left front fender, and left rear fender are labeled differently. It may include a plurality of vehicle images.

In FIGS. 17 and 18, the parts learning data has been described based on a case in which a plurality of vehicle images photographed in the same direction with respect to the vehicle are included, but this is only an example, and the parts learning data may include vehicle images photographed in various directions. can

The neural network model may be supervised, unsupervised, or reinforcement-learned to obtain part information based on a vehicle image by using the part learning data. A neural network model can be trained using a backpropagation method.

The neural network model may be trained to classify a vehicle image according to parts included in the vehicle image by using the component learning data described above. The neural network model may be trained to classify the vehicle image according to the parts included in the vehicle image by using learning data including the vehicle image to which a label corresponding to the part included in the vehicle image is given.

The neural network model may be trained to obtain whether the vehicle image includes a specific part through the part learning data. The neural network model may be trained to obtain whether each part is included in the target image with respect to a plurality of parts through the part learning data.

The neural network model may be trained to acquire region information on a region in which a specific part is distributed in the vehicle image through the part learning data. The neural network model learns to acquire region information on the region in which a specific part is distributed in the vehicle image through the part learning data including the vehicle image in which the type of individual part included in the vehicle image and the region corresponding to the individual part are labeled. can be

The neural network model may be trained to acquire region information on a region in which a specific part is distributed in the vehicle image through the part learning data. The neural network model may be trained to acquire region information on a region in which each part is distributed for each of a plurality of parts included in the vehicle image through the part learning data.

The neural network model may be trained using a plurality of parts learning data. The neural network model is a first part learning data including a vehicle image learned by photographing a vehicle of a first type (or a vehicle of a first manufacturer) and a vehicle of a second type of vehicle (or a vehicle of a second manufacturer) learned by photographing It may be learned using the second part learning data including the vehicle image.

Meanwhile, a plurality of neural network models may be trained and used. The first neural network model may be trained based on the first part learning data, and the second neural network model may be trained based on the second part learning data.

The neural network model may be trained to obtain parts information based on the vehicle image and additional information. The neural network model may be trained to acquire parts information by further using additional information such as vehicle type, model name, manufacturer, and color. The neural network model may be trained to obtain part information based on a first feature set obtained based on the vehicle image and a second feature set obtained based on additional information.

Neural network models can be trained to obtain parts information based on vehicle images and other images. The neural network model may be trained to obtain part information based on a first feature set obtained based on the vehicle image and a second feature set obtained based on another image (eg, a background image obtained by photographing a background). there is.

2.3.2 Obtaining Damage Information

Damage information related to damage to the exterior of the vehicle may be obtained.

The damage information may be information related to damage such as scratches, dents, dents, breaks, loss, cracks, and the like on the exterior of the vehicle. The damage information may include information about the type, location, degree, number, and occurrence time of vehicle exterior damage.

The damage information is included in the vehicle image and may include damage area information corresponding to a location where the damage has occurred. The damaged area information may indicate one or more damaged areas included in the vehicle image. The damaged area information may include a mark for visually distinguishing an area that is included in the vehicle image and corresponds to the damaged area. The damaged area information may include a boundary of the damaged area. The damaged area information may include a bounding box indicating the damaged area. The damaged area information may include coordinate information of the damaged area in the vehicle image. The damage area information may include corner coordinate information in the vehicle image of the bounding box. The damaged area information may include location information or coordinate information of the damaged area with respect to the vehicle. The damaged area information may include absolute position information (eg, coordinate information) of the damaged area based on a specific part of the vehicle (eg, a center line of a license plate, an emblem, etc.).

The damage information may include damage area information corresponding to the damaged area included in the vehicle image, and information related to the damaged area and matching the damage area information (eg, the time of occurrence of damage, type of damage, degree of damage, etc.). there is.

19 is a view for explaining damage information according to an embodiment. Referring to FIG. 19A , the damage information may include information related to damage to the exterior of the vehicle included in the vehicle image. Damage information may be obtained in the form of tags or metadata attached to the vehicle image. Referring to (b) of FIG. 19 , the damage information may include a mark for highlighting the damaged area in the vehicle image. Referring to FIG. 19C , the damage information may include an indication, for example, a heat map, for indicating an area where damage is likely to be located in a vehicle image. Referring to (d) of FIG. 19 , the damage information may include a bounding box for displaying a region where the damage is located in the vehicle image.

Damage information may be obtained based on image data. Damage information may be obtained using a neural network model trained to obtain damage information based on the vehicle image. Damage information may be obtained using the neural network model described above with reference to FIGS. 10 to 13 .

The neural network model may acquire damage information by inputting a vehicle image obtained by photographing a part of the vehicle. The neural network model may acquire whether a damaged part exists in the photographed vehicle based on the input vehicle image. The neural network model may acquire whether a damaged part exists in a specific area of the photographed vehicle based on the input vehicle image. The output layer of the neural network model may include an output node that obtains a probability value indicating whether a damaged region is included in the vehicle image. The neural network model may acquire whether a damaged area is included in the photographed vehicle with respect to one or more types of damage based on the input vehicle image. The output layer of the neural network model may include a plurality of output nodes for obtaining a probability value indicating whether or not damage regions respectively corresponding to a plurality of damage types are included in the vehicle image.

The neural network model may acquire a damaged area based on a vehicle image obtained by photographing a part of the vehicle. The neural network model may detect a region corresponding to external damage to the vehicle from the vehicle image. The neural network model may acquire a bounding box or segmentation for a region included in the vehicle image and corresponding to external damage to the vehicle. The neural network model may acquire segmentation for a plurality of damaged regions. The neural network model may obtain a heatmap. The neural network model may acquire a saliency map in the form of a heat map for indicating an area where damage is located as illustrated in FIG. 19(c) .

Referring to FIG. 20 , the neural network model according to an embodiment may acquire damage area information corresponding to a damaged area by inputting an image (a) of a part of a vehicle taken. Damage region information indicating the damaged area may include a bounding box B1 obtained through a neural network model. Referring to FIG. 20 , the neural network model according to an embodiment may obtain damage area information B2 corresponding to a damaged area by inputting an image (b) of one side of the vehicle as an input. Damage region information indicating the damaged area may include a bounding box obtained through a neural network model. The damage area information may further include a damage type corresponding to the damage area corresponding to each bounding box. The damage type can be obtained by the classifier part of the neural network model.

A neural network model can be trained to obtain impairment information. The neural network model may be trained using damage learning data including a plurality of vehicle images labeled with damage information.

The damage learning data may include a plurality of vehicle images. The damage learning data may include a plurality of vehicle images taken from different directions with respect to the vehicle. The damage learning data may include a plurality of vehicle images taken in the same direction with respect to the vehicle. Damage learning data may include a plurality of damaged vehicle images obtained by photographing damaged vehicles of various types, models, or manufacturers. The damage learning data may include a plurality of damaged vehicle images obtained by photographing a damaged vehicle of a single type, model, or manufacturer.

The damage learning data may include a vehicle image to which a damage label indicating the presence or absence of damage is assigned. Damage learning data may include vehicle images obtained by photographing various types of vehicles and to which damage labels are assigned. The damage label may be provided differently depending on the type of damage (eg, scratch, dent, break, etc.).

The damage learning data may include a masked vehicle image in the damaged area. The damage learning data may include masked (or labeled) vehicle images with respect to one or more damaged areas. The damage learning data may include vehicle images that are masked (or labeled) differently with respect to a plurality of types of damage areas. For example, impairment learning data may include vehicle images displayed in a first color for a first type of damage (eg, scratches) and in a second color for a second type of damage (eg, dents). can do.

21 is a diagram for illustrating impairment learning data. Referring to (a) of FIG. 21 , the damage learning data may include a vehicle image in which the damaged area is masked (M1). Referring to (b) of FIG. 21 , the damage learning data is masked (M2) with a first color with respect to the damage site where the first type of damage (eg, dent) is located, and the second type of damage ( For example, it may include a vehicle image that is masked (M3) with the second color with respect to the damaged area where the scratch is located.

In FIG. 21, the damage learning data has been described based on a case that includes a plurality of vehicle images photographed in the same direction with respect to the vehicle, but this is only an example, and the damage learning data may include vehicle images photographed in various directions. .

The neural network model can be trained using the damage learning data described above. The neural network model may be supervised, unsupervised, or reinforcement trained to obtain damage information based on a vehicle image by using the damage learning data. A neural network model can be trained using a backpropagation method.

The neural network model may be trained to classify a vehicle image according to whether or not it includes a damaged area using damage learning data including a vehicle image labeled as to whether or not the vehicle is damaged. The neural network model may be trained to classify the vehicle image according to the type of damage included in the vehicle image. The neural network model may be trained to obtain whether a target image includes each type of damage with respect to a plurality of injuries through damage learning data including a vehicle image labeled with respect to the presence or absence of multiple types of damage.

The neural network model may be trained to acquire damage region information through damage learning data. The neural network model may be trained to detect a damaged area from a target image using damage learning data including a vehicle image masked to the damaged area for one or more types of damage. The neural network model uses damage learning data including a vehicle image labeled for a plurality of types of damage regions, and for each of a plurality of injuries included in the vehicle image, region information about the area and/or type of each damage is distributed. can be learned to acquire.

The neural network model may be trained using a plurality of damage learning data. The neural network model is trained by photographing a vehicle having a first type of damage (eg, dent) and first damage learning data including a vehicle image learned by photographing a vehicle having a first type of damage (eg, scratch) It may be learned using the second damage learning data including the vehicle image.

Meanwhile, a plurality of neural network models may be trained and used. The first neural network model may be trained based on the first impairment learning data, and the second neural network model may be trained based on the second impairment learning data. The second impairment learning data may be at least partially different from the first impairment learning data.

The neural network model may be trained to obtain damage information based on the vehicle image and additional information. The neural network model may be trained to acquire damage information by further using additional information such as vehicle type, model name, manufacturer, and color. The neural network model may be trained to obtain damage information based on a first feature set obtained based on the vehicle image and a second feature set obtained based on additional information.

Neural network models can be trained to obtain damage information based on vehicle images and other images. The neural network model may be trained to obtain damage information based on a first feature set obtained based on the vehicle image and a second feature set obtained based on another image (eg, a background image taken in the background). there is.

According to one embodiment, the damage information may be used together with the parts information. In this regard, it will be described in more detail below.

2.3.3 Acquisition of vehicle type information

Vehicle type information related to the type of vehicle may be acquired.

The vehicle model information may indicate the vehicle model of a vehicle, such as a sedan, a coupe, a wagon, a convertible, a hatchback, an SUV, a van, a truck, and a bus.

The vehicle model information may be obtained based on the non-image data. The vehicle model information may be obtained based on a vehicle registration number, VIN, model name, and the like. The vehicle model information may be obtained from a pre-stored database.

Vehicle model information may be obtained using image data. The vehicle type information may be obtained using a neural network model trained to obtain vehicle type information based on the vehicle image. The vehicle model information may be obtained using the neural network model described above with reference to FIG. 10 .

The neural network model may acquire vehicle model information based on the vehicle image. The neural network model may classify vehicle images with respect to a plurality of vehicle model labels. For example, the neural network model may include at least one convolutional neural network layer and an output layer having one or more output nodes. The neural network model may include an output node that acquires a feature set related to a vehicle model based on a vehicle image through a convolutional neural network layer, and acquires a probability value for at least one car model label based on the feature set.

The neural network model may be trained to obtain vehicle model information. The neural network model may be trained using vehicle model learning data including a plurality of vehicle images labeled with vehicle model information. The neural network model may be supervised, unsupervised, or reinforcement trained to obtain damage information based on a vehicle image using vehicle model learning data. A neural network model can be trained using a backpropagation method.

The vehicle model learning data may include a plurality of vehicle images. The vehicle model learning data may include a plurality of vehicle images photographed in different directions or the same direction with respect to the vehicle. The vehicle model learning data is obtained by photographing the first vehicle model of the first manufacturer, and includes an image to which the first vehicle model label is given and an image obtained by photographing the first vehicle model of the second manufacturer and to which the first vehicle model label is assigned. can do.

The vehicle model learning data may include a plurality of vehicle images obtained by photographing vehicles of various colors or manufacturers for a plurality of vehicle types, respectively. For example, the vehicle model learning data may include a first image set including a plurality of images obtained by photographing a vehicle belonging to a first vehicle model (eg, SUV) of various manufacturers and to which a first vehicle model label is assigned, and a first image set of various manufacturers. It may include a second image set including a plurality of images obtained by photographing a vehicle belonging to a second vehicle model (eg, a sedan) and to which a second vehicle model label is assigned.

The neural network model may be trained to acquire whether the target image is an image of a specific vehicle model. The neural network model uses car model learning data including a first image to which a first label indicating that the image is related to the first car model is assigned and a second image to which a second label is assigned to represent that the image is not related to the first car model. It may be learned to classify the vehicle image according to whether the first vehicle type is included.

The neural network model may be trained to classify vehicle images according to vehicle types by using vehicle model learning data. The neural network model uses vehicle model learning data including a first image to which a first label indicating that it is an image related to a first vehicle model is attached and a second image to which a second label indicating that it is an image related to a second vehicle model is attached, It can be learned to classify images according to related vehicle types.

A plurality of neural network models may be trained and used. The first neural network model may be trained based on first car model learning data related to the first car model, and the second neural network model may be trained based on second car model learning data related to the second car model. The second car model learning data may be at least partially different from the first car model learning data. The first vehicle model learning data may include a vehicle image labeled whether related to a first vehicle model, and the second vehicle model learning data may include a vehicle image labeled whether related to a second vehicle model different from the first vehicle model.

According to an embodiment, a vehicle photographing guide may be provided based on vehicle model information. In this regard, it will be described in more detail below.

2.3.4 Obtaining Identification Information

Vehicle identification information for identifying the vehicle may be obtained.

The vehicle identification information may be a vehicle registration number, VIN, or the like.

The vehicle identification information may be obtained based on non-image data. Vehicle identification information may be obtained through a user input. The vehicle identification information may be obtained from a pre-stored database.

The vehicle identification information may be obtained based on image data. The vehicle identification information may be obtained based on a vehicle image including an area corresponding to the license plate of the vehicle. The vehicle identification information may be obtained from an image obtained by photographing the front or rear portion of the vehicle. Vehicle identification information may be obtained through an algorithm, program, or neural network model for acquiring vehicle identification information.

The vehicle identification information may be obtained based on the pre-processed image. For example, vehicle identification information may be obtained from an image obtained through preprocessing such as grayscale, boundary emphasis (e.g., Thresholding), contouring, rotation, and the like. The vehicle identification information may be obtained based on a region included in the vehicle image and corresponding to the identification information.

The area corresponding to the identification information may be extracted from the vehicle image using a horizontal/vertical edge 　 method. For example, the region corresponding to the identification information may be extracted by detecting horizontal and vertical edge images in the vehicle image and setting a rectangle appearing as an edge as a candidate region.

The region corresponding to the identification information may be obtained through a change in brightness. For example, in the area corresponding to the identification information, when one line is read in the horizontal direction of the vehicle image, the contrast value in the number or letter part of the license plate has a continuous intensity change vector from negative to positive or from positive to negative. It can be obtained by estimating that there is a license plate area where the contrast change occurs symmetrically, and extending the target area from the estimated coordinates.

The region corresponding to the identification information may be obtained using a template. For example, the region corresponding to the identification information may be obtained using a template for recognizing a character by applying a reference circular set to a vehicle image, that is, matching an image input to a standard pattern.

The region corresponding to the identification information may be obtained using color information.

The region corresponding to the identification information may be obtained using Hough transform. For example, the region corresponding to the identification information may be extracted by detecting an edge from the vehicle image using a Sobel operator and then searching for a group of vertical and horizontal straight lines using Hough transform.

The region corresponding to the identification information may be obtained using run-length. For example, the region corresponding to the identification information obtains a binary image using the gradient in the neighboring direction, then projects it to an arbitrary angle theta, and applies the run-length method to the projected straight line to find a candidate region, It can be obtained by detecting a position where the ratio of two straight lines is 2:1.

Pre-processing may be performed on a region corresponding to the identification information. For example, processing such as rotation for tilt correction, affine transformation, linear regression, and the like may be performed. Alternatively, pre-processing for shading correction may be performed. Alternatively, pre-processing for noise removal may be performed.

Vehicle identification information may be obtained using an algorithm for recognizing a license plate. The vehicle identification information may be obtained using an OCR (Optical Character Recognition) algorithm. Vehicle identification information may be obtained using an OCR algorithm implemented through Tesseract, OpenCV, EAST detector, Python, or the like.

Vehicle identification information may be obtained through a neural network model. The neural network model may acquire vehicle identification information based on the vehicle image. The neural network model may acquire vehicle identification information based on a vehicle front image obtained by photographing a front portion of the vehicle and/or a vehicle rear image obtained by photographing a rear portion of the vehicle. The neural network model is included in the vehicle front image and may obtain vehicle identification information based on the front license plate area corresponding to the license plate of the vehicle or the rear license plate area included in the vehicle rear image and corresponding to the license plate of the vehicle. The neural network model may acquire vehicle identification information based on a VIN region included in a vehicle image and corresponding to a VIN.

The neural network model may include a convolutional neural network. The neural network model may include a Recurrent Neural Network (RNN) and/or Long Short Term Memory (LSTM). The neural network model may acquire a feature set from a vehicle image through a convolutional neural network. The neural network model may acquire vehicle identification information (eg, vehicle number or VIN) corresponding to the vehicle image based on the feature set through a long-term memory or a recurrent neural network.

The neural network model may be trained to obtain vehicle identification information. The neural network model may be trained to acquire vehicle identification information based on the vehicle image by using identification learning data including vehicle identification information or vehicle images labeled with text information constituting vehicle identification information. The neural network model may be supervised, unsupervised, or reinforcement trained to obtain damage information based on a vehicle image, using identification learning data. A neural network model can be trained using a backpropagation method.

The identification information learning data may include a vehicle image photographed in various environments and labeled with vehicle identification information. The identification information learning data may include a vehicle image photographed in an environment such as night, daytime, dawn, rain, fog, snowfall, etc. and assigned an identification information label.

The identification information learning data may include a vehicle image in which a region corresponding to identification information of the vehicle is displayed. The identification information learning data may include one or more images masked in a region corresponding to identification information of the vehicle and to which an identification information label is assigned.

Identification information learning data is obtained by photographing the front of the vehicle and is obtained by photographing the front image of the vehicle and/or the rear of the vehicle to which a label indicating the identification number of the vehicle is given of the vehicle to which the label indicating the identification number of the vehicle is assigned. It may include a back image.

The identification information learning data may include an identification region image of a region corresponding to vehicle identification information. For example, the identification information learning data is obtained by photographing the front license plate of the vehicle and assigned an identification number label, the rear license plate image obtained by photographing the rear license plate of the vehicle and assigned an identification number label, and/or the VIN of the vehicle It may include a VIN image obtained by photographing and to which an identification number (VIN) is assigned.

According to an embodiment, driver identification information for identifying a driver (or owner) of a vehicle may be obtained. The driver identification information may be obtained based on non-image data such as a user input or image data such as an image obtained by photographing the driver's identification card. When driver identification information is obtained based on image data, the aforementioned algorithm, program or neural network model may be similarly used.

The identification information of the vehicle or the driver identification information may be used to manage the history of the vehicle or the driver, respectively. In this regard, it will be described in more detail in the vehicle information management item.

2.3.5 Vehicle Information Acquisition Example 1

According to an embodiment, part-damage information indicating damage for each part may be obtained.

Referring to FIG. 22 , the part-damage information device 220 including the vehicle image acquisition unit 221 , the parts information acquisition unit 223 , the damage information acquisition unit 225 , and the parts-damage information acquisition unit 227 . ) can be obtained by Hereinafter, the acquisition of parts-damage information will be described with reference to FIG. 22 .

The vehicle image acquisition unit 221 may acquire a vehicle image obtained by photographing the exterior of the vehicle. The vehicle image acquisition unit 221 may acquire a vehicle image through a camera module or acquire a vehicle image from an external device.

The parts information acquisition unit 223 may acquire parts information constituting the vehicle based on the vehicle image. The parts information acquisition unit 223 may acquire a type (or identification information) of a part included in the input target vehicle image and part region information corresponding to each part. Regarding the parts information, the contents described in relation to obtaining the parts information may be applied.

The damage information acquisition unit 225 may acquire damage information related to damage to the exterior of the vehicle based on the vehicle image. The damage information obtaining unit 225 may acquire information on whether a damaged area exists in the input target vehicle image and/or damage area information corresponding to the damaged area from the target vehicle image. Regarding the damage information, the contents described in relation to the acquisition of the damage information may be applied.

The part-damage information acquisition unit 227 may acquire part-damage information indicating damage information for each individual part included in the target image based on the part information and the damage information. The part-damage information may include damage area information indicating whether a damaged part exists for each individual part included in the target image, the number of damage and/or the damaged part.

For example, the part-damage information acquisition unit 227 acquires parts information and damage information, and determines whether a damaged region overlapping a part region corresponding to an individual part exists based on the part region information and the damaged region information. can The part-damage information acquisition unit 227 acquires part-damage information indicating that damage exists in the corresponding part (or indicating the number of damage) when there is a damaged area overlapping the part area corresponding to the specific part. can do. The part-damage information obtaining unit 227 may acquire part-damaged area information indicating the location of a damaged part existing in the part when there is a damaged area overlapping the part area corresponding to a specific part.

As a specific example, when the input target vehicle image is related to the first part and the second part, the first part corresponds to the first area of the target vehicle image, and the second part corresponds to the second area of the target vehicle image, The part information may include first part identification information, second part identification information, a first area corresponding to the first part identification information, and a second area corresponding to the second part identification information. When the first part of the target vehicle is damaged, the damage information may indicate that the damaged area is included in the target vehicle image and may include damage area information indicating the location of the damaged area. At this time, the parts-damage information obtaining unit 227 compares the damaged area information with the first and second areas in the target vehicle image, and determines that damage exists in the first area where the damaged areas overlap more. can do. The part-damage information acquisition unit 227 may acquire part-damage information indicating that damage exists in the first part.

23 is a diagram for explaining acquisition of vehicle information according to an exemplary embodiment. Referring to FIG. 23 , the parts information acquisition unit may acquire a segmentation image b including region information corresponding to each part based on the vehicle image a. The damage information obtaining unit may obtain, based on the same vehicle image (a), a bounding box (BB) indicating an area in which the damage is located or an image (c) including the bounding box (BB). The part-damage information acquisition unit may acquire damage information for each part based on the location of the bounding box BB and the segmentation corresponding to each part. Referring to FIG. 23 , the parts-damage information obtaining unit acquires parts-damage information indicating that damage is located in the left front door and left rear door of the vehicle, based on the distribution coordinates and segmentation information of the bounding box BB. can do.

2.3.6 Vehicle Information Acquisition Example 2

According to an embodiment, part-damage information indicating damage for each part may be obtained based on the part information and the damage information.

Referring to FIG. 24 , parts-damage information is a vehicle image acquisition unit 241 , a parts information acquisition unit 243 , a damage information acquisition unit 245 , and a part-damage information acquisition unit 240 including a part 247 . ) can be obtained by The contents described above may be similarly applied to the vehicle image obtaining unit 241 , the parts information obtaining unit 243 , and the damage information obtaining unit 245 unless otherwise specified. Hereinafter, the acquisition of parts-damage information will be described with reference to FIG. 24 .

The part information acquisition unit 243 may acquire a part region (eg, a segmentation or a bounding box of the part region) corresponding to an individual part.

The damage information acquisition unit 245 may include damage information indicating whether or not damage is included in the target part area, the type and number of damage included in the target part area, based on the part area corresponding to the individual part included in the vehicle image, and / Alternatively, damage area information in which damage is located among the target part areas may be acquired. The damage information acquisition unit 245 may acquire damage information for each individual part by using a neural network model trained to acquire damage information for each individual part area.

The part-damage information acquisition unit 247 may acquire part-damage information according to the target image based on damage information related to damage for each individual part. The part-damage information may include damage area information indicating whether a damaged part exists for each individual part included in the target image, the number of damage and/or the damaged part.

As a specific example, when the input target vehicle image is related to the first part and the second part, the first part corresponds to the first area of the target vehicle image, and the second part corresponds to the second area of the target vehicle image, The part information may include first part identification information, second part identification information, a first area corresponding to the first part identification information, and a second area corresponding to the second part identification information. When damage occurs to the first part of the target vehicle, the damage information obtaining unit 245 may include first damage information indicating that the damage is located in the first area and second damage information indicating the location of the damaged area included in the first area. 1 Damage area information can be obtained. The damage information obtaining unit may obtain damage information indicating that damage does not exist in the second part. The part-damage information acquisition unit 247 may acquire part-damage information for the first part and the second part based on the damage information and the damaged area information.

2.3.7 Vehicle Information Acquisition Example 3

According to an embodiment, part-damage information indicating damage for each part may be obtained based on the part information and the damage information.

Referring to FIG. 25 , parts-damage information includes a vehicle image acquisition unit 251 , a vehicle model information acquisition unit 253 , a parts information acquisition unit 255 , a damage information acquisition unit 257 , and a parts-damage information acquisition unit ( 259). The contents described above may be similarly applied to the vehicle image obtaining unit 251 , the parts information obtaining unit 253 , and the damage information obtaining unit 255 unless otherwise specified. Hereinafter, the acquisition of parts-damage information will be described with reference to FIG. 25 .

The vehicle model information acquisition unit 253 may acquire vehicle model information related to the vehicle model of the target vehicle based on the vehicle image. The vehicle model information acquisition unit 253 may be acquired through a user input or a pre-stored database.

The parts information acquisition unit 255 may acquire parts information based on vehicle model information. The parts information acquisition unit may acquire the parts information by considering the vehicle model information and the vehicle image together. The parts information acquisition unit 255 may acquire parts information using a neural network model that uses vehicle model information as an additional input and acquires a feature set based on a vehicle image. The parts information acquisition unit 255 may acquire parts information and/or parts region information by mapping the vehicle image and the parts region information using parts information and/or parts region information prepared in advance for each vehicle type.

In general, since parts constituting a vehicle tend to vary for each vehicle type, accuracy may be improved when a component or a component area is determined in consideration of the vehicle type. Meanwhile, although the description is based on a case in which vehicle model information is obtained based on a vehicle image, the vehicle model information may be obtained from a user input or pre-stored data.

The part-damage information acquisition unit 257 may acquire part-damage information based on the part information and the damage information. With respect to the acquisition of part-damage information, the above-described contents with reference to FIGS. 22 and 23 may be similarly applied.

2.4 Vehicle information management

The vehicle management process may include managing vehicle information. Managing the vehicle information may include managing the aforementioned part information, damage information, vehicle type information, and/or identification information.

Managing the vehicle information may include managing the vehicle information in time series. Managing the vehicle information may include classifying the vehicle information. Managing the vehicle information may include managing one or more pieces of information in association.

Hereinafter, management of vehicle information will be described with reference to some embodiments.

2.4.1 History Management

Managing the vehicle information may include managing the history of the vehicle information. History management of vehicle information may mean listing and/or managing one or more vehicle information acquired at different times in time series.

Managing the vehicle information may include managing the parts information in time series based on one or more parts information. Managing vehicle information may mean managing, for each individual part, part information indicating replacement, defect, damage, etc. of a corresponding part together with time information indicating a time at which each part information was acquired. For example, managing the vehicle information includes acquiring first information indicating that the first part is damaged at a first time point, acquiring second information indicating that the first part has been replaced at a second time point after the first time point, and It may include managing the first information and the second information in chronological order. The first information or the second information may be obtained based on the vehicle image.

Managing the vehicle information may include managing damage information (or parts-damage information) in time series based on one or more pieces of damage information. The vehicle management process may mean managing, with respect to a specific vehicle, damage information indicating damage to the vehicle along with time information indicating a time at which the damage information was acquired. For example, managing the vehicle information acquires first information indicating that the first damage has occurred to the first vehicle at a first time point, and the second damage has occurred to the first vehicle at a second time point that is after the first time point It may include acquiring second information indicating The first information or the second information may be obtained based on the vehicle image. The first information or the second information may include area information corresponding to each damage.

Managing the vehicle information may include time-series management of damage information based on one or more images. For example, managing the vehicle information includes first damage information obtained based on a first vehicle image obtained at a first time point and a second vehicle image obtained based on a second vehicle image obtained at a second time point after the first time point. It may include managing damage information in consideration of time.

Managing vehicle information may be performed in consideration of user information. For example, managing the vehicle information includes first damage information obtained based on the first vehicle image captured by the first user at the first time point and the second vehicle image captured by the first user at the second time point. It may include managing the acquired second damage information based on time in consideration.

Managing vehicle information in consideration of user information will be described in more detail in the following on-site vehicle management process and mobile-based vehicle management process.

2.4.2 Damage history management Example 1 - New damage

History management of damage information may include acquiring new damage information for new damage that has occurred. History management of damage information may include acquiring new damage information based on a plurality of images obtained by photographing the same vehicle at various time points.

The new damage information may be obtained based on a plurality of pieces of damage information. The new damage information may be acquired based on the first damage information corresponding to the first time point and the second damage information corresponding to the second time point after the first time point.

The new damage information may include the number, degree, and location (eg, coordinates) of the newly generated damage.

The new damage information is based on first damage information acquired based on a first image captured at a first time point for the first vehicle and a second image captured at a second time point after the first time point with respect to the first vehicle It may be obtained based on the obtained second damage information. The new damage information may include information on damage newly generated to the first vehicle after the first time point and before the second time point. For example, the new damage information may include the number of new damage acquired based on the number (or degree) of damage according to the first damage information and the number of damage according to the second damage information. The first image and the second image may be images obtained by photographing the vehicle in the same (or similar) direction.

The new damage information may include the location, size, distribution, etc. of a region corresponding to the new damage in the vehicle image.

The new damage information is based on first damage area information obtained based on a first image captured at a first time point for the first vehicle and a second image captured at a second time point after the first time point with respect to the first vehicle and may be obtained based on the obtained second damaged area information. The new damage information is based on the first damage area information and the second damage area information, after the first time point and before the second time point, the location, size, or distribution of the damage area newly generated in the first vehicle in the vehicle image. may contain information.

New damage information may be acquired for each part. The new damage information may include the number, degree, location (eg, coordinates) of the damage newly generated with respect to individual parts, and the location, size, distribution, and the like of a corresponding area. By independently managing the new damage information for each part, it is possible to easily manage the state of each individual part.

The new damage information includes the first damage information for the first part obtained based on the first image captured at the first time point for the first vehicle and the second time point photographed at a second time point after the first time point with respect to the first vehicle. 2 It may be obtained based on the second damage information on the first part obtained based on the image. The new damage information may indicate the number, size, location, etc. of damage newly generated to the first part after the first time point and before the second time point based on the first damage information and the second damage information.

26 is a view for explaining new damage information according to an embodiment. Referring to FIG. 26 , new damage information may be acquired based on a first vehicle image (a) acquired at a first time point and a second vehicle image (b) captured at a second time point.

First damage information indicating that one damage is present in the vehicle may be obtained based on the first vehicle image, and second damage information indicating that two damages are present in the vehicle may be obtained based on the second vehicle image. there is. Based on the first damage information and the second damage information, new damage information indicating that one new damage (B2 of the rear fender) has occurred may be obtained.

First damage information including the first damage area information B1 may be obtained based on the first vehicle image a. Second damage information including the first damage area information B1 and the second damage area information B2 may be obtained based on the second vehicle image b. Based on the first damage information and the second damage information, damage to the rear fender of the vehicle occurring after the first time point and before the second time point may be determined as a new damage. New damage information may include second damage area information corresponding to damage to the rear fender.

2.4.3 Damage history management Example 2 - Damage overlap

There may be cases where the newly generated damage overlaps at least partially with the existing damage. In this case, the number of damages may not be changed even though new damage has occurred. In particular, when the new damaged area is included in the existing damaged area or the existing damaged area is included in the new damaged area, such a problem may occur. Even in this case, a means for notifying the user that new damage has occurred is required.

27 is a view for explaining damage history management when new damage overlapping existing damage occurs. The damage history when a new damage overlaps with the existing damage occurs may be obtained using the location, size, coordinates, etc. of the damaged area.

Referring to FIG. 27 , managing the damage history includes acquiring the first vehicle image (a) photographed at the first time point and including the first damage area information (B1) based on the first vehicle image (a) and obtaining first damage information. Managing the damage history includes acquiring a second vehicle image (b) photographed at a second time point that is after the first time point and including second damage area information (B2) based on the second vehicle image (b) It may include obtaining damage information.

Referring to FIG. 27 , the second damaged area information B2 may overlap with the first damaged area information B1 . The first damaged area information B1 may be included in the second damaged area information B1. When the first damaged area information B1 at least partially overlaps with the second damaged area information B1, the number of damages according to the first damage information and the number of damages according to the second damage information may be the same.

The new damage information may be obtained based on a difference between the second damaged area information B2 and the first damaged area information B1. The new damage information may be obtained based on a difference between the size of the second damaged area (the second bounding box) and the size of the first damaged area (the first bounding box) according to the second damaged area information B2 . New damage information is to be obtained based on the difference between the position (eg, coordinates of the central point) of the second damaged region (second bounding box) and the position (eg, coordinates of the central point) of the first damaged region (first bounding box) can The new damage information may be obtained based on a difference between the coordinates of the corners of the second damaged area (the second bounding box) and the coordinates of the corners of the first damaged area (the second bounding box).

2.5 Vehicle information output and vehicle information output interface

The vehicle management system may output vehicle information. The vehicle information may be output through a user terminal having a display, a manager device, or the like.

2.5.1 Output information

The vehicle management system may output the obtained vehicle information and/or the obtained auxiliary information based on the vehicle information.

The vehicle management system may output vehicle information such as parts information, damage information, vehicle type information, and identification information. The vehicle management system may output user information and/or location information together.

The vehicle management system may output a vehicle image and vehicle information corresponding to the vehicle image. The vehicle management system may output a plurality of vehicle images and vehicle information corresponding to each vehicle image. The plurality of vehicle images may be images captured at various angles with respect to the vehicle.

The vehicle management system may output the vehicle image and a direction indicator indicating a photographing direction of the vehicle of the corresponding image together. The photographing direction may be obtained based on the vehicle image. The direction in which the vehicle was photographed may be obtained through a user input.

28 is a diagram for explaining an output screen according to an exemplary embodiment.

Referring to FIG. 28 , an output screen according to an exemplary embodiment includes a vehicle image display unit (CI), a direction indicator (DI) indicating a photographing direction of a vehicle image, a vehicle-related non-image information display unit (IF), and vehicle damage information. It may include a damage information table (DT) indicating the.

The vehicle image display unit CI may display an image from which noise is removed. The vehicle image display unit CI may display damaged area information. The damaged area information may include an existing damaged area ED and a new damaged area ND. The existing damaged area ED and the new damaged area ND may be displayed differently. Referring to FIG. 28 , the existing damaged area ED and the new damaged area ND may be displayed with different colors or different types of bounding boxes.

The direction indicator DI may indicate a photographing direction of the vehicle image being displayed. Referring to FIG. 28 , the direction indicator DI may indicate a left direction of the vehicle, and the vehicle image display unit CI may display a vehicle image photographed from the left side of the vehicle. Vehicle images photographed in different directions may correspond to each of the plurality of directions indicated by the direction indicator DI. A corresponding vehicle image may be displayed in response to a user input for selecting a direction of the direction indicator DI.

The non-image information display unit IF may display non-image information related to the vehicle. Referring to FIG. 28 , the non-image information display unit IF may display a time point at which a vehicle image is captured, a vehicle number, a vehicle VIN, a vehicle model, and the like.

The damage information table DT may indicate the number of damage for each part corresponding to the vehicle image. The part corresponding to the image may mean a part corresponding to the direction of the currently displayed vehicle image. For example, in FIG. 28 , since an image taken from the left side of the vehicle is displayed, the damage information table DT may indicate the number of damages for each component with respect to the components located on the left side of the vehicle. The number of damages per component may include the number of existing damages and the number of new damages (separated by +) per component. Referring to FIG. 28 , the damage information table DT may indicate that existing damage is present in the left front door, the left door handle, and the left rear fender, and new damage is present in the left front door and the left rear fender.

Meanwhile, the damage information table DT is only an example of the damage information display unit, and information on existing damage or new damage may be displayed in another form.

According to an embodiment, the vehicle management system may output auxiliary information obtained based on vehicle information. The auxiliary information is obtained based on vehicle information and may be information for assisting vehicle management.

The auxiliary information may be determined according to the degree of damage according to the damage information. For example, the auxiliary information may instruct necessary actions such as repair, painting, replacement of some parts, etc. of the vehicle according to the degree of damage.

The auxiliary information may include repair cost information. The repair cost information may be obtained based on the damage information. The repair cost information may include an estimate of a cost required to properly take a repair action according to the damage to the vehicle based on the damage information.

The auxiliary information may include insurance-related information. The auxiliary information may include insurance claim information determined based on pre-stored insurance data.

2.5.2 Vehicle information output Example 1 - Selection of output image when using multiple images

According to an embodiment, a plurality of vehicle images may be used to obtain vehicle information. For example, as described above in 2.2.3, a plurality of images may be used to remove noise from a vehicle image. In the case of outputting vehicle information obtained using a plurality of images as described above, it will be described which image among the plurality of images is used as a reference to output vehicle information.

When damage information (or parts-damage information) of a vehicle is obtained using a plurality of images, the vehicle image and the obtained damage information, particularly damage area information, may be provided to the user by overlapping. In this case, the vehicle image on which the damaged area information is overlapped may be any one of a plurality of images. The vehicle image on which the damaged area information is overlapped may be an image generated based on a plurality of images.

29 is a view for explaining the display of damaged area information according to an embodiment. Referring to (a), (b) and (c) of FIGS. 29 , a process for removing noise, such as a background, may be performed in the same manner as described with reference to FIG. 7 . Referring to FIG. 29D , a vehicle image in which a bounding box BB corresponding to a damaged portion of the vehicle is overlapped may be provided. The vehicle image on which the bounding box BB overlaps may be a vehicle image from which a background generated based on a plurality of vehicle images (a), (b) and (c) is removed. Alternatively, the vehicle image on which the bounding box BB overlaps may be an image in which the location of the vehicle is close to the center of the frame among the plurality of vehicle images (a), (b) and (c). Alternatively, the vehicle image on which the bounding box BB overlaps may be an image having the highest vehicle display ratio among the plurality of vehicle images (a), (b), and (c). The display ratio of the vehicle may mean a ratio of a region displayed on the image among the entire exterior of the vehicle.

In FIG. 29 , a case in which noise removal processing for removing a background is performed based on a plurality of images has been described as a reference, but this may be similarly applied to a case in which noise removal processing for removing reflected light in a vehicle is performed. In the case of noise removal processing for removing reflected light, damage region information may be superimposed on an image from which reflected light is removed and output.

3. On-site fleet management system

3.1 On-site system

According to an embodiment, a vehicle management system for managing the state of a vehicle using an image captured by an on-site vehicle may be provided.

34 is a diagram for explaining the configuration of an on-site vehicle management system.

Referring to FIG. 34 , the on-site vehicle management system according to an embodiment may include a management device 341 , a camera module 343 , and a detection means 345 .

The management device 341 may be connected to the camera module 343 and the detection means 345 to control the camera module 343 and the detection means 345 . The management device 341 may receive data from the camera module 343 and the detection means 345 . The management device 341 may include at least one of a display, a user interface, a communication module, a memory, and a controller.

The management device 341 may be composed of at least one electronic device. The management device 341 may be implemented in the form of a tablet, a smartphone, and a notebook computer.

Although not shown, the management device 341 may be connected to a server. When the management device 341 is connected to a server, the management device 341 may exchange data with the server. In this case, the server may be connected to the plurality of management devices 341 to receive and process data from the plurality of management devices 341 .

Alternatively, the management device 341 may serve as a server. In this case, the management device 341 may be connected to another management device 341 or other electronic devices to exchange and process data.

The camera module 343 may photograph a vehicle entering or leaving the vehicle. The camera module 343 may transmit photographing data of the vehicle to the management device 341 .

The sensing means 345 may detect the position of the vehicle entering or leaving the vehicle. The sensing means 345 may detect a vehicle entering or exiting. The sensing means 345 may be an optical sensor or a circuit breaker (PB).

The sensing means 345 may transmit a sensed value to the management device 341 , and the management device 341 may extract image data used for vehicle management based on the sensed value. The management device 341 may extract image data from the captured data based on the sensed value.

The on-site fleet management system may be used for fleet fleet management.

An on-site vehicle management system can be used for parking control. More specifically, the on-site vehicle management system may be used for a parking manager to compare and analyze an image of a vehicle entering a parking area and an image of a vehicle leaving the parking area.

In addition, the on-site vehicle management system may be used for taxi management tasks of a taxi company. More specifically, the on-site vehicle management system may be used for a taxi company to compare and analyze vehicle images before and after the driver's taxi operation.

3.2 On-site fleet management process

According to an embodiment, a vehicle management process for managing the state of a vehicle using an image captured on-site may be provided. The on-site vehicle management process may be performed by the above-described on-site vehicle management system.

35 is a diagram for explaining a series of steps performed by the on-site vehicle management process.

Referring to FIG. 35 , the on-site vehicle management process includes obtaining vehicle data photographed on-site related to the vehicle ( S3510 ), obtaining vehicle information based on the obtained vehicle data ( S3530 ), and the vehicle It may include a step of managing information (S3550) and a step of outputting vehicle information (S3570). Hereinafter, each step will be described in more detail.

3.2.1 On-Site Vehicle Data Acquisition

The on-site vehicle management process may include acquiring vehicle data required for vehicle management ( S3510 ).

The vehicle data may be provided in various forms. For example, the vehicle data may be image data or non-image data. The vehicle data may be pre-stored in the device or obtained from outside. For example, the vehicle data may be obtained through the camera module 343 or may be obtained through a user input module.

The vehicle data and the acquisition of the vehicle data will be described in more detail below with reference to specific examples.

3.2.1.1 Image data acquisition

FIG. 36 is a diagram for explaining a step ( S3510 ) of acquiring vehicle data required for vehicle management in an on-site vehicle management process. Hereinafter, the step of acquiring photographed vehicle data ( S3510 ) will be described in more detail with reference to FIG. 36 .

Referring to FIG. 36 , the step of obtaining vehicle data according to an embodiment ( S3510 ) includes obtaining image data including a plurality of images ( S3511 ) and extracting effective image data from among the plurality of obtained images. (S3513) may be included.

According to an embodiment, the vehicle data may be image data including a vehicle image obtained by photographing the vehicle. The vehicle image may be an image including some parts of the vehicle. The vehicle image may be an image obtained by photographing a part of the vehicle to include identification information for identifying the vehicle.

The vehicle data may be vehicle image data obtained by photographing the vehicle. The vehicle data may include image data and image data extracted from the image data.

The image data may refer to data captured by the camera module 343 between a plurality of specific viewpoints. In this case, the plurality of specific time points may mean different time points. Hereinafter, the plurality of specific time points will be described in more detail with specific examples with reference to the drawings.

FIG. 37 is a diagram for explaining that image data is captured by the camera module 343 at a specific point in time when image data is detected by the sensing means 345 according to an embodiment.

Referring to FIG. 37 , the vehicle management system according to an embodiment may include a management device 341 , a detection means 345 , and a camera module 343 . In addition, the vehicle management system may further include a circuit breaker (PB) for controlling the entrance or exit of the vehicle.

The camera module 343 may include at least one camera. The camera module 343 may include a first camera C1 and a second camera C2.

The sensing means 345 may include at least one or more sensors. The sensing means 345 may include a first sensor S1 and a second sensor S2.

The circuit breaker PB may be controlled by the management device 341 . The management device 341 may control the circuit breaker PB when a vehicle entering or leaving the vehicle approaches the circuit breaker PB. The management device 341 may control the circuit breaker PB when a vehicle entering or leaving the vehicle approaches the circuit breaker PB, and entry or exit is permitted. The management device 341 may control the circuit breaker PB according to a sensing value of at least one of the first sensor S1 and the second sensor S2. The management device 341 may control the circuit breaker PB when the vehicle is detected by the first sensor S1.

The camera module 343 may be installed in an area adjacent to a movement path of a vehicle entering or exiting the parking lot in order to photograph the vehicle entering or exiting the parking lot. Hereinafter, the camera module 343 for photographing a vehicle entering the vehicle will be described.

The first camera C1 and the second camera C2 may be installed to photograph the other side of the vehicle. For example, the first camera C1 may be installed at a position capable of photographing the first side of the vehicle, and the second camera C2 may be installed at a position capable of photographing the second side of the vehicle. The first camera C1 and the second camera C2 may be installed to face each other.

The first camera C1 may be installed at a position capable of photographing the side and rear of a vehicle entering or exiting the vehicle. The first camera C1 may be installed such that the second camera C2 is located in a photographing area. The first camera C1 may be installed toward the second camera C2. The first camera C1 may additionally photograph the rear of the entering or exiting vehicle, so that non-image data may be additionally obtained.

The second camera C2 may be installed at a position capable of photographing the side and front of the vehicle entering or leaving the vehicle. The second camera C2 may be installed such that the first camera C1 is located in a photographing area. The second camera C2 may be installed toward the first camera C1. The second camera C2 may additionally photograph the front of the vehicle entering or leaving the vehicle, so that non-image data may be additionally obtained.

The first camera C1 and the second camera C2 may be positioned parallel to the circuit breaker PB.

Alternatively, the first camera C1 and the second camera C2 may be spaced apart from each other. The first camera C1 may be located in an area spaced apart from the blocker PB, and the second camera C2 may be located in an area corresponding to the blocker PB. Since the first camera C1 is located in an area separated from the circuit breaker PB, non-image information of the vehicle can be acquired at a faster time, thereby shortening the processing time. The first sensor S1 may be located in an area spaced apart from the circuit breaker PB, and the second sensor S2 may be located in an area corresponding to the circuit breaker PB. The first sensor S2 may be installed to be spaced apart from the breaker PB to detect a vehicle entering the parking lot, and the second sensor S2 may be installed to detect whether the vehicle is adjacent to the breaker PB. PB) and may be installed to be located in the corresponding area.

The management device 341 may acquire photographing data from the first camera C1 and the second camera C2 . The management device 341 may extract image data from the captured data based on the sensed values received from the first sensor S1 and the second sensor S2 . Here, the image data may include data substantially used for vehicle management.

Alternatively, the management device 341 may include at least one of the first camera C1 and the second camera C2 based on the sensed value transmitted to at least one of the first sensor S1 and the second sensor S2. You can control one. For example, when the entrance of the vehicle is recognized by the first sensor S1, the first camera C1 and the second camera C2 may be controlled to be in a state in which the vehicle can be photographed, and the second When the entrance of the vehicle is completed by the sensor S2 , the first camera C1 and the second camera C2 may be controlled to change to a state in which the vehicle is not photographed.

The management device 341 may define image data as image data as captured data between a time when the vehicle completely passes through the first sensor S1 and a time when the vehicle is first detected by the second sensor S2 . In this case, a time when the vehicle is recognized by the first sensor S1 and the recognition is finished may be defined as a start time, and a time when the vehicle is recognized by the second sensor S2 may be defined as an end time. The management device 341 may define the captured data obtained between the start time and the end time as image data.

The management device 341 defines, as image data, the photographing data between the time when the vehicle completely passes the first sensor S1 and the time when the vehicle is first detected by the second sensor S2 as image data, and uses the image data Thus, the amount of data stored in the storage unit of the management device 341 can be reduced by using it as data for vehicle management, and the data transmitted from the first camera C1 and the second camera C2 to the management device 341 can be reduced. This has the effect of reducing the amount of data.

The first camera C1 may be located in an area corresponding to the first sensor S1 , and the second camera C2 may be located in an area corresponding to the second sensor S2 . By positioning the first camera C1 in an area corresponding to the first sensor S1 and defining a point in time when the vehicle completely passes the first sensor S1 as a starting point, the first camera C1 becomes the vehicle Data obtained by photographing all areas on one side of the image may be defined as image data. In addition, by positioning the second camera C2 in an area corresponding to the second sensor S2 and defining the time when the vehicle is first detected by the second sensor S2 as the end time, the second camera C2 is Data obtained by photographing all areas of the other side of the vehicle may be defined as image data. Accordingly, the image data of the first camera C1 and the second camera C2 are defined as data obtained by photographing all areas on both sides of the vehicle, and accordingly, damage to the vehicle can be recognized more accurately.

38 is a diagram for explaining that image data is captured by the camera module 343 at a specific point in time when image data is sensed by a sensor, according to another embodiment.

Referring to FIG. 38 , the management device 341 according to another embodiment captures image data between a time when the vehicle is first detected by the first sensor S1 and a time when the vehicle is first detected by the second sensor S2 . data can be defined. In this case, a time when the vehicle is recognized by the first sensor S1 may be defined as a start time, and a time when the vehicle is recognized by the second sensor S2 may be defined as an end time. The management device 341 may define the captured data obtained between the start time and the end time as image data.

The management device 341 defines the image data between the time when the vehicle is first detected by the first sensor S1 and the time when the vehicle is first detected by the second sensor S2 as image data, so that the start time point compared to FIG. 37 . can be advanced The management device 341 defines the captured data at an earlier point in time as image data, thereby accurately securing vehicle identification information, such as vehicle number or VIN information located in the front of the vehicle, so that vehicle management can be performed more smoothly. there is

FIG. 39 is a diagram for explaining that image data is captured by the camera module 343 at a specific point in time when image data is sensed by a sensor, according to another embodiment.

Referring to FIG. 39 , the management device 341 according to another embodiment captures the photographing data between the time when the vehicle is first detected by the first sensor S1 and the time when the vehicle completely passes the second sensor S2 . It can be defined as image data. In this case, a time when the vehicle is recognized by the first sensor S1 may be defined as a start time, and a time when the vehicle is recognized and then recognized by the second sensor S2 may be defined as an end time. The management device 341 may define the captured data obtained between the start time and the end time as image data.

37 and 38 by defining the photographing data between the time when the vehicle is first detected by the first sensor S1 and the time when the vehicle completely passes the second sensor S2 as image data. The start time can be earlier and the end time can be delayed. The management device 341 defines the captured data at an earlier time point as image data and defines the captured data at a later time point as image data, thereby managing the vehicle using the image data in which the main part of the vehicle is maximally secured. can be performed, and as a result, there is an effect of minimizing the omission of damage information occurring on the exterior of the vehicle.

Meanwhile, although not shown in the drawings, the vehicle management system according to an embodiment may further include a third sensor, a fourth sensor, a third camera, and a fourth camera.

The above-described first sensor S1 , second sensor S2 , first camera C1 , and second camera C2 may be for securing image data when entering a parking area of a vehicle, and the third sensor , the fourth sensor, the third camera, and the fourth camera may be for securing image data when the vehicle is pulled out of a parking area.

In other words, a third sensor and a third camera may be further included when the vehicle is pulled out, and the third sensor may play a role corresponding to the first sensor S1 when entering the vehicle, and the third camera may be a third camera when entering the vehicle. 1 It may play a role corresponding to the camera C1. In this case, the third camera may be installed to photograph the same side of the vehicle photographed by the first camera C1. In addition, the vehicle may further include a fourth sensor and a fourth camera when entering the vehicle, and the fourth sensor may serve to correspond to the second sensor S2 upon entering the vehicle, and the fourth camera may be a second camera upon entering the vehicle. It may play a role corresponding to the camera C2. In this case, the fourth camera may be installed to photograph the same side of the vehicle photographed by the second camera C2.

The first sensor S1 , the second sensor S2 , the first camera C1 , and the second camera C2 are described above for the third sensor, the fourth sensor, the third camera, and the fourth camera. Among the related contents, contents overlapping with the first sensor S1 , the second sensor S2 , the first camera C1 and the second camera C2 will be omitted.

FIG. 40 is a diagram for explaining that image data is captured by the camera module 343 at a time point within a predetermined range from a time point when the circuit breaker PB operates, according to another embodiment.

The vehicle management system according to another embodiment of FIG. 40 is the same as the embodiment of FIG. 37 except that the circuit breaker PB is used as the sensing means 345 as compared with FIG. 33 . Accordingly, in describing the vehicle management system according to the embodiment of FIG. 40 , the same reference numerals are assigned to parts common to those of FIG. 37 and detailed descriptions thereof will be omitted.

Referring to FIG. 40 , the vehicle management system according to another embodiment may include at least one circuit breaker PB and at least one camera module 343 .

The camera module 343 may include a first camera C1 and a second camera C2. The first camera C1 may be located in an area spaced apart from the blocker PB, and the second camera C2 may be located in an area corresponding to the blocker PB. The first camera C1 and the second camera C2 may be installed to be spaced apart, and the first camera C1 and the second camera C2 may photograph the other side of the vehicle.

The management device 341 may acquire photographing data from the first camera C1 and the second camera C2 . The management device 341 may extract image data from among the photographing data obtained from the first camera C1 and the second camera C2 . Here, the image data may include data substantially used for vehicle management.

The image data may mean an image captured between a plurality of specific time points among the captured data captured by the camera module 343 . In this case, the plurality of specific time points may mean a time point within a certain range from a time point at which the circuit breaker PB operates. On the other hand, the time point within a certain range from the time when the circuit breaker PB operates is a time point having a certain range such that at least some of the main parts of the entire exterior of the vehicle can be included in the image data when the vehicle is photographed from the point of time within the predetermined range. can mean That is, when the vehicle approaches the parking area, the circuit breaker PB is activated. The plurality of specific time points are the first time point and a predetermined time point before a certain point based on the time when the circuit breaker PB operates according to the approach of the vehicle. It may mean a second time point which is later.

In other words, the image data may refer to an image captured by the camera module 343 between a first time point before a predetermined time point and a second time point after a predetermined time point based on the time when the circuit breaker PB operates.

The management device 341 may define image data as image data captured by the vehicle between a first time point and a second time point. In this case, it can be defined as the start point before a certain point based on the time when the circuit breaker (PB) operates, and when the vehicle completely passes through the circuit breaker (PB) after the circuit breaker (PB) operates as the end point. can The management device 341 may define the captured data obtained between the start time and the end time as image data.

The management device 341 defines the captured data obtained between the start time and the end time as image data, and uses the image data as data for vehicle management by using the image data as data for vehicle management. It has the effect of enabling the vehicle management function to be performed using only (PB).

In addition, the second time point may be a time point corresponding to a time point at which the circuit breaker PB operates. In this case, there is an effect of reducing the amount of data transmitted from the first camera C1 and the second camera C2 to the management device 341 .

Meanwhile, the camera module 343 according to an exemplary embodiment may be a fixed type in which a photographing time point and an angle are fixed during photographing, but is not limited thereto. As another example, the camera module 343 according to an embodiment may be of a rotation type that rotates according to the movement of the vehicle when photographing.

When the camera module 343 according to an embodiment is rotatably provided, the plurality of specific viewpoints described above may be variable within a range in which the image data may include at least some of the main parts of the exterior of the vehicle.

Hereinafter, a method of extracting image data suitable for an on-site vehicle management process from the image data will be described with reference to the drawings.

The image data extracted from the image data may be any one of vehicle images captured by the camera module 343 at a plurality of specific points in time. Also, the image data extracted from the image data may be any one of vehicle images captured by the camera module 343 between a plurality of specific time points. The image data may be any one of vehicle images captured by the camera module 343 between a start time and an end time.

41 is a diagram for describing a method of extracting image data from image data according to an exemplary embodiment.

As described above, image data acquired by the camera module 343 at a plurality of specific viewpoints may include a plurality of images. In this case, valid image data from among the plurality of images may be extracted so that the on-site vehicle management process can be normally performed.

When the photographing data acquired between the start time and the end time by the management device 341 is defined as image data, the effective image data may be at least one of the image data between the start time and the end time.

The effective image data may be at least one image in an intermediate section of the image data. The intermediate section may be a specific section between the start time and the end time. The intermediate section may include an intermediate point between the start time and the end time. In this case, the time interval between the start time and the intermediate time point and the time interval between the intermediate time point and the end time point may correspond to each other.

The effective image data may refer to at least one image data including a main part of the exterior of the vehicle among a plurality of images included in the image data.

As a result, the management device 341 can reduce the amount of data stored in the storage unit of the management device 341 by using the effective image data as data for vehicle management, and effectively reduce the main part of the vehicle exterior. It has a manageable effect.

The image data acquired by the management device 341 is 41 (a) to (c) may be a plurality of images including the images. The management device 341 can effectively manage the main parts of the vehicle exterior by extracting the (b) image showing the main parts as effective image data.

Meanwhile, the effective image data may be an image including only a partial area of the vehicle. Even in this case, the effective image data may be in a form including a major part of the exterior of the vehicle, thereby having an effect of more clearly determining the exterior of the vehicle.

3.2.1.2 Acquisition of non-image data

According to an embodiment, vehicle data in the form of a non-image other than an image may be obtained. The vehicle data may be non-image data. The vehicle data may be non-image data including information related to the vehicle. The non-image data may be previously stored or obtained. The non-image data may be acquired by the camera module 343 . The non-image data may be obtained based on photographing data captured by the camera module 343 . The management device 341 may acquire non-image data based on the image data.

3.2.1.3 Data processing

In the on-site vehicle management process, processing may be performed on the acquired data.

In the on-site vehicle management process, pre-processing (or post-processing) may be performed on the acquired image data.

The management device 341 may pre-process at least one or more valid image data among the image data.

The management device 341 may perform noise removal processing based on a plurality of images in an intermediate section between a start time and an end time. The management device 341 may perform noise removal processing based on a plurality of valid image data.

42 is a diagram for illustrating noise removal processing using a plurality of images. Referring to FIG. 42 , the management device 341 obtains a plurality of valid image data (a) to (c) having a vehicle region and a background region, and based on the plurality of valid image data (a) to (c), A noise removal process may be performed to obtain the image d from which the background region is removed. The management device 341 removes an area (ie, a background area) corresponding to the background, which is an object with a fixed position in the effective image data, based on the plurality of valid image data (a) to (c), An image d including a region (ie, a vehicle region) corresponding to a vehicle, which is an object whose position is changed in the valid image data, may be obtained.

Illustratively, in the case of a parking control company or a taxi company, the camera module 343 captures a moving vehicle at a fixed position, and thus effective image data may include a moving vehicle image and a fixed background image. In this case, the background image may be repeatedly learned by the neural network model as a non-vehicle image. Accordingly, the noise removal process may include acquiring a vehicle image from which a background region is removed from the valid image data by using a neural network model in which information about the background image is repeatedly learned.

The management device 341 may perform a noise removal process for removing reflected light, etc., based on a plurality of valid image data. The management device 341 acquires a plurality of valid image data, and an object whose position is changed (eg, a reflective region corresponding to reflected light) and an object whose position does not change (eg, a vehicle region or damage corresponding to damage to the vehicle) area) to remove an object whose position is changed or an object whose position does not change. For example, the management device 341 may be configured to obtain a vehicle image from which the reflection area is removed based on effective image data including a damaged area corresponding to a damaged part of the vehicle and a reflection area corresponding to light reflected by the surface of the vehicle. A removal treatment may be performed.

43 is a diagram for illustrating noise removal processing using a plurality of images. Referring to FIG. 43 , the management device 341 obtains a plurality of valid image data (a) and (b) including a damaged region and a reflective region, and removes noise to obtain an image (c) from which the reflective region is removed. processing can be performed. Referring to Figure 43, the management device 341, based on the plurality of valid image data (a) and (b), in the image or in the vehicle area, the position or state is changed in the image (c) is removed. ) can be subjected to noise removal processing to obtain

The image (c) from which the reflection region is removed may be generated based on any one of a plurality of valid image data (a) and (b). For example, the management device 341 obtains a reflection area based on the plurality of effective image data (a) and (b), and removes the reflection area from any one of the plurality of effective image data (a) and (b). , an image from which noise has been removed can be obtained.

The management device 341 may perform noise removal processing using the neural network model. The neural network model may include a convolutional neural network layer, and may be trained to obtain an image in which the reflective region is removed by using training data including a plurality of vehicle images in which the reflective region is masked. The management device 341 may perform a noise removal process for obtaining a vehicle image from which a reflection region is removed from valid image data by using the learned neural network model.

An image on which the above-described noise removal processing has been performed may be defined as a damaged image. The management device 341 may determine vehicle information including damage to the vehicle based on the damage image.

When the camera module 343 is installed in an environment in which noise is not generated, the management device 341 may omit the noise removal process. In this case, vehicle information, which will be described later, may be obtained based on valid image data. In this case, the damaged image may be the valid image data.

3.2.2 Acquisition of on-site vehicle information

The on-site vehicle management process may include obtaining vehicle information based on vehicle data.

The vehicle information may include parts information related to parts constituting the target vehicle, damage information related to damage inside/outside the target vehicle, vehicle model information related to the type of the target vehicle, and/or identification information for identifying the target vehicle. .

Vehicle information may be extracted from vehicle data or obtained by processing vehicle data. Vehicle information may be obtained using vehicle data as input data, an algorithm designed to obtain vehicle information, or a neural network model trained to obtain vehicle information.

Acquiring vehicle information based on vehicle data has been described above in 2.3, and thus redundant information is omitted.

According to an embodiment, the management device 341 may acquire vehicle information based on the damaged image. The management device 341 may acquire damage information based on the damage image.

According to an embodiment, in the case of a parking control company, the management device 341 may acquire parking vehicle information using a pre-stored database. The parked vehicle information may include vehicle type information indicating a vehicle model, a manufacturer, a model name, and a year of the parked vehicle. In addition, the parking vehicle information may further include part information indicating the number, type, name, identification information, etc. of the parts constituting the parking vehicle.

The parking vehicle information may include additional damage information occurring inside/outside the target vehicle. The additional damage information may be calculated based on a damaged image of a vehicle entering and a damaged image of a vehicle leaving the vehicle. The additional damage information may be calculated by comparing a damaged image of a vehicle entering and a damaged image of an exiting vehicle.

The additional damage information may be calculated based on damage information calculated based on the damage image of the vehicle entering and damage information calculated based on the damage image of the vehicle leaving the vehicle.

The management device 341 stores at least one of the damage image and damage information of the vehicle entering, and calculates additional damage information by comparing it with at least any one of the damage image and damage information of the vehicle exiting the vehicle by retrieving it when exiting. can do.

The management device 341 may store non-image data of the vehicle entering the vehicle in association with the damaged image and damage information. For example, the management device 341 may store a damage image and damage information of a vehicle entering the vehicle in association with the vehicle number of the vehicle.

The management device 341 loads the damage image and damage information linked to the vehicle number of the vehicle based on the recognized vehicle number when the vehicle is pulled out, and compares it with the damage image and damage information obtained when the vehicle is pulled out. Additional damage information can be calculated.

As another example, when the parked vehicle uses the parking facility twice or more, the management device 341 may further include additional information including an accident history of the vehicle.

According to another embodiment, in the case of a taxi company, the management device 341 may acquire taxi vehicle information using previously stored data. The taxi vehicle information may include vehicle type information indicating a vehicle type, manufacturer, model name, year, manufacturing time, manufacturing location, manufacturer, location, registration base, and the like of the taxi vehicle. In addition, the taxi vehicle information may further include parts information indicating the number, type, name, identification information, etc. of the parts constituting the taxi vehicle.

The taxi vehicle information may include additional damage information generated inside/outside the target vehicle. The additional damage information may be calculated based on the damage image of the vehicle acquired before the taxi operation and the damage image of the vehicle acquired after the taxi operation. The additional damage information may be calculated by comparing the damage image of the vehicle acquired before the taxi operation with the damage image of the vehicle acquired after the taxi operation.

The additional damage information may be calculated based on damage information calculated on the basis of the damage image of the vehicle obtained before taxi operation and damage information calculated on the basis of the damage image of the vehicle obtained after taxi operation.

The management device 341 stores at least one of the damage image and damage information of the vehicle acquired before the taxi operation, and calls it after the taxi operation, and at least one of the damage image and the damage information of the vehicle acquired after the taxi operation Comparisons can yield additional damage information.

The management device 341 may store the non-image data of the vehicle acquired before driving the taxi in association with the damaged image and damage information. For example, the management device 341 may store the damage image and damage information of the vehicle acquired before driving the taxi in association with the vehicle number of the vehicle.

The management device 341 loads the damage image and damage information linked to the vehicle number of the vehicle based on the recognized vehicle number when the vehicle has completed taxi operation, and the damage image and damage information obtained after taxi operation Comparing with , additional damage information can be calculated.

3.2.3 Vehicle information management

The on-site vehicle management process may include managing vehicle information. Managing the vehicle information may include managing the aforementioned part information, damage information, vehicle type information, and/or identification information.

In addition, managing the vehicle information may include managing the vehicle information in time series. Managing the vehicle information may include classifying the vehicle information. Managing the vehicle information may include managing one or more pieces of information in association.

Hereinafter, management of vehicle information will be described with reference to some embodiments.

3.2.3.1 History Management

Managing the vehicle information may include managing the history of the vehicle information. History management of vehicle information may mean listing and/or managing one or more vehicle information acquired at different times in time series.

According to one embodiment, managing vehicle information in the on-site vehicle management process means managing damage information indicating damage to the vehicle for a specific vehicle together with time information indicating a time at which the damage information was acquired. can For example, managing the vehicle information acquires first information indicating that the first damage has occurred to the first vehicle at a first time point, and the second damage has occurred to the first vehicle at a second time point that is after the first time point It may include acquiring second information indicating

For example, managing the vehicle information in the parking management process may mean managing damage information indicating damage to the vehicle with respect to the parked vehicle along with time information indicating the time at which the damage information was acquired. For example, managing the vehicle information in the parking management process means that when the first parked vehicle enters the parking area, acquiring first information indicating that the first parking vehicle has suffered the first damage, and the first parked vehicle enters the parking area It may include acquiring second information indicating that the second damage has occurred to the first parked vehicle when taking the vehicle out, and managing the first information and the second information in chronological order.

As another example, managing vehicle information in the taxi vehicle management process may mean managing damage information indicating damage to the vehicle for a taxi vehicle together with time information indicating a time at which the damage information was acquired. For example, managing the vehicle information in the taxi vehicle management process means that the first taxi obtains first information indicating that the first damage has occurred in the first taxi before driving, and after the first taxi drives the first taxi, the second in the first taxi It may include acquiring second information indicating that damage has occurred, and managing the first information and the second information in chronological order.

3.2.3.2 Damage history management Example 1 - Management by driver

It is possible to manage the history of damage information in the on-site vehicle management process. History management of damage information may include acquiring new damage information for new damage that has occurred. History management of damage information may include acquiring new damage information based on a plurality of images obtained by photographing the same vehicle at various time points.

The damage information history may be managed for each driver in consideration of driver information.

According to an embodiment, the management device 341 may manage the damage information history of the taxi vehicle for each driver in consideration of driver information. The management device 341 manages the damage information history of the taxi vehicle by driver, so that when damage occurs to the taxi vehicle, it is possible to clarify the location of responsibility, It is easy to determine whether it is the case, and it is possible to provide the effect of reducing the taxi vehicle management cost accordingly. In addition, the management device 341 manages the damage information history of the taxi vehicle for each driver, so that the driver can be evaluated, for example, a safe driving score, etc. It is possible to provide the effect of facilitating the management of drivers within the company by granting a fine or granting a prize in case of no accident for a certain period of time.

Exemplarily, the management device 341 managing the damage information history of the taxi vehicle for each driver in consideration of the driver information means that the first damage information obtained based on the first damage image calculated before the first taxi driver is driven And it may include managing the second damage information obtained based on the second damage image calculated after the driving of the first taxi driver in consideration of time.

As another example, the management device 341 managing the damage information history of the taxi vehicle in consideration of the driver information means that the first taxi driver is the first obtained based on the first damage image calculated before driving the first taxi. The first damage history of the first user obtained by comparing the damage information and the second damage information obtained based on the second damage image calculated after the first taxi driver drives the first taxi, and the first taxi driver 2 The third damage information obtained on the basis of the third damage image calculated before driving the taxi and the fourth damage information obtained on the basis of the fourth damage image calculated after the first taxi driver drives the second taxi It may include managing the second damage history of the first user obtained by comparison in consideration of time.

3.2.3.3 Damage history management Example 2 - Management by vehicle

The damage information history may be managed for each vehicle.

According to one embodiment, the management device 341 may manage the damage information history of the taxi vehicle for each vehicle. Since the management device 341 manages the damage information history of the taxi vehicle for each vehicle, it is possible to provide an effect of easily identifying damage information that has occurred from the past to the present in the taxi vehicle in time series. In addition, the management device 341 manages the damage information history of the taxi vehicle for each vehicle, so that the repair and inspection schedule of the taxi vehicle can be planned according to the condition of the vehicle as a customized schedule for each vehicle, so that efficient vehicle management can be achieved effect can be provided.

Illustratively, the management device 341 managing the damage information history of the taxi vehicle for each vehicle means that the first damage information obtained on the basis of the first image taken before the operation of the first taxi and after the operation of the first taxi It may include managing the second damage information acquired based on the second image taken in consideration of time.

As another example, the management device managing the damage information history of the taxi vehicle for each vehicle means that the first damage information and the first obtained based on the first damage image calculated before the first taxi is driven by the first taxi driver The first damage history of the first taxi obtained by comparing the second damage information obtained based on the second damage image calculated after the taxi is driven by the first taxi driver, and the first taxi by the second taxi driver By comparing the third damage information obtained on the basis of the third damage image calculated before driving and the fourth damage information obtained on the basis of the fourth damage image calculated after the first taxi is driven by the second taxi driver, It may include managing the obtained second damage history of the first taxi in consideration of time.

According to another embodiment, the management device 341 may manage the damage information history of the parked vehicle for each vehicle.

Exemplarily, the management device 341 managing the damage information history of the parked vehicle for each vehicle means that the first damage information and the first parking vehicle are obtained based on the first damage image calculated when the first parked vehicle is entered. It may include managing the second damage information obtained based on the calculated second damage image when taking out the vehicle in consideration of time.

3.2.3.4 Vehicle information management embodiment

According to an embodiment, a vehicle damage management apparatus for acquiring information related to damage to the outside of a vehicle based on a vehicle image may be provided.

The vehicle damage management apparatus includes an image acquisition unit configured to acquire at least one target vehicle image obtained by photographing the exterior of the target vehicle, and damage information related to exterior damage of the target vehicle based on the at least one target vehicle image It may include a processing unit.

The at least one target vehicle image may include a first target vehicle image obtained by photographing one surface of the target vehicle. One surface of the target vehicle may be the front or rear of the vehicle. The at least one target vehicle image may include a vehicle image representing at least one of a front surface, a rear surface, a left surface, a right surface, a left front surface, a left rear surface, a right front surface, a right rear surface, and an upper surface of the vehicle.

The first target vehicle image may include a first area corresponding to the target vehicle and a second area in which identification information of the target vehicle is located. The first area may be larger than the second area. The first area may include a second area.

The processor may obtain vehicle model information of the target vehicle determined based on the second area and the damage information obtained based on the vehicle model information and the first target vehicle image and related to the first area. For example, the damage information may indicate whether damage exists in the vehicle target vehicle included in the first target vehicle image, and the location, degree, size, type, area, and the like of the damage.

The vehicle damage management apparatus may further include an output unit for outputting the damage information. The output unit may output the damage information of the target vehicle. The output unit may output damage information through means such as an image, video, or sound. The output unit may output damage information to an external device through a communication means.

The damage information may include damage area information corresponding to damage to the target vehicle. The output unit may output the target vehicle image and the damaged area information superimposed on the target vehicle image.

The vehicle damage management apparatus may further include an input unit for obtaining a user input. The processing unit, in response to the output of the damage information, may obtain a user response to the damage information through the input unit.

The processor may acquire identification information of the target vehicle determined based on the second area. The processing unit may acquire the vehicle number or VIN of the target vehicle determined based on an area corresponding to the license plate or VIN of the vehicle.

The processing unit obtains identification information of the target vehicle determined based on the second area, obtains a damage history of the target vehicle corresponding to the identification information of the target vehicle,

The damage history may be acquired based on a second target vehicle image captured before the first target vehicle image and may include existing damage information related to the exterior damage of the target vehicle. Existing damage information is damage information acquired on the basis of the second target vehicle image taken before the time when the first target vehicle image is taken, and means damage information, not new damage information to be determined through the first target vehicle image. can

The processing unit may acquire new damage information corresponding to a time point at which the first target vehicle image of the target vehicle is captured, based on the existing damage information and the damage information.

The damage information may include first damage information related to the second area included in the first target vehicle image and second damage information stored in advance in association with identification information of the target vehicle obtained based on the first area. It can be created based on

The at least one target vehicle image may further include a second target vehicle image obtained by photographing another surface different from the one surface of the target vehicle. Here, the second target vehicle image may be an image obtained by photographing an area that does not include identification information of the target vehicle. For example, when the first target vehicle image is a front or rear image obtained by photographing an area including a license plate of the vehicle, the second target vehicle image may be an image obtained by photographing a side surface of the vehicle.

The second target vehicle image may include a third area corresponding to the target vehicle. The processing unit obtains the vehicle model information of the target vehicle determined based on the second area of the first target vehicle image and the damage information obtained based on the vehicle model information and the second target vehicle image and related to the third area can do.

The processing unit may be configured to: based on the first target area image and the vehicle model information, a first part area included in the first area and corresponding to a first part of the target vehicle and a first part area included in the first area and of the target vehicle Part information including a second part area corresponding to the second part may be acquired.

The processing unit may acquire part region information corresponding to the damage based on the damage information and the part information.

The processing unit may obtain the second area corresponding to the license plate of the target vehicle from the first target vehicle image, and obtain the vehicle model information based on the second area corresponding to the license plate.

The processing unit may obtain the damage information based on the target vehicle image by using a neural network model trained to obtain the damage information based on the target vehicle image. The neural network model may be trained to acquire damage information based on the vehicle image by using training data including a vehicle image labeled with a damaged area or damage information. The damage information may include area information indicating an area corresponding to the damage outside the vehicle among the first areas.

According to another exemplary embodiment, there may be provided a vehicle damage management method using an apparatus including an image obtaining unit obtaining a vehicle image and a processing unit obtaining damage information.

31 is a view for explaining a vehicle damage management method according to an embodiment. Referring to FIG. 31 , the vehicle damage management method includes acquiring a target vehicle image ( S3110 ), acquiring vehicle model information of the target vehicle ( S3130 ), and acquiring damage information based on the target vehicle image and vehicle model information (S3150) may be included.

The vehicle damage management method may include acquiring at least one target vehicle image obtained by photographing the exterior of the target vehicle through the image acquisition unit ( S3110 ). The at least one target vehicle image includes a first target vehicle image obtained by photographing one surface of the target vehicle, and the first target vehicle image includes a first area corresponding to the target vehicle and identification information of the target vehicle It may include a second area in which is located.

The vehicle damage management method may include, through the processing unit, acquiring vehicle model information of the target vehicle determined based on the second area ( S3130 ).

The method may include, through the processing unit, acquiring damage information related to the exterior damage of the target vehicle based on the vehicle model information and the first target vehicle image ( S3150 ).

A vehicle damage management method comprising: outputting damage information; may further include. The damage information may include damage area information corresponding to damage to the target vehicle, and outputting the damage information may further include outputting the target vehicle image and the damage area information superimposed on the target vehicle image.

According to an embodiment, a vehicle damage management method for acquiring identification information and acquiring a damage history and/or new damage information of a vehicle based thereon may be provided. 32 is a view for explaining a vehicle damage management method according to an embodiment. Referring to FIG. 32 , the vehicle damage management method includes the steps of obtaining identification information of a target vehicle (S3171), obtaining a damage history of the target vehicle (S3173), and/or obtaining new damage information (S3175) may further include.

The vehicle damage management method may further include, through the processing unit, obtaining identification information of the target vehicle determined based on the second area ( S3171 ).

The vehicle damage management method may further include, through the processing unit, acquiring a damage history of the target vehicle corresponding to the identification information of the target vehicle ( S3173 ). The damage history may be acquired based on a second target vehicle image captured before the first target vehicle image and may include existing damage information related to the exterior damage of the target vehicle.

The vehicle damage management method includes, through the processing unit, based on the existing damage information and the damage information, acquiring new damage information corresponding to a time point at which the first target vehicle image of the target vehicle is captured (S3175) may further include.

The damage information is included in the first area and may include damage area information corresponding to damage to the target vehicle.

The at least one target vehicle image may further include a second target vehicle image obtained by photographing another surface different from the one surface of the target vehicle, and the second target vehicle image may include a third area corresponding to the target vehicle. can

The obtaining of the damage information includes: vehicle model information of the target vehicle determined based on the second area of the first target vehicle image, and obtained based on the vehicle model information and the second target vehicle image, wherein the third area It may further include obtaining the damage information related to.

According to an embodiment, there may be provided a vehicle damage management method for acquiring parts information and acquiring parts area information corresponding to the damage based thereon. The vehicle damage management method may further include obtaining parts information of the target vehicle ( S3191 ) and obtaining parts area information corresponding to the damage ( S3193 ).

The vehicle damage management method includes, through a processing unit, a first part region and the first region included in the first region and corresponding to the first part of the target vehicle, based on the first target region image and the vehicle model information. The method may further include obtaining part information including a second part area included in the target vehicle and corresponding to the second part of the target vehicle ( S3191 ).

The vehicle damage management method may further include, through the processing unit, obtaining part area information corresponding to the damage based on the damage information and the parts information ( S3193 ). Acquiring the part area information corresponding to the damage may include obtaining the name, identification code, number, and the like of the part in which the damaged area is located. Acquiring the part area information corresponding to the damage may include obtaining the part-damage information described herein.

The vehicle damage management method may further include, through the processing unit, obtaining the second area corresponding to the license plate of the target vehicle from the first target vehicle image. The obtaining of the vehicle model information may further include obtaining the vehicle model information based on the second area corresponding to the license plate.

The obtaining of the damage information may further include obtaining the damage information based on the target vehicle image by using a neural network model trained to obtain the damage information based on the target vehicle image.

The vehicle damage management method may further include an input unit for obtaining a user input. In this case, the vehicle damage management method may further include, in response to the processing unit outputting the damage information, obtaining a user response to the damage information through the input unit.

The above-described vehicle damage management method may be provided as a computer-readable recording medium storing a program for performing the method.

3.2.4 Vehicle information output

3.2.4.1 Output information

The on-site vehicle management system may output vehicle information and auxiliary information obtained based on the vehicle information through the display of the management device 341 . Also, the management device 341 may output vehicle information and auxiliary information through an electronic device connected to the management device 341 .

The management device 341 may output vehicle information such as parts information, damage information, vehicle type information, and identification information. The vehicle management system may output user information and/or location information together.

The management device 341 may output a vehicle image and vehicle information corresponding to the vehicle image. The vehicle management system may output a plurality of vehicle images and vehicle information corresponding to each vehicle image. The plurality of vehicle images may be images captured at various angles with respect to the vehicle.

Hereinafter, vehicle information output screens of the parking management system and the taxi vehicle management system will be described with reference to FIGS. 44 to 47 .

3.2.4.2 Vehicle information output Example 1 - Parking control company

44 to 48 are diagrams for explaining a screen for outputting vehicle information and/or auxiliary information in the management device 341 according to an exemplary embodiment.

According to an embodiment, the management device 341 may output the obtained vehicle information and/or auxiliary information obtained based on the vehicle information. The management device 341 may output vehicle information and/or auxiliary information to the information communication device of the manager. Although not shown in the drawings, the above-described vehicle information and/or auxiliary information may be output to an information communication device of a vehicle driver.

The management device 341 may output vehicle information and/or auxiliary information when new damage information is detected in the parked vehicle. In addition, the management device 341 may output vehicle information and/or auxiliary information based on an input by the user interface of the management device 341 when the driver asserts that the parked vehicle is damaged during parking. . In addition, the management device 341 may output vehicle information and/or auxiliary information even when new damage information is not detected or the driver does not claim damage during parking for parking vehicle management.

In addition, when new damage information of an outgoing vehicle is detected, the management device 341 may output that additional damage has occurred to the driver of the vehicle in a state in which the circuit breaker PB is not opened. In this case, the management device 341 can output that additional damage has occurred to the information and communication device of the vehicle driver, and when the vehicle driver confirms the additional damage, the management device 341 is a circuit breaker (PB) may be opened.

Alternatively, the management device 341 may transmit, to the information communication device of the vehicle driver, that additional damage has occurred after opening the circuit breaker PB when new damage information of the outgoing vehicle is detected.

44 is a view for explaining a screen for outputting damage information among the output screens of the management device 341 .

The output screen according to an embodiment includes a vehicle image display unit (CI), a direction indicator (DI) indicating a photographing direction of a vehicle image, a vehicle-related non-image information display unit (IF), and a damage information table indicating vehicle damage information ( DT) may be included.

The vehicle image display unit CI according to an exemplary embodiment may display an image from which noise is removed. The vehicle image display unit CI may display a vehicle image photographed when the vehicle is taken out of the parking area. The vehicle image display unit CI may display a vehicle image including damage area information of the vehicle photographed when the vehicle is taken out of the parking area. The damaged area information may include an existing damaged area ED and a new damaged area ND. The existing damaged area ED and the new damaged area ND may be displayed differently. In addition, the existing damaged area ED and the new damaged area ND may be displayed with different colors or different types of bounding boxes.

In addition, the vehicle image displayed on the vehicle image display unit CI may be output in a form that can be enlarged or reduced by the user.

The direction indicator DI according to an exemplary embodiment may indicate a photographing direction of a vehicle image being displayed. Also, the direction indicator DI may indicate a left direction of the vehicle, and the vehicle image display unit CI may display a vehicle image photographed from the left side of the vehicle. Vehicle images photographed in different directions may correspond to each of the plurality of directions indicated by the direction indicator DI. A corresponding vehicle image may be displayed in response to a user input for selecting a direction of the direction indicator DI.

The non-image information display unit IF according to an embodiment may display non-image information related to the vehicle. Referring to FIG. 44 , the non-image information display unit IF may display a time point at which a vehicle image is captured, a vehicle number, a vehicle model, and the like. For example, the time at which the vehicle image is captured may mean the time of entering the parking area and the time of leaving the parking area. In addition, the non-image information display unit IF may display a vehicle number and a vehicle model name.

The damage information table DT according to an embodiment may indicate the number of damage for each part corresponding to the vehicle image. The part corresponding to the image may mean a part corresponding to the direction of the currently displayed vehicle image. For example, in FIG. 44 , since the image taken from the left side of the vehicle is displayed, the damage information table DT may indicate the number of damages for each component with respect to the components located on the left side of the vehicle. The number of damages per component may include the number of existing damages and the number of new damages (separated by +) per component. Referring to FIG. 44 , the damage information table DT may indicate that existing damage is present in the left front door, the left door handle, and the left rear fender, and new damage is present in the left front door and the left rear fender.

Meanwhile, the damage information table DT is only an example of the damage information display unit, and information on existing damage or new damage may be displayed in another form.

45 is a view for explaining a screen for selecting whether or not the user or the administrator agrees to the damage information among the output screen of the management device 341, Figure 46 is related to the damage information among the output screen of the management device (341) It is a diagram for explaining a screen including detailed information.

Referring to (a) of FIG. 45 , a screen for outputting damage information among the output screens of the management device 341 according to an embodiment may include a detailed confirmation button CO. If the user or manager does not agree with the information on the new damaged area ND or wants to obtain more specific information even if they agree, they can click the detailed confirmation button CO included in the vehicle image display unit CI.

When the user or the manager clicks the detailed confirmation button CO, the output screen of the management device 341 may display the damage image calculated when entering the vehicle and the damage image calculated when exiting the vehicle. In this case, the damaged image calculated at the time of entering the vehicle and the damaged image calculated upon exiting the vehicle may be vehicle images photographed in a direction corresponding to each other. As another example, the damage image calculated when entering the vehicle and the damage image calculated when leaving the vehicle may include at least one main part of the vehicle, and at least one main part of the vehicle may be common.

Alternatively, the output screen of the management device 341 may display an image in which incident value damage information is mapped to valid image data derived when entering a vehicle. Also, the output screen of the management device 341 may display an image in which damage information is mapped when taking out of a vehicle to valid image data derived when taking out a vehicle.

The damaged image calculated upon entering the vehicle may include the existing damaged area ED, and the damaged image calculated upon exiting the vehicle may include the new damaged area ND. That is, when the user or manager clicks the detailed check button, the user or manager can obtain more specific information about the new damage by comparing the vehicle image taken when entering the vehicle and the damage image calculated when leaving the vehicle.

Referring to (b) of FIG. 45 , the screen for outputting damage information among the output screens of the management device 341 according to another embodiment may include a selection window CO for selecting whether to agree to the new damage information. can The user or administrator may click the 'Confirm' button if they agree to the information on the new damaged area (ND), or click the 'Objection' button if they do not agree.

Referring to FIG. 46 , when the user or the manager clicks the 'Objection' button, the output screen of the management device 341 may display a damaged image calculated when entering a vehicle and a damaged image calculated when leaving the vehicle.

FIG. 47 is a diagram for explaining an output screen including information on a damage history among the output screens of the management device 341 .

Referring to FIG. 47 , the output screen including information about the damage history according to an embodiment may include a vehicle image display unit CI, a vehicle-related non-image information display unit IF, and a damage history display unit DH. can

Since the vehicle image display unit CI and the non-image information display unit IF have been described above, detailed descriptions thereof will be omitted.

The damage history display unit DH may display damage information that has occurred in the corresponding vehicle from the past to the present. In addition, the damage history display unit DH may display information including the date of occurrence of the damage, the time of occurrence of the damage, the number of damage, the location of the damage, and the like for damage that has occurred in the corresponding vehicle from the past to the present.

3.2.4.3 Vehicle information output Example 2 - Taxi company

The information output to the display of the management device 341 of the taxi vehicle management system corresponds to the information output from the parking management system described in 3.2.4.2. Accordingly, descriptions overlapping with those described in the parking management system will be omitted.

According to an embodiment, the management device 341 of the taxi vehicle management system may output the obtained vehicle information and/or the obtained auxiliary information based on the vehicle information. The management device 341 may output vehicle information and/or auxiliary information to the information communication device of the manager. Although not shown in the drawings, the above-described vehicle information and/or auxiliary information may be output to an information communication device of a taxi driver.

FIG. 48 is a diagram for explaining an output screen including information about a damage history among the output screens of the management device 341 .

Referring to FIG. 48 , the screen for classifying and outputting the history of damage in the taxi by driver may include a driver-related non-image information display unit (IF), a damage history display unit (DH), and a vehicle image display unit (CI). .

The non-image information display unit IF according to an embodiment may display information about the driver, for example, information about the driver's name, driver's license information, and the driver's address. In addition, the non-image information display unit IF may display information about the total mileage driven by the driver, the total driving time, the total number of accidents, the total number of repairs/replacements, the total repair/replacement costs, and the like.

The damage history display unit DH according to an exemplary embodiment may display damage information caused by the driver from the past to the present. In addition, the damage history display unit DH may display information on damage caused by the driver from the past to the present, such as a damage occurring vehicle number, a damage occurring date, the number of damage, and a location of damage.

The vehicle image display unit CI according to an embodiment may display details regarding damage information selected by the driver or manager when the driver or manager selects any one of the driver's damage histories. In this case, the detailed information on the damage information selected by the driver or the manager may include information on the insurance number of the vehicle in which the accident occurred, whether parts are replaced/repaired, and the price of parts replacement/repair, and the like.

In addition, the damage history information output screen may classify and output the damage history generated in the taxi for each vehicle. The management device 341 may link and display the driver when damage has occurred for each damaged area generated in the taxi.

4. Mobile-based vehicle management system

4.1 Mobile Systems

According to an embodiment, a vehicle management system for managing the state of a vehicle using an image captured by a mobile device may be provided.

49 is a diagram for explaining the configuration of a mobile-based vehicle management system.

Referring to FIG. 49 , the mobile-based vehicle management system according to an embodiment may include a terminal 490 and a server 499 .

The terminal 490 may be connected to the server 499 . When the terminal 490 is connected to the server 499 , the terminal 490 may exchange data with the server 499 . In this case, the server 499 may be connected to the plurality of terminals 490 to receive and process data from the plurality of terminals 490 .

The terminal 490 may include at least one camera module 491 , a control unit 492 , a storage unit 493 , a communication unit 494 , an input unit 495 , and an output unit 496 .

The camera module 491 may photograph the exterior of the vehicle. The camera module 491 may photograph the exterior of the vehicle before and after the shared vehicle is used.

The storage unit 493 may store photographing data captured by the camera module 491 . Also, the storage unit 493 may store non-image data.

The communication unit 494 is configured to transmit/receive data. The terminal 490 and the server 499 may be connected to each other through the communication unit 494 to exchange data.

The input unit 495 is configured to receive data from a user or an administrator. The data may include information about the vehicle, user information, and the like.

The output unit 496 is a configuration for outputting vehicle-related information to a user or a manager, and may refer to a module such as a display.

The control unit 492 may control the storage unit 493 , the communication unit 494 , the input unit 495 , and the output unit 496 . The control unit 492 controls so that the photographed data photographed by the camera module 491 can be stored in the storage unit 493 , and controls data to be transmitted/received from the communication unit 494 in the form of various embodiments, and the input unit 495 ) can be controlled to receive vehicle-related information and user information, and the like, and can be controlled to output vehicle-related information to a user or manager through the output unit 496 .

4.2 Mobile-based vehicle management process

According to an embodiment, a vehicle management process for managing the state of a vehicle using an image captured by a mobile device may be provided. The mobile-based vehicle management process may be performed by the aforementioned mobile-based vehicle management system.

50 is a diagram for explaining a series of steps performed by the mobile-based vehicle management process.

Referring to FIG. 50 , the mobile-based vehicle management process includes the steps of obtaining vehicle data photographed from the mobile (S510), obtaining vehicle information based on the obtained vehicle data (S530), and managing the vehicle information (S510). S550) and outputting vehicle information (S570). Hereinafter, each step will be described in more detail.

4.2.1 Mobile Vehicle Data Acquisition

The mobile-based vehicle management process may include acquiring vehicle data required for vehicle management ( S510 ). The vehicle data may be provided in various forms. For example, the vehicle data may be image data or non-image data. The vehicle data may be pre-stored in the device or obtained from outside.

The vehicle data and the acquisition of the vehicle data will be described in more detail below with reference to specific examples.

4.2.1.1 Image data acquisition

Acquiring image data according to an embodiment may include acquiring image data including a plurality of images and extracting effective image data from among the plurality of acquired images. The image data and the effective image data have a configuration corresponding to the image data and the effective image data described in 3.2.1.1, respectively, and since the contents have been described above, the overlapping contents will be omitted.

51 is a diagram for explaining a request for re-photographing when vehicle damage information included in image data is not sufficient, according to an exemplary embodiment.

The mobile-based vehicle management process can operate normally only when the image data includes information on the number of damage, types of damage, degree of damage, location of damage, and information on damaged parts. Accordingly, when the information about the vehicle damage is not sufficiently expressed in the image data, the terminal 490 may request the user or the manager to re-photograph.

That is, the image data photographed by the terminal 490 is transmitted to the server 499, and the server 499 determines whether the information about vehicle damage is sufficiently expressed in the image data received, and the result is described above. may be transmitted to the terminal 490 . The terminal 490 may display the determination result to the user through the output unit 496 . The terminal 490 can inform the user that the photographed image data is suitable for shooting when information about vehicle damage is sufficiently expressed, and when the photographed image data does not sufficiently express information about vehicle damage, You can notify the user that a retake is required.

Alternatively, the terminal 490 may determine whether the photographed image data sufficiently expresses information about vehicle damage.

Referring to (a) of FIG. 51 , the image data MBB1 captured by the camera module 491 does not clearly include information regarding the number of damages, types of damage, degree of damage, and the like. In addition, referring to (b) of FIG. 51 , as a result of magnifying the damaged part, the image data MBB2 photographed by the camera module 491 includes information on the location where the damage occurred during the exterior of the vehicle, information on the damaged parts, etc. is not clearly included.

In this case, the control unit 492 of the terminal 490 sends the user or the manager through the output unit 496 on the number of damage, the type of damage, and the degree of damage to the image data MBB3 as shown in FIG. 51 ( c ). A message requesting re-shooting can be displayed so that information about the location of damage, information on damaged parts, etc. can all be included.

Alternatively, the control unit 492 of the terminal 490 may display a message requesting re-photographing of the existing damaged portion when the photographing of the existing damaged portion stored in the storage 493 has not proceeded.

Meanwhile, the image data may refer to data captured by the camera module 491 between a plurality of specific viewpoints. In this case, the plurality of specific time points may mean different time points. Hereinafter, the plurality of specific time points will be described in more detail with specific examples with reference to the drawings.

52 is a diagram for explaining that image data is captured by the camera module 491 at a specific time point according to an embodiment.

The image data may include a plurality of images. The image data may include a first image and a second image. The first image is an image captured by the terminal 490 located at a first point having a first distance D1 from a specific point of the vehicle being photographed. The second image is an image captured by the terminal 490 located at a second point having a second distance D2 from the specific point of the vehicle. In this case, the first distance D1 and the second distance D2 may be different.

For example, the image data may be an image captured by a user's manipulation. The first image and the second image may be images captured by a user's manipulation. That is, the first image may be photographed by the user's first manipulation of the terminal 490 , and the second image may be captured by the second manipulation.

Alternatively, the terminal 490 may extract some of the image data captured by the user's manipulation. That is, the terminal 490 is photographed in the form of a moving picture by a user's manipulation, and the terminal 490 may extract image data in which a main part of the image data is photographed as a first image and a second image. Alternatively, the server 499 may receive the image data, and the server 499 may extract image data in which a main part of the image data is captured as a first image and a second image.

When the image data is acquired by the camera module 491 at a point having a first distance D1 and a second distance D2 different from the first distance D1, the image data includes the corresponding damage in addition to vehicle damage information. Various information other than information may be included. Accordingly, the terminal 490 may detect effective image data in which noise is removed from the image data, that is, more accurate information on vehicle damage is included.

In addition, the first image is an image in which a first area related to the exterior of the vehicle is photographed, and the first area includes at least one main part of the vehicle. The second image is an image in which a second area related to the exterior of the vehicle is photographed, and the second area includes at least one main part of the vehicle. In this case, at least some of the main parts of the vehicle photographed by the terminal 490 may be included in common in the first area and the second area.

53 is a view for explaining a photographing guide provided based on information on a target vehicle when photographing a vehicle with the camera module 491 .

Referring to (a) of FIG. 53 , the terminal 490 may display a shooting guide through the output unit 496 based on the information of the target vehicle when the user or manager takes a picture of the vehicle with the camera module 491 . there is. The information of the target vehicle may be obtained using a neural network model trained to obtain information input by a vehicle user, reservation information of the vehicle user, or vehicle information. For example, when the vehicle user makes a reservation for a specific vehicle type before using the vehicle, the terminal 490 may provide a shooting guide for the vehicle type to the user or the manager through the output unit 496 based on the user's reservation information. there is. As another example, when a vehicle user takes a picture of a vehicle with the camera module 491 for vehicle use, the terminal 490 obtains information about the vehicle model through a neural network model trained to acquire vehicle information, and then relates to the vehicle model. A shooting guide may be provided to a user or an administrator through the output unit 496 . Meanwhile, the photographing guide shown in (a) of FIG. 53 is exemplary, and the photographing guide according to an embodiment may include various known types of photographing guides.

Referring to (b) and (c) of FIG. 53 , when the user takes a picture of a vehicle based on the above-described shooting guide, when the corresponding vehicle is properly located in the shooting guide by a certain ratio or more, the controller 492 of the terminal 490 ) activates the shooting button to enable the user's vehicle shooting and storage.

In addition, when the user takes a picture of a vehicle based on the above-described shooting guide, when the vehicle is located in the shooting guide at a certain ratio or more, the control unit 492 of the terminal 490 automatically controls the camera module 491 of the vehicle. After controlling to photograph the exterior, it is possible to control so that the photographed image is stored in the storage unit 493 of the terminal 490 .

Meanwhile, although not shown in FIG. 53 , the terminal 490 may provide a photographing guide for each part for photographing specific parts of the vehicle as well as providing a photographing guide for photographing the entire vehicle. The photographing guide for each part may be obtained using a neural network model trained to obtain information input by a vehicle user, reservation information of a vehicle user, or vehicle information, and since this has been described above, overlapping content will be omitted.

54 is a view for explaining a photographing map provided when photographing a vehicle with the camera module 491 .

Referring to FIG. 54 , the terminal 490 may display a photographing map to the user through the output unit 496 . The terminal 490 may display, through the output unit 496 , a photographing map that guides the user so that all major parts of the exterior of the vehicle can be photographed when the user photographs the vehicle with the camera module 491 . More specifically, the photographing map may include a UI for inducing photographing of the vehicle in various directions so that all major parts of the exterior of the vehicle may be photographed. Exemplarily, the main parts of the vehicle exterior may include a front surface, a rear surface, a front passenger seat front, a passenger seat rear surface, a driver's seat front surface, and a driver's seat rear surface of the vehicle.

The photographing map may include a guide unit GU and a photographing unit CB. Hereinafter, specific embodiments of the guide unit GU and the photographing unit CB will be described with reference to the drawings.

The terminal 490 may guide the photographing through the guide unit GU so that a main part of the vehicle exterior may be photographed when the user photographs the exterior of the vehicle with the camera module 491 . The guide unit GU may include at least one guide so that all major parts of the exterior of the vehicle can be photographed. The terminal 490 may induce vehicle photographing through the guide unit GU so that the vehicle may be photographed in different directions.

Illustratively, the guide unit GU includes a first guide for inducing photographing of the front of the vehicle, a second guide for inducing photographing of the front side of the passenger seat, a third guide for inducing photographing of the rear of the passenger seat, and photographing of the rear of the vehicle. It may include a fourth guide for inducing, a fifth guide for inducing photographing of the rear side of the driver's seat, and a sixth guide for inducing photographing of the front side of the driver's seat.

Meanwhile, the photographing unit CB may include a function of photographing the exterior of the vehicle and storing the photographed image. The photographing unit CB may include a photographing button capable of photographing a main part of the exterior of the vehicle. More specifically, when the user selects any one of the guides described above, the photographing unit CB may include a photographing button capable of photographing the exterior of the vehicle in which the corresponding guide induces photographing.

For example, when the user selects a first guide for inducing photographing of the front portion of the vehicle in the guide unit GU, the terminal 490 uses a photographing button capable of photographing the front portion of the vehicle through the output unit 496 . The activated photographing unit CB may be displayed. In this case, when the user clicks the photographing button to photograph the front portion of the vehicle, the controller 492 of the terminal 490 may classify the photographed image as the front portion of the vehicle and store it in the storage 493 .

Meanwhile, image data acquired by the camera module 491 may include a plurality of images. In this case, valid image data among the plurality of images may be extracted so that the mobile-based vehicle management process can be normally performed.

Since the step of extracting valid image data suitable for the mobile-based vehicle management process from the image data corresponds to step S3513 of FIG. 36 , overlapping content is omitted.

4.2.1.2 Acquisition of non-image data

According to an embodiment, vehicle data in the form of a non-image other than an image may be obtained. The vehicle data may be non-image data. The vehicle data may be non-image data including information related to the vehicle. Since the non-image data has been described above in 3.2.1.2, redundant contents will be omitted.

The non-image data may be previously stored or obtained. The non-image data may be acquired by the camera module 491 . The non-image data may be obtained based on photographing data captured by the camera module 491 . The management device 341 may acquire non-image data based on the image data.

4.2.1.3 Data processing

In the mobile-based vehicle management process, processing may be performed on the acquired data.

In the mobile-based vehicle management process, pre-processing (or post-processing) may be performed on the acquired data. The terminal 490 may perform pre-processing on at least one valid image data among the image data. The terminal 490 may perform pre-processing based on the first image and the second image among the image data.

The terminal 490 may perform noise removal processing based on a plurality of images captured between a plurality of specific points. The terminal 490 may perform noise removal processing based on a plurality of valid image data.

An image on which the above-described noise removal processing has been performed may be defined as a damaged image. The terminal 490 may determine vehicle information including damage to the vehicle based on the damage image.

When the camera module 491 is installed in an environment in which noise is not generated, the management device 341 may omit the noise removal process. In this case, vehicle information, which will be described later, may be obtained based on valid image data. In this case, the damaged image may be the valid image data.

The terminal 490 may perform noise removal processing for removing reflected light, etc., based on a plurality of valid image data. The terminal 490 acquires a plurality of valid image data, and an object whose position is changed (eg, a reflective region corresponding to reflected light) and an object whose position does not change (eg, a vehicle region or a damaged region corresponding to damage to the vehicle) ), it is possible to remove an object whose position changes or an object whose position does not change. For example, the terminal 490 removes noise to obtain a vehicle image from which the reflection area is removed based on effective image data including a damaged area corresponding to a damaged part of the vehicle and a reflection area corresponding to light reflected by the surface of the vehicle. processing may be performed. The effective image data may be image data including a main part.

Since the terminal 490 is not photographed in a fixed form, the first image having a first distance from the main part and the second image having a second distance to the main part are between the camera module 491 from the main part. It is highly probable that the distance of That is, the first distance and the second distance may be different. Accordingly, when the user captures the first image and the second image by the first and second manipulations while carrying the terminal 490 , and when the user carries the terminal 490 , one In the case of capturing a video by manipulation and extracting the first image and the second image based thereon, the first distance and the second distance are highly likely to be different.

The first distance may be a distance between the camera module 491 and a first feature point of the vehicle when a first image is captured, and the second distance is between the camera and a first feature point when a second image is captured It can be the distance between

Therefore, it is possible to obtain effective image data on which noise processing can be performed without extra effort, and the terminal 490 can perform noise removal processing based on this.

Although it has been described that the noise removal processing is performed in the terminal 490 , the noise removal processing may be performed in the server 499 .

4.2.2 Obtaining mobile vehicle information

The mobile-based vehicle management process may include obtaining vehicle information based on vehicle data.

The vehicle information may include parts information related to parts constituting the target vehicle, damage information related to damage inside/outside the target vehicle, vehicle model information related to the type of the target vehicle, and/or identification information for identifying the target vehicle. .

Vehicle information may be extracted from vehicle data or obtained by processing vehicle data. Vehicle information may be obtained using vehicle data as input data, an algorithm designed to obtain vehicle information, or a neural network model trained to obtain vehicle information.

Acquiring vehicle information based on vehicle data has been described above in 2.3, and thus redundant information is omitted.

According to an embodiment, the terminal 490 may acquire vehicle information based on the damaged image. The terminal 490 may acquire damage information based on the damage image.

According to an embodiment, in the case of a car-sharing company, the terminal 490 may acquire shared vehicle information using a pre-stored database. The shared vehicle information may include vehicle type information indicating a vehicle type, manufacturer, model name, year, and the like of the shared vehicle. In addition, the shared vehicle information may further include parts information indicating the number, type, name, identification information, and the like of parts constituting the shared vehicle. In addition, the shared vehicle information may further include additional information including personal information of the shared vehicle user or an accident history of the shared vehicle.

The shared vehicle information may include additional damage information generated inside/outside the target vehicle. The additional damage information may be calculated based on the damage image of the vehicle before the user uses the vehicle and the damage image of the vehicle after the user uses the vehicle. The additional damage information may be calculated by comparing the damage image of the vehicle before the user uses the vehicle with the damage image of the vehicle after the user uses the vehicle.

The additional damage information may be calculated based on damage information calculated based on the damage image of the vehicle before the user uses the vehicle and damage information calculated based on the damage image of the vehicle after the user uses the vehicle.

The terminal 490 stores at least one of the damage image and damage information of the vehicle before the user uses the vehicle, and calls it after the user uses the vehicle, and the damage image and damage information of the vehicle after the user uses the vehicle It is possible to calculate additional damage information by comparing with at least one of the.

The terminal 490 may store non-image data of the vehicle before the user uses the vehicle in association with the damaged image and damage information. For example, the terminal 490 may store the damage image and damage information of the vehicle before the user uses the vehicle in association with the vehicle number of the vehicle.

When the user ends using the vehicle, the terminal 490 loads the damage image and damage information linked with the vehicle number of the vehicle based on the recognized vehicle number, and uses this information on the vehicle after the user uses the vehicle. Comparing with the damage image and damage information, additional damage information can be calculated.

4.2.3 Vehicle information management

The mobile-based vehicle management process may include managing vehicle information. Managing the vehicle information may include managing the aforementioned part information, damage information, vehicle type information, and/or identification information.

In addition, managing the vehicle information may include managing the vehicle information in time series. Managing the vehicle information may include classifying the vehicle information. Managing the vehicle information may include managing one or more pieces of information in association. The vehicle information management may be performed by the server 499 .

Hereinafter, management of vehicle information will be described with reference to some embodiments.

4.2.3.1 History management

Managing the vehicle information may include managing the history of the vehicle information. History management of vehicle information may mean listing and/or managing one or more vehicle information acquired at different times in time series.

According to an embodiment, managing vehicle information in the mobile-based vehicle management process may mean managing damage information indicating damage to the vehicle for a specific vehicle together with time information indicating a time at which the damage information was acquired. there is. For example, managing the vehicle information acquires first information indicating that the first damage has occurred to the first vehicle at a first time point, and the second damage has occurred to the first vehicle at a second time point that is after the first time point It may include acquiring second information indicating

For example, managing vehicle information in the car-sharing process may mean managing damage information indicating damage to the vehicle with respect to the shared vehicle together with time information indicating a time at which the damage information was acquired.

More specifically, managing vehicle information in the car-sharing process means that the first user acquires first information indicating that the first damage has occurred in the shared vehicle before using the first vehicle, and the first user uses the first vehicle After obtaining second information indicating that the second damage has occurred in the shared vehicle, the first user obtains third information indicating that the third damage has occurred in the shared vehicle before using the second vehicle, and the first user After obtaining the fourth information indicating that the fourth damage has occurred to the shared vehicle after using the second vehicle, it may include managing the first information, the second information, the third information, and/or the fourth information in chronological order. can

In addition, managing vehicle information in the car-sharing process means acquiring first information indicating that the first damage has occurred to the shared vehicle after the first user's use of the shared vehicle, and providing the shared vehicle with the second user before using the shared vehicle. After obtaining the second information indicating that the second damage has occurred, and after obtaining the third information indicating that the third damage has occurred in the shared vehicle after the second user uses the shared vehicle, the first information, the second information and / or may include managing the third information in chronological order. In this case, the first user may mean a person who has used the shared vehicle in preference to the second user.

4.2.3.2 Damage history management Example 1 - By driver

It is possible to manage the history of damage information in the mobile-based vehicle management process. History management of damage information may include acquiring new damage information for new damage that has occurred. History management of damage information may include acquiring new damage information based on a plurality of images obtained by photographing the same vehicle at various time points.

Damage information history can be managed for each user in consideration of user information.

According to an embodiment, the terminal 490 may manage the damage information history of the shared vehicle for each user in consideration of user information. The terminal 490 manages the damage information history of the shared vehicle for each user, so that when damage occurs to the shared vehicle, it is possible to clarify the location of responsibility, and to determine which user caused the damage that occurred in the shared vehicle in the present or in the past. It is easy to understand, and it can provide the effect of reducing the cost of managing a shared vehicle. In addition, the terminal 490 may provide an effect of performing an evaluation of the user by managing the damage information history of the shared vehicle for each user.

For example, that the terminal 490 manages the damage information history of the shared vehicle for each user in consideration of user information is first information indicating that the first damage occurred in the shared vehicle before the first user used the first vehicle. and managing additional damage information that occurred during the first user's use of the first vehicle obtained by comparing the second information indicating that the second damage occurred to the shared vehicle after the first user used the first vehicle, and the first user The third information obtained by comparing the third information indicating that the third damage occurred to the shared vehicle before using the second vehicle and the fourth information indicating that the fourth damage occurred to the shared vehicle after the first user used the second vehicle 1 may mean managing additional damage information that occurred while using the user's second vehicle.

4.2.3.3 Damage history management Example 2 - Management by vehicle

The damage information history may be managed for each vehicle.

According to an embodiment, the terminal 490 may manage the damage information history of the shared vehicle for each vehicle. Since the terminal 490 manages the damage information history of the shared vehicle for each vehicle, it is possible to provide the effect of easily identifying damage information that has occurred from the past to the present in the shared vehicle in time series. In addition, the terminal 490 manages the damage information history of the shared vehicle for each vehicle, so that the repair and inspection schedule of the shared vehicle can be planned according to the condition of the vehicle as a customized schedule for each vehicle, thereby enabling efficient vehicle management effect can be provided.

For example, when the terminal 490 manages the damage information history of the shared vehicle for each vehicle, the sharing of the first information and the second user indicating that the first damage has occurred in the shared vehicle after the first user uses the shared vehicle It may mean managing additional damage information that occurs during parking of the shared vehicle by comparing the second information indicating that the second damage has occurred in the shared vehicle before using the vehicle.

As another example, when the terminal 490 manages the damage information history of the shared vehicle for each vehicle, the first information indicating that the first damage has occurred in the shared vehicle after the first user's use of the shared vehicle or the second user's shared vehicle Comparing the second information indicating that the second damage occurred to the shared vehicle before use with the third information indicating that the third damage occurred to the shared vehicle after the second user used the shared vehicle, the second user is using the shared vehicle It may mean managing additional damage information that has occurred.

4.2.4 Vehicle information output

4.2.4.1 Output information

The mobile-based vehicle management system may output vehicle information and auxiliary information obtained based on the vehicle information through the output unit 496 of the terminal 490 . Also, the terminal 490 may output vehicle information and auxiliary information through the server 499 connected to the terminal 490 .

The terminal 490 may output vehicle information such as parts information, damage information, vehicle type information, and identification information. The vehicle management system may output user information and/or location information together.

The terminal 490 may output a vehicle image and vehicle information corresponding to the vehicle image. The vehicle management system may output a plurality of vehicle images and vehicle information corresponding to each vehicle image. The plurality of vehicle images may be images captured at various angles with respect to the vehicle.

Hereinafter, a vehicle information output screen of the shared vehicle management system will be described with reference to FIG. 55 .

4.2.4.2 Example of vehicle information output

55 is a diagram for explaining a screen for outputting vehicle information and/or auxiliary information in the terminal 490 according to an embodiment.

According to an embodiment, the terminal 490 may output the obtained vehicle information and/or auxiliary information obtained based on the vehicle information. The terminal 490 may output vehicle information and/or auxiliary information to the information communication device of the user or manager.

The terminal 490 may output vehicle information and/or auxiliary information when additional damage information is detected in the shared vehicle. Also, the terminal 490 may output vehicle information and/or auxiliary information when additional damage is detected in the shared vehicle before/after the user's shared vehicle use. In addition, the terminal 490 may output vehicle information and/or auxiliary information even if additional damage information is not detected for management of the shared vehicle.

55 (a) is a diagram for explaining a screen for outputting information about the current external state of the shared vehicle.

Referring to (a) of FIG. 55 , the shared vehicle exterior information screen according to an embodiment may include a damage information table and a damage image on which damage information of the vehicle exterior is displayed.

The vehicle exterior information output screen may include a damage information table that displays information on damage that has occurred in the shared vehicle. The damage information table may include damaged vehicle parts, types of damage, degree of damage, number of damage, and the like.

The vehicle exterior information output screen may include a damage image in which damage information is displayed. The damaged image may include damaged area information, and the damaged area information may include an existing damaged area (ED).

The vehicle exterior information output screen may include a selection window for selecting whether to agree to the existing damage information. Users can click the 'Confirm' button if they agree to the information about the damaged area, or click the 'Objection' button if they do not agree. When the 'Confirm' button is clicked by the user, the terminal 490 may inform the server 499 that the collective agreement for the existing damage information has been performed.

Alternatively, the vehicle exterior information output screen may display a damage image in which damage information is displayed. The terminal 490 may display the vehicle exterior information output screen so that the user's individual consent to the damaged image can be progressed. That is, the information on the damaged image of the terminal 490 may be mapped to the vehicle image and output, and the vehicle image may be rotated, enlarged, reduced, or moved by a user's manipulation. The terminal 490 may display a button for consenting to the damaged part of the vehicle image, and when the user presses the button, it is determined that individual consent has been made, and the consent information is transferred to the server 499 can be sent to

55B is a view for explaining a vehicle exterior photographing request screen among output screens of the shared vehicle management system.

Referring to FIG. 55B , the vehicle exterior photographing request screen according to an embodiment may include a guide unit and a photographing unit. The vehicle exterior photographing request screen may include a guide unit for guiding photographing so that a major part of the exterior of the vehicle can be photographed, and a photographing unit including a photographing button for photographing the exterior of the vehicle. In this case, the vehicle exterior photographing request screen according to an embodiment may be a screen that is essentially output before the user uses the vehicle, but is not limited thereto. That is, the vehicle exterior photographing request screen may be selectively output by the user.

FIG. 55C is a view for explaining a screen for outputting information on whether additional damage has occurred to the vehicle after photographing the exterior of the vehicle is completed.

Referring to (c) of Figure 55, the vehicle damage information screen according to an embodiment is a damage information table, a selection window for selecting whether to confirm or agree to the occurrence of additional damage and/or damage in which additional damage information is displayed May contain images.

The screen for outputting additional damage information may include a damage information table that displays information about damage that has occurred to the shared vehicle. The damage information table may include damaged vehicle parts, types of damage, degree of damage, number of damage, and the like.

The screen for outputting the additional damage information may include a selection window for selecting whether to agree to the new damage information. The user can click the 'Confirm' button if they agree to the information about the new damaged area (ND), or click the 'Objection' button if they do not agree. When the 'Confirm' button is clicked by the user, the terminal 490 may inform the server 499 that the collective agreement for the existing damage information has been performed.

Alternatively, the terminal 490 may perform a procedure of obtaining an individual consent for the new damage information by using a plurality of individual buttons, like the procedure for obtaining the consent for the existing damage information described above.

The screen outputting the additional damage information may include a damage image in which the additional damage information is displayed. The damaged image may include damaged area information, and the damaged area information may include an existing damaged area ED and a new damaged area ND. In this case, the existing damaged area ED and the new damaged area ND may be displayed differently. The existing damaged area ED and the new damaged area ND may be displayed with different colors or different types of bounding boxes.

55 (d) is a view for explaining a screen including detailed information regarding damage information that has occurred in the shared vehicle.

Referring to (d) of FIG. 55 , when the user clicks the 'Objection' button, the output screen of the shared vehicle management system shows the damaged image extracted before the user's shared vehicle use and the damaged image extracted after the user's shared vehicle use. can be displayed. In this case, the damaged image extracted before using the shared vehicle and the damaged image extracted after using the shared vehicle may be damaged images detected in directions corresponding to each other. As another example, the damaged image extracted before using the shared vehicle and the damaged image extracted after using the shared vehicle may include at least one main part of the vehicle, and at least one main part of the vehicle may be common. On the other hand, the damaged image extracted before using the shared vehicle and the damaged image extracted after using the shared vehicle may be in the form of mapped onto a processed image, eg, a 3D image of a vehicle, rather than an actual photographed image.

4.3 Confirmation Scenario

According to an embodiment, the mobile-based vehicle management process may include a step in which the vehicle return process is completed only after the shared vehicle user uses the shared vehicle, photographs the exterior of the shared vehicle, and passes a consent step for additional damage information.

In addition, the mobile-based vehicle management process may include enabling the vehicle to be used when the shared vehicle user agrees to the current appearance state of the shared vehicle before using the shared vehicle.

In addition, in the mobile-based vehicle management process, if the shared vehicle user does not agree with the current exterior state of the shared vehicle before using the shared vehicle, the steps of re-photographing and registering the part of the vehicle exterior or the entire exterior of the vehicle must be performed. It may include the step of making it available.

56 is a diagram schematically illustrating a process in which a plurality of shared vehicle users use the shared vehicle and end the use according to time sequence.

Referring to FIG. 56 , the first user may start using the shared vehicle at a first time point T1 and end using the shared vehicle at a second time point T2 . In addition, the second user may start using the shared vehicle at the third time point T3 and end the shared vehicle use at the fourth time point T4 . In this case, it may be assumed that the first user and the second user use the same shared vehicle. In addition, the first to fourth time points may refer to different time points, respectively, and the first time point may be a time prior to the third time point.

In addition, referring to FIG. 56 , the shared vehicle use time of the first user is a first section P1 between the first time point and the second time point, and the second user's shared vehicle use time is the third time point and the fourth time point. It is the third section P3 that is in between. That is, the first section P1 may mean a period in which the first user uses the vehicle, and the third section P3 may mean a period in which the second user uses the vehicle.

On the other hand, the period from the end of the first user's use of the shared vehicle to the start of the second user's use of the shared vehicle may mean a time when the shared vehicle is parked without being driven by the user. That is, the second section P2, which is a period between the second time point T2 and the third time point T3, may mean a section in which the shared vehicle is parked without being driven by the user.

Hereinafter, referring to FIG. 56 , the step of using the vehicle after the shared vehicle user consents to the existing damage information of the shared vehicle before using the shared vehicle, the step of using the vehicle after re-photographing the exterior in case of disagreement, and the user of the shared vehicle using the shared vehicle The steps of returning the shared vehicle after taking a picture of it after use will be described in detail.

4.3.1 Vehicle exterior photography and additional damage management after vehicle use

57 is a view for explaining a vehicle return process in which a shared vehicle user takes a picture of the vehicle exterior and checks additional damage information after using the shared vehicle. Hereinafter, with reference to FIGS. 56 and 57, a process in which a shared vehicle user photographs the exterior of the vehicle after using the shared vehicle and confirms additional damage information will be described.

Referring to FIG. 57 , the mobile-based vehicle management system may include a terminal 490 and a server 499 . The terminal 490 may be the terminal 490 of the first user. The vehicle use time of the first user may end at a second time point T2.

The first user's terminal 490 may recognize that the vehicle use time has expired under the control of the first user, or the first user's terminal 490 based on information received from the server 499 It may be displayed that the vehicle use time has ended.

The first user's terminal 490 may display a shooting guide through the output unit 496 . The display of the photographing guide through the output unit 496 may be an optional process as described in 4.2.1.1.

The first user's terminal 490 may take a first image through the camera module 491 under the control of the first user. The first image may include at least one image related to the exterior of the vehicle used by the first user. The first image may be a plurality of images including a main part of the exterior of the vehicle. The first image may be an image captured at a time point corresponding to the second time point T2.

The second user's terminal 490 may also take a third image using the camera module 491 at a fourth time T4 when the vehicle use time ends under the control of the second user. The third image may include at least one image related to the exterior of the vehicle used by the second user. The third image may be a plurality of images including a main part of the exterior of the vehicle. The third image may be an image captured at a time point corresponding to the fourth time point T4.

The terminal 490 of the first user may transmit the first image to the server 499 through the communication unit 494 after storing the first image in the storage unit 493 . The first user's terminal 490 may transmit the first image to the server 499 at a time point corresponding to the second time point T2 . Also, the second user's terminal 490 may transmit the third image to the server 499 at a time point corresponding to the fourth time point T4 .

The server 499 may receive the first image from the first user terminal 490 .

The server 499 may determine whether there is additional damage to the vehicle based on the first image.

The server 499 may compare the first image with data stored in the server 499 to determine whether the shared vehicle is further damaged. The data stored in the server 499 may be existing damage information of the shared vehicle. The server 499 may analyze the first image to determine whether there is additional damage to the shared vehicle. That is, the server 499 may determine whether there is damage to the vehicle used by the shared vehicle user while the shared vehicle user is driving.

In the end, the server 499 compares the image of the shared vehicle after the shared vehicle user uses the shared vehicle with the existing damage information of the shared vehicle to determine additional damage information that occurred in the vehicle, thereby taking responsibility for the damage caused to the shared vehicle. can be clarified, and damage that occurs to shared vehicles can be managed in a time-series manner to enable systematic management. Illustratively, the server 499 compares the third image with the first image to determine additional damage information, thereby starting from the second time point T2 which is the end point of the first user's use to the second time point which is the end point of the second user's use. Damage information that occurred between 4 time points (T4) can be identified and managed. In other words, the server 499 may identify and manage damage information occurring in the second section P2 or the third section P3 by comparing the third image and the first image to determine additional damage information.

The server 499 may update damage information about the vehicle when it is determined that no additional damage has occurred to the vehicle. That is, when the server 499 determines that no additional damage has occurred to the vehicle, no additional damage has occurred to the vehicle, and the damage information about the vehicle is the same as the existing damage information. can be updated. In this case, the updated damage information may be defined as first information.

For example, when the server 499 determines that no additional damage has occurred to the vehicle from the first image or the third image captured at the second time point T2 or the fourth time point T4, damage to the vehicle Damage information about the vehicle can be updated with the content that the information is the same as the existing damage information.

As another example, when the server 499 determines that no additional damage has occurred to the vehicle from the third image captured at the fourth time point T4 , the damage information on the vehicle is the second time point T2 captured at the second time point T2 . 1 It can be updated with the same damage information as determined from the image.

When the server 499 determines that additional damage has occurred to the vehicle, the server 499 may transmit the corresponding additional damage information to the user's terminal 490 .

The user's terminal 490 may receive additional damage information from the server 499 .

When the additional damage information is received, the user's terminal 490 may confirm whether the user agrees to the additional damage.

The user's terminal 490 may display whether or not to consent to the additional damage to the user by using the output unit 496 and allow the user to select whether to agree to the additional damage information. The user may select whether to agree to whether the additional damage information is damage that occurred during the period of using the vehicle through the input unit 495 .

In this case, the additional damage information may include additional damage information that occurred during the period of use of the current user. Illustratively, the additional damage information may refer to damage information additionally generated to the vehicle between the first time point T1, which is the start time of the first user's use, and the second time point T2, which is the end point of the first user's use. can That is, the additional damage information may refer to damage information additionally generated in the vehicle in the first section P1, which is the period of use of the first user.

The additional damage information at the fourth time point T4 is additionally generated in the vehicle between the third time point T3, which is the start time of the second user's use, and the fourth time point T4, which is the use end time point of the second user. It may mean damage information. That is, the additional damage information may refer to damage information additionally generated in the vehicle in the third section P3, which is the period of use of the second user.

The additional damage information may refer to damage information additionally generated in the vehicle between the second time point T2, which is the end point of the first user's use, and the fourth time point T4, which is the end point of the second user's use. That is, the additional damage information may refer to damage information additionally generated in the vehicle in the second section P2 or the third section P3.

In other words, when the second user terminal 490 receives the additional damage information from the server 499, the additional damage information is transmitted between the second time point T2 or the third time point T3 and the fourth time point T4. You can choose whether or not to agree on whether or not the damage has occurred. As another example, whether the second user agrees on whether the additional damage information is damage occurring in the second section (P2) or the third section (P3) when the terminal 490 receives the additional damage information from the server 499 can be selected.

The vehicle return process can be completed only after a procedure of confirming the additional damage information by the user's terminal 490 . That is, when the time reserved by the user elapses, the use of the vehicle is terminated, but the vehicle return process can be completed only after the user agrees to or object to the additional damage information.

If the user agrees to the additional damage information, the vehicle return process can be completed. If the user acknowledges that there is no objection to the additional damage information, the vehicle return process may be completed.

For example, when the first user agrees to the additional damage information using the first user terminal 490 , the vehicle return process may be completed.

When the second user uses the second user terminal 490 to determine whether the additional damage information is additional damage to the vehicle between the second time point T2 or the third time point T3 and the fourth time point T4, If it is agreed that the damage is additionally caused to the vehicle in the second section P2 or the third section P3, the vehicle return process may be completed.

If the user does not agree to the additional damage information, the user may object to the reason for not agreeing to the additional damage information, and when the objection is completed, the vehicle return process may be completed. That is, when the user does not agree in whole or in part with the additional damage information, the objection procedure may be performed for the part to which the user does not agree, and when the objection procedure is completed, the vehicle return process may be completed.

Illustratively, if the second user receives the additional damage information from the second time point (T2) or the third time point (T3) to the fourth time point (T4), the damage additionally occurred to the vehicle, or the second section (P2) or In the third section (P3), if you do not fully agree or partially agree that it is an additional damage to the vehicle, you may raise an objection to the part you do not agree with. Thereafter, when the objection procedure is completed, the vehicle return process may be completed.

When the vehicle return process is completed, the first user's terminal 490 may transmit the additional damage information to the server 499 by reflecting the user's consent.

In addition, the server 499 may update the vehicle damage information based on the additional damage information in which the consent of the first user received from the terminal 490 of the first user is reflected. In this case, the updated vehicle damage information may be defined as first information.

As such, by updating the vehicle damage information based on the information obtained from the terminal 490, the server 499 can manage the damage information occurring on the exterior of the vehicle in time-series, and damage information on the exterior of the vehicle. By clarifying the location of responsibility for repair, inspection and replacement, it can be done smoothly.

4.3.2 Whether you agree to the existing damage information and re-photographed the vehicle exterior before using the vehicle

According to an embodiment, the mobile-based vehicle management process may include enabling the vehicle to be used when the shared vehicle user agrees to the existing damage information of the vehicle before using the vehicle. In addition, in the mobile-based vehicle management process, if the shared vehicle user does not agree to all or part of the existing damage information of the vehicle before using the vehicle, the step of re-photographing the part of the vehicle exterior or the entire exterior of the vehicle must be performed. It may include steps to make this possible.

58 is a diagram for explaining a vehicle use process in which a shared vehicle user displays whether or not to agree to the existing damage information of the shared vehicle before using the shared vehicle, and takes a picture of the exterior of the vehicle when the user does not agree. Hereinafter, a series of steps that the shared vehicle user must perform before using the shared vehicle in order to use the shared vehicle will be described with reference to FIGS. 56 and 58 .

Referring to FIG. 58 , the user's terminal 490 may activate the smart key before a predetermined time of using the vehicle. The server 499 may control the smart key of the user's terminal 490 to be activated. The second user's terminal 490 may activate the smart key before a predetermined time of the third time point.

When the smart key is activated, the second user's terminal 490 may request the server 499 for existing damage information on the vehicle.

In this case, the existing damage information may mean information about the current appearance state of the shared vehicle. In addition, the existing damage information may mean damage information of the vehicle updated by a previous user who used the vehicle just before the current user uses the vehicle. For example, the existing damage information may mean damage information that has already occurred in the shared vehicle before the shared vehicle user uses the shared vehicle. As another example, the existing damage information may refer to damage information generated in the shared vehicle by a previous shared vehicle user. As another example, the existing damage information may mean damage that occurred before the current shared vehicle user uses the shared vehicle after the previous shared vehicle user completes returning the shared vehicle, for example, damage while parking.

Also, referring to FIGS. 56 and 58 , the existing damage information may mean first information updated from a first image captured at a second time point T2 after the first user uses the vehicle. For example, if the second user is defined as the current user and the first user is defined as the previous user, the existing damage information is between the first time point (T1) and the second time point (T2) or the third time point (T3). It may mean damage information that has occurred in , and the existing damage information may mean damage information that occurred in the first section P1 or the second section P2. Alternatively, when the first user does not proceed with any photographing, the existing damage information may be an image photographed by the user who has photographed the most recent image.

Meanwhile, the existing damage information may be expressed through an actual image of the shared vehicle or may be expressed through a processed 3D image. For example, the existing damage information may be provided in a form in which a damaged area is displayed as a bounding box on an actual image of the shared vehicle. As another example, the existing damage information may be provided in a form in which a damaged area is displayed as a bounding box on a processed 3D image in which information about a shared vehicle is reflected. In this case, when the existing damage information is provided in the form of a processed 3D image, the shared vehicle user can more conveniently grasp the overall damage information of the vehicle exterior through a method of rotating the 3D image by 360 degrees.

After receiving the existing damage information request message from the second user's terminal 490 , the server 499 may search for the existing damage information and transmit the corresponding content to the second user's terminal 490 . After that, the second user's terminal 490 receives the existing damage information, displays the existing damage information on the second user's terminal 490, and then displays a message to the second user to check the vehicle exterior information. .

The second user can choose whether to agree to the existing damage information. That is, the second user's terminal 490 displays information on the existing damage that has occurred in the shared vehicle to the shared vehicle user, and then displays a screen where the second user can select whether to agree to the existing damage information. there is. The consent may include a collective consent and a partial consent for the entire damage information.

The terminal 490 of the second user of the mobile-based vehicle management process may activate the door opening function and the starting function of the shared vehicle when the second user agrees to the existing damage information of the shared vehicle before using the shared vehicle. The second user's terminal 490 may activate a door opening function and a starting function of the shared vehicle when the second user inputs a collective agreement through the terminal 490 . That is, the second user can use the shared vehicle when collectively agreeing to the existing damage information of the shared vehicle before using the shared vehicle.

In this case, that the second user agrees to the existing damage information of the vehicle may mean that the existing damage information displayed on the terminal 490 of the second user agrees with the current exterior state of the vehicle.

Although not shown, when the collective agreement by the second user is input, the second user's terminal 490 may transmit information on the collective agreement to the server 499 . The server 499 may transmit data so that the terminal 490 of the second user can activate the door opening and starting functions after storing the information on the collective agreement.

When the collective consent by the second user is input, the server 499 may store information indicating that the exterior state of the vehicle is the same as the existing one.

When the second user does not agree to the existing damage information before using the vehicle, the second user's terminal 490 may display the second user to record and register the current exterior state of the vehicle. In this case, the second user's terminal 490 may provide a photographing guide so that the second user can photograph the current exterior of the vehicle before using the vehicle. Since the photographing guide overlaps with the contents described in 4.2.1.1, a detailed description thereof will be omitted.

More specifically, referring to FIG. 56 , the second user's terminal 490 may display the display so that the second user can take a vehicle exterior photograph at a third time point T3 when the vehicle use time starts. In this case, the vehicle image taken by the second user before the start of use may be the second image. That is, an image captured by the second user's terminal 490 at a time point corresponding to the third time point T3 may be defined as the second image. The second image is an image related to the exterior of the vehicle, and may include a plurality of images including a main part of the exterior of the vehicle. The first user may also photograph the exterior of the vehicle at the first time point T1 when the vehicle use time starts.

The second user's terminal 490 proceeds with the procedure for photographing the exterior only when the second user does not agree with the existing damage information, so that the verification process before using the shared vehicle in case there is no dispute over the damage information It can be further simplified.

The image data photographed by the second user may be stored and registered in the terminal 490 of the second user and then transmitted to the server 499 . For example, the second image captured by the second user at the third time point T3 may be stored in the terminal 490 of the second user and then transmitted to the server 499 . In addition, an image captured by the first user at the first time point T1 may also be transmitted to the server 499 .

After receiving the second image from the terminal 490 of the second user, the server 499 may determine whether additional damage has occurred to the vehicle based on the second image and the existing damage information. More specifically, after receiving the second image from the second user's terminal 490, the server 499 may compare the second image with existing damage information of the vehicle to determine whether additional damage has occurred to the vehicle. That is, the server 499 may determine whether the current exterior state of the vehicle is different from the existing damage information.

Exemplarily, the server 499 compares and analyzes the second image captured by the second user at the third time point T3 before use with the first image captured at the second time point T2 after the first user uses it. It can be determined whether additional damage has occurred to the vehicle. That is, the server 499 may determine whether additional damage has occurred in the second section P2 between the second time point T2 and the third time point T3 .

As a result, the server 499 can update information about the current appearance state of the shared vehicle by determining whether additional damage has occurred, can identify damage information that was missing from the existing damage information, and terminate the use of the previous user It is possible to provide the effect of identifying additional damage information generated during parking after parking. Illustratively, the server 499 compares the second image with the first image to determine additional damage information, so that from the second time point (T2), which is the end point of the first user's use, the second time point that is the use start time of the second user. Damage information occurring between the three time points T3, that is, damage information occurring in the second section P2, may be identified and managed.

On the other hand, the second user may not agree with all or part of the existing damage information of the shared vehicle, and the scope of re-photographing the exterior of the shared vehicle may vary depending on whether all or part of the consent is given. A detailed description thereof will be provided later.

When determining that additional damage has occurred to the vehicle, the server 499 may update damage information about the corresponding vehicle. That is, when determining that additional damage has occurred to the vehicle, the server 499 may update the existing damage information to match the current exterior state of the vehicle. In this case, the updated damage information may be defined as second information.

For example, if the server 499 determines that additional damage has occurred to the vehicle from the second image captured at the third time point T3, the server 499 may update the existing damage information to match the current exterior state of the vehicle. . The update content may relate to damage information that was missing from the existing damage information, and may relate to additional damage information that occurred while parking after the previous user's use of the vehicle was terminated.

When the server 499 determines that no additional damage has occurred to the vehicle, the server 499 may transmit the determination result to the second user's terminal 490 , and the second user's terminal 490 receives the determination from the server 499 . Upon receiving the result, the second user may select whether to agree to the determination result. In this case, the determination result may include information indicating that there is no problem in the existing damage information, that is, information that the existing damage information is the same as the current exterior state of the vehicle.

The second user's terminal 490 may activate the door opening and starting functions of the vehicle when the second user completes the procedure for confirming the determination result.

The second user's terminal 490 may activate the door opening and starting functions of the vehicle when the second user agrees to the determination result.

Although not shown, the server 499 can control the door opening and starting functions of the vehicle to be activated when the second user's terminal 490 transmits the second image, that is, the additional damage described above. Before the determination and consent procedure for , the second user's terminal 490 may activate the door opening and starting functions of the vehicle. In this case, in the process of driving the vehicle, the server 499 may proceed with the additional damage determination and consent procedure, and proceed with the additional damage determination and consent procedure until the operation of the vehicle is terminated. You can end the return process. Alternatively, the server 499 may perform the additional damage determination and consent procedure after the vehicle return procedure. The server 499 has the effect of simplifying the procedures before starting the vehicle by controlling the door opening and starting functions of the vehicle to be activated when the second user's terminal 490 transmits the second image. there is.

When consent to the determination result is inputted by the second user, the server 499 may store information indicating that the exterior state of the vehicle is the same as before.

Also, when the second user does not agree with the determination result, the second user's terminal 490 may display a display so that the second user can raise an objection to the reason for not agreeing with the determination result. Thereafter, when the second user's objection is completed, the second user's terminal 490 may activate the door opening and starting functions of the vehicle.

When an objection to the determination result is input by the second user, the server 499 may store information indicating that the exterior state of the vehicle is the same as before or additional damage has occurred according to the objection result. In other words, when the objection to the determination result input by the second user is accepted, the server 499 may store information indicating that additional damage has occurred in the exterior state of the vehicle, and the objection is not accepted. If not, information indicating that the exterior state of the vehicle is the same as the existing one may be stored.

In addition, the server 499 may update the vehicle damage information based on the existing damage information in which the user's consent and/or objection information received from the terminal 490 is reflected. In this case, the updated vehicle damage information may be defined as second information.

Illustratively, the second user's terminal 490 is the vehicle when the second user agrees that the existing damage information corresponds to the second image captured at the third time point T3, which is the start point of the second user's use. You can activate the door opening and starting function of the vehicle, and if you do not agree, you can activate the vehicle's door opening and starting function when the objection procedure is completed.

Although not shown, the server 499 can control the door opening and starting functions of the vehicle to be activated after the objection procedure by the second user's terminal 490 and the processing procedure therefor are completed, that is, The server 499 may activate the door opening and starting function of the vehicle after the second user's terminal 490 has completed an objection procedure and all consent procedures therefor. The server 499 controls the door opening and starting functions of the vehicle to be activated after the objection procedure by the second user's terminal 490 and the processing procedure therefor are completed, thereby preventing damage to the vehicle that may occur later. There is an effect that can prevent related disputes. In this way, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, so that damage information occurring on the exterior of the vehicle can be managed in a time-series manner, In addition, it is possible to make the repair, inspection, and replacement cost request smoothly by clarifying the location of responsibility for the damage information generated on the exterior of the vehicle.

In addition, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, so that the shared vehicle user clarifies the damage information that has already occurred before using the vehicle. You can make it clear that you are not responsible. Furthermore, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, so that the shared vehicle user can be held responsible only for damage that occurs additionally while using his or her vehicle, and the manager can also damage the vehicle Responsibilities can be clearly identified.

In addition, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, so that the manager can grasp the damage information that occurred when the vehicle is parked without being used by the user, and respond accordingly and action may be possible. For example, referring to FIG. 56 , the server 499 updates the vehicle damage information based on the information obtained from the terminal 490 , so that the manager can use the second time point T2 and the second time point T2, which is the end point of the first user's use. Damage information that occurred between the third time point T3, which is the start time of 2 users, that is, damage information that occurred in the second section P2 may be grasped, and corresponding responses and measures may be taken.

On the other hand, as described above, the shared vehicle user may not agree with all of the current appearance state of the shared vehicle, and may not partially agree. That is, the shared vehicle user can partially indicate whether or not to agree to the existing damage information of the shared vehicle.

59 is a view for explaining a process of re-photographing the appearance of the shared vehicle when the shared vehicle user does not agree with the existing damage information of the shared vehicle. Hereinafter, a process of re-photographing the exterior of the shared vehicle when the shared vehicle user does not agree with the existing damage information of the shared vehicle will be described with reference to FIG. 59 .

Referring to (a) of FIG. 59 , when the second user does not agree with all of the existing damage information of the shared vehicle, the second user's terminal 490 displays the entire exterior of the shared vehicle to be re-photographed. can do. In this case, the second user's terminal 490 may output a rephotographing guide that guides all major parts of the exterior of the vehicle to be photographed. The re-photography guide may be provided based on information input by the second user during reservation or information obtained through a learned neural network model.

Referring to (b) of Figure 59, when the second user does not partially agree with the existing damage information of the shared vehicle, the second user's terminal 490 guides the re-photographing of the part that does not agree. can The second user's terminal 490 may display a rephotographing guide corresponding to a part the second user does not agree with. As the second user proceeds to take a picture according to the re-shooting guide, the second user terminal 490 may obtain an image for a part to which the user does not agree. The re-photography guide may be provided based on information input by the shared vehicle user when making a reservation or information obtained through a learned neural network model. Through this process, the second user's terminal 490 may store the image of the part that does not agree and transmit it to the server 499 . The second user's terminal 490 may simplify the re-photographing procedure by controlling the re-photography to be performed only for the part the second user does not agree with.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (18)

an output unit for outputting a damaged image; and
A control unit for controlling the output unit to output the damaged image based on a first image photographed at a first time point by the camera and a second image photographed at a second time point by the camera,
In the first image, a first area related to the exterior of the vehicle is photographed, and the inside of the first area includes at least one fixed area and a variable area, which are at least one or more areas,
In the second image, a second region related to the exterior of the vehicle is photographed, and at least a partial region of the fixed region and at least a partial region of the variable region are included in the second region,
The damaged image is an image in which the variable region is removed from the first image and the second image,
The fixed region is a region for damage to the vehicle, and the variable region is a region that is not a region for damage to the vehicle,
The first time point and the second time point are different time points,
Fleet Management Device.
The method of claim 1,
The first image and the second image include a first feature point among a plurality of feature points related to the vehicle,
The first feature point of the first image has a first coordinate,
The first feature point of the second image has a second coordinate,
Coordinates representing the fixed area included in the first image are third coordinates,
The coordinates representing the fixed area included in the second image are the fourth coordinates,
A first distance between the first coordinates and the third coordinates and a second distance between the second coordinates and the fourth coordinates correspond to each other,
Fleet Management Device.
3. The method of claim 2,
Coordinates representing the variable region included in the first image are fifth coordinates,
The coordinates representing the variable region included in the second image are the sixth coordinates,
a third distance between the first coordinate and the fifth coordinate is different from a fourth distance between the second coordinate and the sixth coordinate;
Fleet Management Device.
4. The method of claim 3,
a difference between the first distance and the second distance is less than a difference between the third distance and the fourth distance;
Fleet Management Device.
According to claim 1,
The control unit classifies a fixed region and a variable region in the abnormal region, and outputs the damaged image from which the variable region is removed,
The first image and the second image include a first feature point among a plurality of feature points related to the vehicle,
The first feature point of the first image has a first coordinate,
The first feature point of the second image has a second coordinate,
The coordinates representing the abnormal region of the first image are the seventh coordinates,
The coordinates representing the abnormal region of the second image are the eighth coordinates,
The control unit classifies the fixed region and the variable region based on a fifth distance between the first coordinate and the seventh coordinate and a sixth distance between the second coordinate and the eighth coordinate,
Fleet Management Device.
6. The method of claim 5,
When the difference between the fifth distance and the sixth distance is greater than or equal to a predetermined value, the controller determines that the region is a variable region, and when the difference between the fifth distance and the sixth distance is less than a predetermined value, determines that the region is a fixed region;
Fleet Management Device.
According to claim 1,
The variable region is a background region outside the vehicle,
Fleet Management Device.
According to claim 1,
The variable region is a region included in the image by reflecting an external object by external light.
Fleet Management Device.
According to claim 1,
The damaged image is an image in which the variable region is removed from the first image,
The first image is an image in which the vehicle is located closer to the central region of the image compared to the second image,
Fleet Management Device.
According to claim 1,
Further comprising a communication unit for receiving a signal from the sensor unit,
The sensor unit includes a first sensor and a second sensor,
The first sensor is located in an area close to the vehicle's entry direction, and the second sensor is located in an area far from the vehicle's entry direction,
Fleet Management Device.
11. The method of claim 10,
The first time point and the second time point are time points within the validity period between the start time and the end time,
The start time is the time when the communication unit receives the first signal from the first sensor,
The end time is a time when the communication unit receives the second signal from the second sensor.
wherein the first image and the second image are included in a plurality of images acquired within the validity period;
Fleet Management Device.
12. The method of claim 11,
The starting time is a time when the vehicle is detected by the first sensor and then the detection of the vehicle is finished,
The end time is a time when the vehicle is first detected by the second sensor,
Fleet Management Device.
12. The method of claim 11,
The starting time is a time when the vehicle is first detected by the first sensor,
The end time is a time at which the detection is terminated after the vehicle is detected by the second sensor,
Fleet Management Device.
According to claim 1,
Further comprising a communication unit for receiving the open information from the breaker,
The first time point and the second time point are time points within the validity period between the start time and the end time,
The time when the breaker is opened is the end time, and the time before the breaker is opened by a predefined amount is the start time,
wherein the first image and the second image are included in a plurality of images acquired within the validity period;
Fleet Management Device.
11. The method of claim 10,
The camera comprises a first camera and a second camera,
The first camera is located in an area corresponding to the first sensor to photograph one side of the vehicle,
The second camera is positioned in an area corresponding to the second sensor to photograph the other side of the vehicle,
Fleet Management Device.
According to claim 1,
The first image and the second image include a first feature point among a plurality of feature points related to the vehicle,
At the first time point, the camera is located at a position having a first distance from the first feature point,
At the second time point, the camera is located at a position having a second distance from the first feature point,
the first distance and the second distance are different;
Fleet Management Device.
According to claim 1,
The vehicle management apparatus is a mobile device,
The camera is a camera built into the mobile device,
Fleet Management Device.
17. The method of claim 16,
The camera shoots a video between the start time and the end time,
The first image and the second image are images extracted from the video,
Fleet Management Device.

Images