KR20180023303A - Moving robot and control method thereof - Google Patents

Abstract

According to one embodiment of the present invention, a moving robot comprises: a driving unit for moving a main body; an image acquiring unit for acquiring several images by sequentially capturing around the main body; a storage unit for storing the sequential images acquired by the image acquiring unit; a sensor unit having at least one sensor for sensing an obstacle while moving; and a controlling unit having an obstacle recognizing module. When the sensor unit detects the obstacle, the obstacle recognizing module can select an image of certain time before when sensing the obstacle among a plurality of the sequential images based on the movement direction and the movement speed of the main body. Moreover, the obstacle recognizing module can recognize an obstacle property included in the selected image of the certain time based on data pre-learned by a machine learning. Therefore, image data for increasing precision of recognition about the obstacle property can be acquired. Thus, the obstacle property can be precisely recognized.

Description

[0001] Moving robot and control method [0002]

The present invention relates to a mobile robot and a control method thereof, and more particularly, to a mobile robot performing obstacle recognition and avoidance based on machine learning and a control method thereof.

Robots have been developed for industrial use and have been part of factory automation. In recent years, medical robots, aerospace robots, and the like have been developed, and household robots that can be used in ordinary homes are being developed. Among these robots, mobile robots capable of traveling by magnetic force are called mobile robots.

A typical example of a mobile robot used in the home is a robot cleaner. The robot cleaner is a device for cleaning a corresponding area by suctioning dust or foreign matter around the robot while running a certain area by itself.

The mobile robot is capable of moving by itself, is free to move, and is equipped with a plurality of sensors for avoiding obstacles during running, and can travel without obstacles.

In general, an infrared sensor or an ultrasonic sensor is used to detect an obstacle of a mobile robot. The infrared sensor senses the presence and distance of the obstacle through the amount of reflected light or the time of reflected light reflected by the obstacle, and when the ultrasonic sensor emits an ultrasonic wave having a predetermined period and there is an ultrasonic wave reflected by the obstacle, And the distance between the obstacle and the obstacle is determined using the time difference between the moment when the obstacle is reflected and the time when the obstacle is reflected.

On the other hand, obstacle recognition and avoidance have a great influence on the cleaning performance as well as the running performance of the mobile robot, and therefore it is required to secure the reliability of the obstacle recognition capability.

The prior art (Patent Registration No. 10-0669892) discloses a technology for implementing a reliable obstacle recognition technology by combining an infrared sensor and an ultrasonic sensor.

However, the prior art (Patent Registration No. 10-0669892) has a problem that it can not discriminate the attribute of the obstacle.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram for explaining an obstacle detection and avoidance method of a conventional mobile robot.

Referring to FIG. 1, the robot cleaner sucks dust and foreign substances while moving and performs cleaning (S11).

The ultrasonic sensor senses the ultrasonic signal reflected by the obstacle and recognizes the existence of the obstacle (S12). It is determined whether the height of the obstacle exceeds the height of the recognized obstacle (S13).

If it is determined that the robot cleaner is high enough, the robot cleaner moves straight (S14). Otherwise, the robot cleaner rotates the robot cleaner 90 degrees (S15).

For example, if the obstacle is a low threshold. The robot cleaner recognizes the threshold, and if the robot cleaner judges that the recognition result can pass, it moves beyond the threshold.

However, if the obstacle judged to be high enough is a wire, the robot cleaner may pass over the wire and be caught by the wire to be restrained.

In addition, since the pedestal of the fan has a height similar to or lower than the threshold, it can be judged that the robot cleaner is an obstacle to overcome. In this case, the robot cleaner rides on the pedestal of the fan, and the wheels can be restrained while idling.

In addition, if the entire human is not recognized and only a part of the human hair such as hair is detected, it can be determined that the human body is over the human body and straight forward. In this case, however, the robot cleaner can suck human hair, have.

Therefore, it is necessary to grasp the property of the front obstacle and to change the movement pattern according to the property.

On the other hand, interest in machine learning such as artificial intelligence and deep running has greatly increased recently.

Conventional machine learning was centered on statistical based classification, regression, and cluster models. In particular, in the learning of classification and regression model learning, a learning model that distinguishes the characteristics of learning data and new data based on these characteristics was defined by a person in advance. Deep learning, on the other hand, is one in which a computer identifies and identifies its own characteristics.

One of the factors that accelerated the development of deep learning is the deep-running framework that is provided as open source. For example, the deep learning frameworks include Theano at the University of Montreal, Toronto, Torch at New York University, Caffe at the University of California at Berkeley, and TensorFlow at Google.

With the release of deep learning frameworks, in addition to deep learning algorithms, extraction and selection of data to apply to learning processes, learning methods, and learning are becoming more important for effective learning and recognition.

Also, researches are being conducted to utilize artificial intelligence and machine learning in various products and services.

Patent Registration No. 10-0669892 (registered on January 10, 2007)

An object of the present invention is to provide a mobile robot and a control method thereof capable of acquiring image data capable of improving the accuracy of obstacle attribute recognition.

It is an object of the present invention to provide a mobile robot and its control method capable of performing obstacle recognition and avoidance operations by determining an attribute of an obstacle and adjusting a traveling pattern according to an obstacle property.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a mobile robot capable of improving the stability of the mobile robot itself and the convenience of the user by improving the operation efficiency and cleaning efficiency by performing actions such as forward, And a control method thereof.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a mobile robot and a control method thereof that can accurately recognize an attribute of an obstacle based on machine learning.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a mobile robot and a control method thereof that can efficiently perform machine learning and extract data that can be used for obstacle attribute recognition.

According to one aspect of the present invention, there is provided a mobile robot including a traveling unit for moving a main body, an image acquisition unit for continuously photographing the periphery of the main body to acquire a plurality of images, A sensor unit including a storage unit for storing a plurality of images, a sensor unit including at least one sensor for detecting obstacles during movement, and a sensor unit for detecting an obstacle, based on a moving direction and a moving speed of the main body, An obstacle recognition module that selects a specific viewpoint image prior to the obstacle detection point of the sensor unit and recognizes an attribute of the obstacle included in the selected point-in-time image based on data learned through machine learning; By acquiring the image data that can improve the accuracy of obstacle attribute recognition, Can.

In order to achieve the above and other objects, a mobile robot according to an aspect of the present invention includes a travel control module that controls driving of a travel part based on a recognized property of an obstacle, Efficiency and cleaning efficiency can be improved.

According to another aspect of the present invention, there is provided a method for controlling a mobile robot, comprising: acquiring a plurality of images by continuously photographing a periphery of a main body during movement through an image acquisition unit; Detecting an obstacle through the sensor unit, detecting, based on the moving direction and the moving speed of the main body, detecting the obstacle of the sensor unit from the plurality of consecutive images, Selecting a previous point-in-time image; recognizing an attribute of an obstacle included in the point-in-time image selected based on the data learned in the machine learning; And controlling the driving of the driving part.

According to at least one of the embodiments of the present invention, it is possible to acquire image data that can improve the accuracy of obstacle attribute recognition.

According to at least one of the embodiments of the present invention, the mobile robot can determine the nature of the obstacle and adjust the traveling pattern according to the obstacle property, thereby performing highly reliable obstacle recognition and avoidance operations.

According to at least one of the embodiments of the present invention, it is possible to improve the stability of the mobile robot itself and the user's convenience by performing operations such as forward, backward, stop, and bypass according to the recognition result of the obstacle, And a control method thereof can be provided.

In addition, according to at least one embodiment of the present invention, it is possible to provide a mobile robot and a control method thereof that can accurately recognize an attribute of an obstacle based on machine learning.

In addition, according to at least one of the embodiments of the present invention, the mobile robot can efficiently perform machine learning and extract data that can be used for obstacle attribute recognition.

Meanwhile, various other effects will be directly or implicitly disclosed in the detailed description according to the embodiment of the present invention to be described later.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram for explaining an obstacle detection and avoidance method of a conventional mobile robot.
2 is a perspective view illustrating a mobile robot and a charging base for charging the mobile robot according to an embodiment of the present invention.
FIG. 3 is a top view of the mobile robot shown in FIG. 2. FIG.
FIG. 4 is a front view of the mobile robot shown in FIG. 2. FIG.
FIG. 5 is a view showing a bottom portion of the mobile robot shown in FIG. 2. FIG.
FIG. 6 and FIG. 7 are block diagrams showing control relationships among main components of a mobile robot according to an embodiment of the present invention.
Figure 8 is an illustration of a simplified internal block diagram of a server in accordance with an embodiment of the present invention.
9 to 12 are diagrams referred to in explanation of Deep Learning.
13 and 14 are diagrams referred to in explaining the obstacle recognition.
15 is a flowchart illustrating a method of controlling a mobile robot according to an embodiment of the present invention.
16 is a flowchart illustrating a method of controlling a mobile robot according to an embodiment of the present invention.
17 to 25 are diagrams referred to in explanation of the control method of the mobile robot according to the embodiment of the present invention.
26 is a diagram for explaining a method of operating a mobile robot and a server according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it is needless to say that the present invention is not limited to these embodiments and can be modified into various forms.

In the drawings, the same reference numerals are used for the same or similar parts throughout the specification.

The suffix "module" and " part "for components used in the following description are given merely for convenience of description and do not give special significance or role in themselves. Accordingly, the terms "module" and "part" may be used interchangeably.

The mobile robot 100 according to an embodiment of the present invention refers to a robot that can move by itself using wheels or the like, and may be a home helper robot and a robot cleaner. Hereinafter, a robot cleaner having a cleaning function of a mobile robot will be described with reference to the drawings, but the present invention is not limited thereto.

2 is a perspective view illustrating a mobile robot and a charging base for charging the mobile robot according to an embodiment of the present invention.

FIG. 3 is a view illustrating a top portion of the mobile robot shown in FIG. 2. FIG. 4 is a front view of the mobile robot shown in FIG. 2. FIG. It is a degree.

FIG. 6 and FIG. 7 are block diagrams showing control relationships among main components of a mobile robot according to an embodiment of the present invention.

3 to 7, the mobile robots 100, 100a, and 100b include a main body 110 and image acquisition units 120, 120a, and 120b that acquire images around the main body 110. FIG.

Hereinafter, in defining each portion of the main body 110, a portion facing the ceiling in the running zone is defined as an upper surface portion (see Fig. 3), and a portion facing the floor in the running zone is defined as a bottom portion And a portion of the portion of the periphery of the main body 110 facing the running direction between the upper surface portion and the bottom surface portion is defined as a front surface portion (see FIG. 4).

The mobile robots 100, 100a and 100b include a traveling unit 160 for moving the main body 110. [ The driving unit 160 includes at least one driving wheel 136 for moving the main body 110. The driving unit 160 includes a driving motor (not shown) connected to the driving wheels 136 to rotate the driving wheels. The driving wheels 136 may be provided on the left and right sides of the main body 110 and will be referred to as the left wheel 136 (L) and the right wheel 136 (R), respectively.

The left wheel 136 (L) and the right wheel 136 (R) may be driven by a single drive motor, but may be driven by a left wheel drive motor and a right wheel 136 (R) And a right wheel drive motor for driving the right wheel drive motor. The running direction of the main body 110 can be switched to the left or right side by making a difference in rotational speed between the left wheel 136 (L) and the right wheel 136 (R).

A suction port 110h for sucking in air may be formed on the bottom surface of the main body 110. A suction device for supplying suction force for sucking air through the suction port 110h is provided in the main body 110 And a dust container (not shown) for dust collecting with the air through the suction port 110h.

The main body 110 may include a case 111 forming a space for accommodating various components constituting the mobile robots 100, 100a, and 100b. An opening for inserting and removing the dust container may be formed in the case 111, and a dust container cover 112 for opening and closing the opening may be rotatably provided with respect to the case 111.

An auxiliary brush 135 having a brush having a plurality of blades extending radially and located on the front side of the bottom surface of the main body 110, May be provided. By the rotation of the brushes 134 and 135, the dusts are separated from the floor in the traveling zone, and the dusts separated from the floor are sucked through the suction port 110h and collected in the dustbin.

The battery 138 supplies not only the drive motor but also the power required for the overall operation of the mobile robots 100, 100a and 100b. When the battery 138 is discharged, the mobile robots 100, 100a, and 100b can travel to return to the charging base 200 for charging. During the return travel, the mobile robots 100, 100a, Can detect the position of the charging stand 200 by itself.

The charging stand 200 may include a signal transmitting unit (not shown) for transmitting a predetermined return signal. The return signal may be an ultrasonic signal or an infrared signal, but is not limited thereto.

The mobile robots 100, 100a and 100b may include a signal sensing unit (not shown) for receiving a return signal. The charging base 200 may transmit an infrared signal through a signal transmitting unit, and the signal sensing unit may include an infrared sensor that senses an infrared signal. The mobile robots 100, 100a and 100b move to the position of the charging base 200 according to the infrared signal transmitted from the charging base 200 and dock the charging base 200. [ The charging is performed between the charging terminal 133 of the mobile robots 100, 100a and 100b and the charging terminal 210 of the charging stand 200 by such docking.

The image acquiring unit 120 photographs a driving area, and may include a camera module. The camera module may include a digital camera. A digital camera includes an image sensor (e.g., a CMOS image sensor) configured with at least one optical lens, and a plurality of photodiodes (e.g., pixels) that are formed by light passing through the optical lens, And a digital signal processor (DSP) that forms an image based on signals output from the photodiodes. The digital signal processor is capable of generating moving images composed of still frames as well as still images.

The image acquisition unit 120 may include a front camera 120a provided to acquire an image of the front of the main body 110 and an image acquisition unit 120b provided on the upper surface of the main body 110 to acquire images of the ceiling And the upper camera 120b, but the position and the photographing range of the image obtaining unit 120 are not necessarily limited thereto.

In the case of this embodiment, a camera is provided at a part (ex, front, rear, bottom) of the mobile robot, and the captured image can be continuously acquired at the time of cleaning. Several cameras may be installed for each part of the camera for photographing efficiency. The image captured by the camera can be used to identify the kind of material such as dust, hair, floor, etc. present in the space, whether it is cleaned, or to confirm the cleaning time.

The front camera 120a can photograph an obstacle present in the traveling direction of the mobile robots 100, 100a and 100b or a situation of the cleaning area.

According to an embodiment of the present invention, the image acquiring unit 120 may acquire a plurality of images by continuously photographing the periphery of the main body 110, and the obtained plurality of images may be stored in the storage unit 150 .

The mobile robots 100, 100a and 100b can improve accuracy of obstacle recognition by using a plurality of images, or by selecting one or more images from a plurality of images and using effective data.

In addition, the mobile robots 100, 100a, and 100b may include a sensor unit 170 including sensors for sensing various data related to the operation and state of the mobile robot.

For example, the sensor unit 170 may include an obstacle detection sensor 131 for sensing an obstacle ahead. The sensor unit 170 may further include a cliff detection sensor 132 for detecting the presence or absence of a cliff on the floor in the driving area and a lower camera sensor 139 for acquiring a bottom image.

 Referring to FIGS. 2 and 4, the obstacle detection sensor 131 may include a plurality of sensors installed on the outer circumferential surface of the mobile robot 100 at regular intervals.

For example, the sensor unit 170 may include a first sensor disposed on the front surface of the main body 110, a second sensor disposed left and right from the first sensor, and a third sensor .

The obstacle detection sensor 131 may include an infrared sensor, an ultrasonic sensor, an RF sensor, a geomagnetic sensor, a position sensitive device (PSD) sensor, and the like.

Meanwhile, the position and type of the sensor included in the obstacle detection sensor 131 may vary depending on the type of the mobile robot, and the obstacle detection sensor 131 may include more various sensors.

The obstacle detection sensor 131 is a sensor for detecting a distance to a wall or an obstacle in a room, and the present invention is not limited to this type, but an ultrasonic sensor will be described below as an example.

The obstacle detection sensor 131 senses an object, particularly an obstacle, existing in a traveling direction (movement direction) of the mobile robot, and transmits the obstacle information to the control unit 140. That is, the obstacle detection sensor 131 can sense the moving path of the mobile robot, the protrusions present on the front or side of the robot, the furniture of the house, the furniture, the wall, the wall edge, and the like and transmit the information to the control unit.

At this time, the controller 140 detects the position of the obstacle based on at least two signals received through the ultrasonic sensor, and controls the movement of the mobile robot 100 according to the position of the detected obstacle.

According to an embodiment, the obstacle detection sensor 131 provided on the outer surface of the case 110 may include a transmitter and a receiver.

For example, the ultrasonic sensor may be provided such that at least one transmitting portion and at least two receiving portions are staggered from each other. Accordingly, it is possible to radiate signals at various angles and to receive signals reflected from the obstacles at various angles.

According to an embodiment, the signal received by the obstacle detection sensor 131 may undergo signal processing such as amplification, filtering, and the like, and then the distance and direction to the obstacle may be calculated.

The sensor unit 170 may further include a motion detection sensor for sensing motion of the mobile robot 100, 100a, and 100b according to driving of the main body 110, and outputting motion information. As the motion detection sensor, a gyro sensor, a wheel sensor, an acceleration sensor, or the like can be used.

The gyro sensor detects the rotation direction and detects the rotation angle when the mobile robots 100, 100a, 100b move according to the operation mode. The gyro sensor detects the angular velocity of the mobile robots 100, 100a and 100b and outputs a voltage value proportional to the angular velocity. The control unit 140 calculates the rotation direction and the rotation angle using the voltage value output from the gyro sensor.

The wheel sensor is connected to the left wheel 136 (L) and the right wheel 136 (R) to sense the number of revolutions of the wheel. Here, the wheel sensor may be a rotary encoder. The rotary encoder senses and outputs the number of rotations of the left wheel 136 (L) and the right wheel 136 (R).

The control unit 140 can calculate the rotational speeds of the left and right wheels using the number of rotations. The control unit 140 can calculate the rotation angle using the difference in the number of revolutions of the left wheel 136 (L) and the right wheel 136 (R).

The acceleration sensor senses a change in the speed of the mobile robots 100, 100a, 100b, for example, changes in the mobile robots 100, 100a, 100b due to start, stop, change of direction or collision with an object. The acceleration sensor is attached to the main wheel or the adjoining positions of the auxiliary wheels, so that the slip or idling of the wheel can be detected.

In addition, the acceleration sensor is embedded in the control unit 140 and can detect a speed change of the mobile robots 100, 100a, and 100b. That is, the acceleration sensor detects the amount of impact according to the speed change and outputs a corresponding voltage value. Thus, the acceleration sensor can perform the function of an electronic bumper.

The control unit 140 can calculate the positional change of the mobile robots 100, 100a, and 100b based on the motion information output from the motion detection sensor. Such a position is a relative position corresponding to the absolute position using the image information. The mobile robot can improve the performance of the position recognition using the image information and the obstacle information through the relative position recognition.

Meanwhile, the mobile robots 100, 100a, and 100b may include a power supply unit (not shown) that includes a rechargeable battery 138 to supply power to the robot cleaner.

The power supply unit supplies driving power and operating power to the respective components of the mobile robots 100, 100a and 100b, and can be charged by receiving a charging current from the charging stand 200 when the remaining power is insufficient.

The mobile robots 100, 100a and 100b may further include a battery sensing unit (not shown) for sensing the state of charge of the battery 138 and transmitting the sensing result to the control unit 140. [ The battery 138 is connected to the battery sensing unit, and the battery remaining amount and charging state are transmitted to the control unit 140. The remaining battery level can be displayed on the screen of the output unit (not shown).

In addition, the mobile robots 100, 100a, and 100b include an operation unit 137 that can turn on / off or input various commands. Various control commands necessary for the overall operation of the mobile robot 100 can be inputted through the operation unit 137. [ In addition, the mobile robots 100, 100a, and 100b can display reservation information, a battery state, an operation mode, an operation state, an error state, and the like, including an output unit (not shown).

Referring to FIGS. 6 and 7, the mobile robots 100a and 100b include a controller 140 for processing and determining various information such as recognizing a current position, and a storage unit 150 for storing various data. In addition, the mobile robots 100, 100a and 100b may further include a communication unit 190 for transmitting and receiving data to / from an external terminal.

The external terminal is provided with an application for controlling the mobile robots 100a and 100b and displays a map of the traveling area to be cleaned by the mobile robots 100a and 100b through execution of the application and cleans a specific area on the map The area can be specified. The external terminal may be, for example, a remote controller, a PDA, a laptop, a smart phone, or a tablet on which an application for setting a map is mounted.

The external terminal can communicate with the mobile robots 100a and 100b to display the current position of the mobile robot together with the map, and information about a plurality of areas can be displayed. Further, the external terminal updates its position and displays it according to the traveling of the mobile robot.

The control unit 140 controls the overall operation of the mobile robot 100 by controlling the image acquisition unit 120, the operation unit 137 and the travel unit 160 constituting the mobile robots 100a and 100b.

The storage unit 150 records various types of information required for controlling the mobile robot 100, and may include a volatile or nonvolatile recording medium. The storage medium stores data that can be read by a microprocessor, and includes a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD- Tape, floppy disk, optical data storage, and the like.

Also, the storage unit 150 may store a map of the driving area. The map may be input by an external terminal, a server or the like capable of exchanging information with the mobile robots 100a and 100b through wired or wireless communication, or may be one generated by self-learning by the mobile robots 100a and 100b have.

The map can display the location of the rooms in the driving area. In addition, the current position of the mobile robots 100a, 100b can be displayed on the map, and the current position of the mobile robots 100a, 100b on the map can be updated in the course of travel. The external terminal stores the same map as the map stored in the storage unit 150.

The storage unit 150 may store cleaning history information. Such cleaning history information can be generated each time cleaning is performed.

The map of the driving area stored in the storage unit 150 may be a navigation map used for traveling during cleaning, a simultaneous localization and mapping (SLAM) map used for position recognition, A learning map for storing information to be used for learning cleaning, a global location map used for global location recognition, an obstacle recognition map for recording information on recognized obstacles, and the like.

Meanwhile, although the maps can be separately stored and managed in the storage unit 150 according to the usage as described above, the maps may not be clearly classified for each use. For example, a plurality of pieces of information may be stored in one map for use in at least two applications.

The control unit 140 may include a travel control module 141, a location recognition module 142, a map generation module 143, and an obstacle recognition module 144.

3 to 7, the travel control module 141 controls traveling of the mobile robots 100, 100a, and 100b, and controls driving of the travel unit 160 according to the travel setting. In addition, the travel control module 141 can grasp the travel paths of the mobile robots 100, 100a, and 100b based on the operation of the travel unit 160. [ For example, the driving control module 141 can grasp the current or past traveling speed and the traveling distance of the mobile robot 100 based on the rotational speed of the driving wheel 136, L), 136 (R)), the current or past direction change process can be grasped. The positions of the mobile robots 100, 100a, and 100b on the map can be updated based on the travel information of the mobile robots 100, 100a, and 100b thus identified.

The map generating module 143 may generate a map of the running area. The map generation module 143 can process the image acquired through the image acquisition unit 120 to generate a map. That is, a cleaning map corresponding to the cleaning area can be created.

Also, the map generation module 143 can process the image acquired through the image acquisition unit 120 at each position and recognize the global position in association with the map.

The position recognition module 142 estimates and recognizes the current position. The position recognition module 142 grasps the position in association with the map generation module 143 using the image information of the image acquisition unit 120 so that even when the positions of the mobile robots 100, 100a, 100b suddenly change The current position can be estimated and recognized.

The mobile robots 100, 100a and 100b are capable of recognizing the position during the continuous running through the position recognition module 142 and can also recognize the map generation module 143 and the obstacle recognition module 144 without the position recognition module 142 , You can learn the map and estimate your current location.

The image acquisition unit 120 acquires images of the surroundings of the mobile robot 100 while the mobile robots 100, 100a, and 100b are traveling. Hereinafter, the image acquired by the image acquisition unit 120 is defined as an 'acquired image'.

The acquired image includes various features such as lights, edges, corners, blobs, ridges, etc., located on the ceiling.

The map generation module 143 detects the feature from each of the acquired images. Various methods for detecting features from an image in the field of Computer Vision (Feature Detection) are well known. Several feature detectors suitable for detecting these features are known. For example, Canny, Sobel, Harris & Stephens / Plessey, SUSAN, Shi & Tomasi, Level curve curvature, FAST, Laplacian of Gaussian, Difference of Gaussians, Determinant of Hessian, MSER, PCBR and Gray-level blobs detector.

The map generation module 143 calculates a descriptor based on each minutiae point. The map generation module 143 may convert a feature point into a descriptor using a Scale Invariant Feature Transform (SIFT) technique for feature detection. The descriptor may be denoted by an n-dimensional vector.

The SIFT can detect unchanging features with respect to changes in scale, rotation, and brightness of an object to be photographed. Even if the same region is photographed with a different attitude of the mobile robot 100 (i.e., -invariant) can be detected. Of course, various other techniques (e.g., HOG: Histogram of Oriented Gradient, Haar feature, Fems, Local Binary Pattern (LBP), Modified Census Transform (MCT)) may be applied.

The map generation module 143 classifies at least one descriptor for each acquired image into a plurality of groups according to a predetermined lower classification rule on the basis of the descriptor information obtained through the acquired image of each position, Can be converted into lower representative descriptors, respectively.

As another example, it is also possible to classify all descriptors gathered from acquired images in a predetermined area, such as a room, into a plurality of groups according to a predetermined sub-classification rule, and write descriptors included in the same group according to the predetermined lower representative rule, . ≪ / RTI >

The map generation module 143 can obtain the feature distribution of each position through such a process. Each position feature distribution can be represented as a histogram or an n-dimensional vector. As another example, the map generation module 143 can estimate an unknown current position based on descriptors calculated from each feature point, without going through a predetermined lower classification rule and a predetermined lower representative rule.

In addition, when the current position of the mobile robots 100, 100a, 100b is in an unknown state due to a positional jump or the like, the current position can be estimated based on data such as a pre-stored descriptor or a lower representative descriptor.

The mobile robots 100, 100a, and 100b acquire an acquired image through the image acquisition unit 120 at an unknown current position. Through the image, various features such as lights, edges, corners, blobs, ridges, etc., are found on the ceiling.

The position recognition module 142 detects the features from the acquired image. Various methods of detecting features from an image in the field of computer vision technology and descriptions of various feature detectors suitable for detecting these features are as described above.

The position recognition module 142 calculates the recognition descriptor through the recognition descriptor calculation step based on each recognition feature point. Here, the recognition minutiae and the recognition descriptor are used to describe the process performed by the obstacle recognition module 144, which is to distinguish the terms used to describe the process performed by the map generation module 143. [ However, the characteristics of the outside world of the mobile robots 100, 100a, and 100b are merely defined by different terms.

The location recognition module 142 may convert the recognized minutiae point into the recognition descriptor using the Scale Invariant Feature Transform (SIFT) technique for detecting the feature. The recognition descriptor may be denoted by an n-dimensional vector.

As described above, the SIFT selects characteristic points that are easily distinguishable, such as corner points, from the acquired image, and then determines the distribution characteristics of the brightness gradient of pixels belonging to a certain region around each characteristic point ) Is an image recognition technique that obtains an n-dimensional vector (vector) in which the degree of change in each direction is a numerical value for each dimension.

Based on at least one recognition descriptor information obtained through an acquired image of an unknown current position, the position recognition module 142 generates position information (for example, feature distribution of each position) (Lower recognition feature distribution).

According to a predetermined lower comparison rule, each position feature distribution can be compared with each recognition feature distribution to calculate each similarity. The similarity degree (probability) is calculated for each position corresponding to each position, and the position where the greatest probability is calculated can be determined as the current position.

In this way, the control unit 140 can distinguish the driving zone, generate a map composed of a plurality of areas, or recognize the current position of the main body 110 based on the pre-stored map.

When the map is generated, the control unit 140 can transmit the generated map to an external terminal, a server, or the like through the communication unit 190. [ Also, as described above, the control unit 140 can store the map in the storage unit when the map is received from an external terminal, a server, or the like.

In addition, when the map is updated during travel, the controller 140 transmits the updated information to the external terminal so that the map stored in the external terminal and the mobile robots 100, 100a, and 100b is the same. As the map stored in the external terminal and the mobile robots 100, 100a, and 100b are kept the same, the mobile robot 100, 100a, and 100b can clean the designated area with respect to the cleaning command from the mobile terminal, So that the current position of the mobile robot can be displayed on the terminal.

At this time, the map is divided into a plurality of areas for the cleaning area, and includes a connection path for connecting the plurality of areas, and may include information about the obstacles in the area.

When the cleaning command is inputted, the control unit 140 determines whether or not the position on the map matches the current position of the mobile robot. The cleaning command can be input from the remote controller, the operation unit, or an external terminal.

If the current position does not coincide with the current position on the map or the current position can not be confirmed, the controller 140 recognizes the current position and restores the current position of the mobile robot 100, The traveling section 160 can be controlled to move to the designated area.

If the current position does not coincide with the position on the map or the current position can not be confirmed, the position recognition module 142 analyzes the acquired image input from the image obtaining unit 120 and estimates the current position based on the map can do. In addition, the obstacle recognition module 144 or the map generation module 143 may also recognize the current position in the same manner.

After the current position of the mobile robots 100, 100a, 100b is recognized and the driving control module 141 calculates the traveling route from the current position to the designated area and controls the traveling unit 160 to be the designated area Move.

In the case of receiving the cleaning pattern information from the server, the travel control module 141 can divide the entire traveling area into a plurality of areas and set one or more areas as designated areas according to the received cleaning pattern information.

Also, the travel control module 141 may calculate the travel route in accordance with the received cleaning pattern information, travel along the travel route, and perform cleaning.

The control unit 140 may store the cleaning history in the storage unit 150 when the cleaning of the designated area is completed.

The control unit 140 may transmit the operation state or the cleaning state of the mobile robot 100 to the external terminal and the server through the communication unit 190 at predetermined intervals.

Based on the received data, the external terminal displays the position of the mobile robot together with the map on the screen of the application being executed, and also outputs information on the cleaning state.

The mobile robots 100, 100a and 100b according to the embodiment of the present invention move until an obstacle or a wall surface is detected in one direction. When the obstacle recognition module 144 recognizes an obstacle, , And the like can be determined.

For example, if the recognized obstacle is an obstacle of a kind that can be passed over, the mobile robots 100, 100a, 100b can continue straight ahead. Alternatively, if the recognized obstacle is an obstacle of a kind that can not be surpassed, the mobile robots 100, 100a and 100b rotate and move a certain distance, move to a distance in which the obstacle is sensed in the opposite direction of the initial moving direction, Can travel in the form of

The mobile robots 100, 100a and 100b according to the embodiment of the present invention can perform obstacle recognition and avoidance based on machine learning.

The control unit 140 includes an obstacle recognition module 144 for recognizing an obstacle learned through machine learning in the input image and a driving unit for driving the driving unit 160 based on the recognized obstacle characteristics And a travel control module 141 for controlling the vehicle.

The mobile robots 100, 100a and 100b according to the embodiment of the present invention may include an obstacle recognition module 144 in which the attributes of obstacles are learned by machine learning.

Machine learning means that the computer learns from the data through the computer, and the computer takes care of the problem, even though the computer does not instruct the person directly with the logic.

Deep Learning (Deep Learning). It is based on Artificial Neural Networks (ANN) for constructing artificial intelligence. It is an artificial intelligence technology that allows a computer to learn like a human being without learning it as a way of teaching people how to think.

The ANN may be implemented in a software form or a hardware form such as a chip.

The obstacle recognition module 144 may include an artificial neural network (ANN) in the form of software or hardware in which the property of the obstacle is learned.

For example, the obstacle recognition module 144 may include a Deep Neural Network (DNN) such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), DBN (Deep Belief Network) . ≪ / RTI >

Deep learning will be described later in detail with reference to Figs. 9 to 12 and the like.

The obstacle recognition module 144 may determine an attribute of the obstacle included in the input image data based on the weights among the nodes included in the DNN.

If the sensor unit 170 senses an obstacle while the mobile robots 100, 100a and 100b are moving, the control unit 140 controls the image acquisition unit (not shown) based on the moving direction and the moving speed of the main body 110 120 may select a specific view image before the obstacle detection point of the sensor unit 170 among the plurality of consecutive images acquired by the sensor unit 170.

When the image acquisition unit 120 acquires an image using the obstacle detection point of the sensor unit 170 as a trigger signal, the obstacle is not included in the acquired image or is included small .

Therefore, an embodiment of the present invention can detect obstacles of the sensor unit 170 among a plurality of consecutive images acquired by the image acquisition unit 120, based on the moving direction and the moving speed of the main body 110 A point-in-time image before the point-in-time can be selected and used as data for obstacle recognition.

On the other hand, the obstacle recognition module 144 can recognize the attributes of the obstacle included in the selected point-in-time image based on the data learned through machine learning.

If the sensor unit 170 senses an obstacle while the mobile robots 100, 100a and 100b are moving, the control unit 140 controls the image acquisition unit (not shown) corresponding to the direction of the obstacle detected by the sensor unit 170 120 to extract a part of the image to be acquired.

The image acquisition unit 120, particularly the front camera 120a, can acquire images within a predetermined angle range in the moving direction of the mobile robots 100, 100a, and 100b.

The control unit 140 may not use the entire image acquired by the image acquisition unit 120, in particular, the front camera 120a, but may determine an attribute of an obstacle existing in a moving direction using only a part of the image.

6, the controller 140 may perform image processing for extracting a partial area of an image acquired by the image acquiring unit 120 corresponding to the direction of an obstacle sensed by the sensor unit 170 Module 145 may be further included.

7, the mobile robot 100b extracts a portion of the image acquired by the image acquisition unit 120 corresponding to the direction of the obstacle detected by the sensor unit 170 And may further include a separate image processing unit 125.

The mobile robot 100a according to the embodiment of FIG. 6 and the mobile robot 100b according to the embodiment of FIG. 7 have the same configuration except for the image processing module 145 and the image processing unit 125. FIG.

Alternatively, in some embodiments, the image acquisition unit 120 may directly extract a portion of the image, unlike the embodiments of FIGS.

The obstacle recognition module 144 learned by the machine learning has a high recognition rate as the learned object occupies a large portion of the input image data.

Accordingly, the present invention can enhance the recognition rate by extracting another region of the image acquired by the image acquisition unit 120 according to the direction of an obstacle detected by the sensor unit 170, such as an ultrasonic sensor, and using the extracted region as recognition data.

The obstacle recognition module 144 can recognize the obstacle based on the data learned by the machine learning in the extracted image.

The driving control module 141 may control driving of the driving unit 160 based on the recognized attributes of the obstacle.

If the obstacle is sensed in the right direction of the front surface of the main body, the control unit 140 extracts a lower right area of the image acquired by the image acquiring unit, detects the obstacle in the left direction of the front surface of the main body A lower left region of the image obtained by the image obtaining unit is extracted and a center lower region of the image obtained by the image obtaining unit is extracted when the obstacle is detected in the front direction of the main body can do.

In addition, the control unit 140 may control the extraction of the extraction target region in the image acquired by the image acquisition unit so as to correspond to the direction of the detected obstacle.

Meanwhile, the storage unit 150 may store input data for determining an obstacle attribute and data for learning the DNN.

The storage unit 150 may store an original image acquired by the image acquisition unit 120 and extracted images from which a predetermined region is extracted.

In addition, according to the embodiment, the weight and the biases constituting the DNN structure may be stored in the storage unit 150. [

Alternatively, according to an embodiment, the weight and the biases constituting the depth-based neural network structure may be stored in an embedded memory of the obstacle recognition module 144.

The obstacle recognizing module 144 may perform a learning process by using the extracted image as training data every time a part of an image acquired by the image acquiring unit 120 is extracted, After the extracted image is acquired, the learning process can be performed.

That is, the obstacle recognition module 144 updates the DNN structure such as a weight by adding a recognition result every time an obstacle is recognized, or acquires a predetermined number of training data The learning process can be performed with the training data to update the DNN structure such as the weight.

Alternatively, the mobile robots 100, 100a, and 100b may transmit the original image or the extracted image acquired by the image acquisition unit 120 to the predetermined server through the communication unit 190, and may receive data related to machine learning from the predetermined server Lt; / RTI >

In this case, the mobile robots 100, 100a, and 100b may update the obstacle recognition module 141 based on the data related to the machine learning received from the predetermined server.

Figure 8 is an illustration of a simplified internal block diagram of a server in accordance with an embodiment of the present invention.

8, the server 70 may include a communication unit 820, a storage unit 830, a learning module 840, and a processor 810.

The processor 810 can control the overall operation of the server 70. [

The server 70 may be a server operated by a home appliance manufacturer such as the mobile robots 100, 100a, 100b or a server operated by a service provider, or may be a kind of a cloud server.

The communication unit 820 can receive various data such as state information, operation information, operation information, and the like from a home appliance such as a mobile terminal, a mobile robot 100, 100a, and 100b, and a gateway.

The communication unit 820 can transmit data corresponding to the received various information to a home appliance such as a portable terminal, a mobile robot 100, 100a, or 100b, a gateway, and the like.

To this end, the communication unit 820 may include one or more communication modules such as an Internet module, a mobile communication module, and the like.

The storage unit 830 may store the received information and may include data for generating the corresponding result information.

In addition, the storage unit 830 may store data used for machine learning, result data, and the like.

The learning module 840 may serve as a learning appliance of the home appliances such as the mobile robots 100, 100a, and 100b.

The learning module 840 may include an artificial neural network such as a CNN (Convolutional Neural Network), a RNN (Recurrent Neural Network), a Deep Belief Network (DBN), and a Deep Neural Network (DNN) Neural networks can be learned.

As the learning method of the learning module 840, both unsupervised learning and supervised learning can be used.

Meanwhile, the controller 810 may control the artificial neural network structure of the home appliances such as the mobile robots 100, 100a, and 100b to be updated to the learned neural network structure after learning according to the setting.

9 to 12 are diagrams referred to in explanation of Deep Learning.

Deep Learning, a type of machine learning, is a multi-level, deep learning process based on data.

Deep learning can represent a set of machine learning algorithms that extract key data from multiple sets of data as they step up.

The deep learning structure may include an artificial neural network (ANN). For example, the deep learning structure may include a deep neural network (DNN) such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), DBN Be able to

Referring to FIG. 9, the ANN may include an input layer, a hidden layer, and an output layer. Each layer contains a plurality of nodes, and each layer is associated with the next layer. The nodes between adjacent layers can be connected to each other with a weight.

Referring to FIG. 10, a computer (machine) finds a certain pattern from inputted input data 1010 and forms a feature map. The computer (machine) can extract the low-level features 1020, the intermediate-level features 1030, and the high-level features 1040, recognize the objects, and output the results 1050.

The artificial neural network can be abstracted into higher-level features as it goes to the next layer.

9 and 10, each node may operate based on an activation model, and an output value corresponding to an input value may be determined according to an activation model.

The output value of any node, e.g., lower level feature 1020, may be input to the next layer associated with that node, e.g., a node of the intermediate level feature 1030. A node of the next layer, for example, a node of the intermediate level feature 1030, may receive values output from a plurality of nodes of the lower level feature 1020.

In this case, the input value of each node may be a value to which a weight is applied to the output value of the node of the previous layer. The weight can mean the link strength between nodes.

Also, the deep running process can be seen as a process of finding an appropriate weight.

On the other hand, the output value of any node, e.g., the intermediate level feature 1030, may be input to the next layer, e.g., the node of the higher level feature 1040, associated with that node. A node of the next layer, e.g., a node of the higher level feature 1040, may receive values output from a plurality of nodes of the intermediate level feature 1030.

The artificial neural network can extract feature information corresponding to each level using a learned layer corresponding to each level. The artificial neural network is sequentially abstracted, and a predetermined object can be recognized using the feature information of the highest level.

For example, if we look at the face recognition process by deep learning, the computer distinguishes between bright pixels and dark pixels according to the brightness of a pixel from the input image, distinguishes simple forms such as a border and an edge, You can distinguish things. Finally, the computer can grasp the form that defines the human face.

The deep running structure according to the present invention can utilize various known structures. For example, the deep learning structure according to the present invention may be a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Deep Belief Network (DBN), or the like.

Recurrent Neural Network (RNN) is widely used in natural language processing, and it is possible to construct an artificial neural network structure by stacking layers at each moment with an effective structure for time-series data processing that varies with time .

Deep Belief Network (DBN) is a deep-run structure consisting of multiple layers of deep-running RBM (Restricted Boltzman Machine). The Restricted Boltzman Machine (RBM) learning is repeated, and a DBN (Deep Belief Network) having a corresponding number of layers can be constituted by a certain number of layers.

CNN (Convolutional Neural Network) is a structure widely used in the field of object recognition, and will be described with reference to FIGS. 11 and 12. FIG.

CNN (Convolutional Neural Network) is a model that simulates a person's brain function based on the assumption that when a person recognizes an object, it extracts the basic features of the object, then undergoes complicated calculations in the brain and recognizes the object based on the result to be.

11 is a view showing a CNN (Convolutional Neural Network) structure.

The CNN (Convolutional Neural Network) may also include an input layer, a hidden layer, and an output layer.

A predetermined image 1100 is input to an input layer.

Referring to FIG. 11, a hidden layer is composed of a plurality of layers, and may include a convolution layer and a sub-sampling layer.

In the CNN (Convolutional Neural Network), various filters for extracting features of images through convolution operations and pooling or non-linear activation functions for adding nonlinear characteristics Is used.

Convolution is mainly used in filter processing in image processing and is used to implement a filter for extracting features from an image.

For example, as shown in FIG. 12, when the convolution operation is repeatedly performed on the whole image while moving the 3 × 3 window, a proper result can be obtained according to the weight value of the window.

Referring to FIG. 12A, when a convolution operation is performed on a predetermined area 1210 of a whole image using a 3 × 3 window, a result 1201 is displayed.

Referring to FIG. 12 (b), when the area 1220 in which the 3X3 window is shifted to the right by 1 is obtained again, a predetermined result 1202 is obtained.

That is, as shown in (c) of FIG. 12, the final result can be obtained by performing an operation on the entire image while moving a predetermined window.

The convolution layer can be used to perform convolutional filtering to filter information extracted from the previous layer using a filter of a predetermined size (e.g., the 3X3 window illustrated in FIG. 12).

The convolution layer performs a convolution operation on the input image data 1100 and 1102 using a convolution filter and generates feature maps 1101 and 1103 in which features of the input image 1100 are expressed do.

As a result of the convolutional filtering, filtering images corresponding to the number of filters can be generated according to the number of filters included in the convolution layer. The convolution layer may consist of the nodes included in the filtered images.

In addition, a sub-sampling layer paired with a convolution layer may include the same number of feature maps as a convolution layer that forms a pair.

The sub-sampling layer reduces the dimension of feature maps 1101 and 1103 through sampling or pooling.

The output layer recognizes the input image 1100 by combining various features expressed in the feature map 1104.

The obstacle recognizing module of the mobile robot according to the present invention can use the various deep learning structures described above. For example, although the present invention is not limited, a CNN (Convolutional Neural Network) structure widely used in object recognition in an image can be used.

On the other hand, the learning of the artificial neural network can be achieved by adjusting the weight of the inter-node connection line so that the desired output is obtained for a given input. Also, the artificial neural network can continuously update the weight value by learning. Back propagation can be used for artificial neural network learning.

Figs. 13 and 14 are diagrams referred to in explaining the obstacle recognition of the obstacle recognition module 144. Fig.

Referring to FIG. 13, the obstacle recognition module 144 can classify obstacles classified into classes such as an electric fan, a home theater, a multi-tap, a lamp base, a human body, and a barrier.

In addition, the obstacle recognition module 144 classifies and classifies the classes of the fan, the home theater, the multi-tap, the lamp base, and the humanoid as a dangerous obstacle super-class in a superordinate concept.

Also, the obstacle recognition module 144 can classify obstacle obstacles that can run straight, such as a barrier, into non-hazardous obstacle super-class, and recognize them.

Referring to FIG. 14A, the obstacle recognition module 144 recognizes the input image, and the recognition result that the fan has a confidence value of 0.95 and the home theater has a confidence value of 0.7 can be obtained . In this case, the obstacle recognition module 144 can output the fan, which is the recognition result having the higher confidence value, as the recognition result on the input image.

On the other hand, the confidence may be normalized to a range of 0.0 to 1.0.

Referring to FIG. 14B, the obstacle recognition module 144 recognizes the input image, and the recognition result that the fan has a confidence value of 0.35 and the home theater has a confidence value of 0.4 can be obtained.

For example, when the confidence value of 0.6 or less is set to not be recognized, the obstacle recognition module 144 can determine that the unknown data is not selected because the confidence values of the two recognition results are lower than the reference value have.

Referring to FIG. 14 (c), the obstacle recognition module 144 recognizes the input image, and the recognition result that the fan has a confidence value of 0.95 and the home theater has a confidence value of 0.9 can be obtained.

For example, when the recognition result having the confidence value of 0.9 or more is set to be selected as the final recognition result, the obstacle recognition module 144 does not select a specific recognition result because the trust values of the two recognition results are higher than the reference value , It can be judged as a dangerous obstacle, which is an upper concept.

Alternatively, even when the recognition result of the highest confidence value is set to be recognized when the difference in the confidence value is 0.15 or more, it can be determined that the dangerous obstacle is an upper concept.

On the other hand, the driving control module 141 can also control the driving unit 160 to avoid the dangerous obstacle and move the dangerous obstacle.

15 is a flowchart illustrating a method of controlling a mobile robot according to an embodiment of the present invention.

Referring to FIGS. 2 to 7 and 15, first, the mobile robots 100, 100a and 100b move according to commands or settings and perform cleaning (S1510).

Meanwhile, the image acquiring unit 120 may acquire a plurality of images by continuously photographing the periphery of the main body 110 during movement (S1520). A plurality of consecutive images acquired by the image acquisition unit 120 may be stored in the storage unit 150.

If an obstacle is sensed through the sensor unit 170 during the movement of the main body 110 in step S1530, the controller 140 controls the sensor unit 170, based on the moving direction and the moving speed of the main body 110, Point image prior to the point of time when the obstacle is detected (S1540).

The movement direction and the movement speed may be calculated by the travel control module 141 or the like based on the output of the motion detection sensor of the sensing unit 170.

On the other hand, when the mobile robot is set to travel at a reference speed during normal driving, the moving speed is fixed, so that the moving direction of the main body 110 can be discriminated and the image at a specific time point can be selected.

On the other hand, in the case where the moving direction is the straight movement or the rotation travel is less than the predetermined reference value (reference angle), the control unit 140 controls the sensor unit 170 to detect the obstacle, You can select images.

When the sensor 170 acquires an image signal indicating that the obstacle is detected, the mobile robot continues to travel, so that the acquired image may not include the characteristics of the obstacle. In addition, when the sensing range of the sensor unit 170 is short, the probability that the acquired image does not include the characteristics of the obstacle may occur with a higher probability.

Therefore, the control unit 140 can select a specific view image before the sensor unit 170 detects the obstacle by reflecting the moving direction and the speed when traveling straight ahead or traveling straight ahead.

In this case, the control unit 140 can select images of the past time point based on the obstacle detection time point of the sensor unit 170 as the movement speed is slower.

As the moving speed increases, the mobile robot 100 travels a longer distance after the point of time when the sensor unit 170 detects the obstacle. In other words, when the image acquisition unit 120 captures and acquires a plurality of images at a constant speed, the mobile robot moves a longer distance between the time at which the image of the specific frame is captured and the time at which the image of the next frame is captured Drive.

Therefore, as the moving speed increases, the probability that the area occupied by the obstacle in the image is high is high in selecting the image at the point of time when the sensor unit 170 detects the obstacle.

Conversely, as the moving speed is slower, the image acquiring unit 120 obtains more images while traveling the same distance. Therefore, it is preferable to select images of the past time point based on the obstacle sensing point of time of the sensor unit 170 can do.

The obstacle recognition module 144 may select the specific viewpoint image and use it as input data for obstacle recognition.

The obstacle recognition module 144 may recognize the attribute of the obstacle included in the selected point-in-time image based on the data learned in the machine learning (S1560).

The obstacle recognition module 144 may include an artificial neural network that is learned to recognize attributes such as the type of the obstacle by machine learning, and may recognize the attributes of the obstacle based on the learned data (S1560).

For example, CNN (Convolutional Neural Network), which is one of the deep learning structures, is installed in the obstacle recognition module 144, and CNN (Convolutional Neural Network) learned the attributes of the obstacles included in the input data, Can be output.

On the other hand, the driving control module 141 can control driving of the driving unit 160 based on the recognized attribute of the obstacle (S1570).

For example, the travel control module 141 can control to travel by bypassing the obstacle when the recognized obstacle is an obstacle of a height not exceedable.

In addition, the travel control module 141 can be controlled so as to continuously travel straight ahead in the case of an obstacle of a height such that the recognized obstacle can pass, such as a low-level barrier.

In addition, the travel control module 141 can control the obstacle to travel while bypassing obstacles that are likely to be restrained even if the obstacle is a low-level obstacle such as a pedestal of a fan, human hair, a multi-tap, or a wire.

The obstacle recognition module 144 learned by the machine learning has a high recognition rate as the learned object occupies a large portion of the input image data.

Accordingly, in operation S1550, the control unit 140 may control to cut out and extract a portion of the selected point-in-time image corresponding to the direction of the obstacle detected by the sensor unit 170 (S1550).

The present embodiment can increase the recognition rate by extracting another area of the image acquired by the image obtaining unit 120 according to the direction of the obstacle sensed by the sensor unit 170 and using the extracted area as recognition data.

16 is a flowchart illustrating a method of controlling a mobile robot according to an embodiment of the present invention.

Referring to FIGS. 2 to 7 and 16, the mobile robots 100, 100a and 100b can move according to an instruction or a setting (S1610).

When the sensor unit 170 includes the ultrasonic sensor, the obstacle can be recognized by sensing the reflected ultrasonic signal (S1620).

On the other hand, the image acquisition unit 120 continuously captures images of the front and the periphery of the mobile robots 100, 100a, and 100b to acquire a plurality of images.

The control unit 140 can select a specific viewpoint image of a past viewpoint in consideration of a moving direction and a velocity among a plurality of images acquired through the image capturing unit 120. Based on the data learned through machine learning, The attribute of the obstacle detected in the selected point-in-time image can be discriminated.

In addition, the controller 140 may determine whether the detected obstacle is at a height that can be exceeded (S1630).

If the recognized obstacle is an obstacle of a height that can not be exceeded (S1630), the control unit 140 may control to turn around the obstacle after traveling 90 degrees (S1655).

Meanwhile, if the recognized obstacle is an obstacle of a height that can be exceeded (S1630), the fisher 140 can determine the property information of the detected obstacle (S1640). That is, the control unit 140 can determine whether the recognized obstacle is an obstacle that may be proceeded because the possibility of restraint is small.

If it is determined that the obstacle is the obstacle to be proceeded, the control unit 140 may control to continue to move straight (S1650).

Conventionally, it is determined whether or not the detected obstacle is a height that can be exceeded, and the vehicle runs straight in the case of a low-height obstacle.

However, in the case of an obstacle such as a wire, the mobile robot is wound around the wire and constrained.

In addition, the mobile robot attempts to escape from the restrained state by applying motion, which is shaken to the left or to the right, when it is restrained, but a safety accident such as peeling of the covering of the wire may occur.

However, the present invention can improve the reliability by recognizing the obstacle attribute information using the machine learning and the image information, and determining the driving pattern according to the recognized obstacle attribute.

17 to 25 are diagrams referred to in explanation of the control method of the mobile robot according to the embodiment of the present invention.

17 illustrates a case in which the obstacle 1700 is detected in the front direction of the mobile robot 100. [

Referring to FIG. 17, while the mobile robot 100 is traveling, the image acquiring unit 120 can continuously photograph and acquire a plurality of images. The mobile robot 100 acquires the first image 1711 at the first position 1710 and obtains the second image 1721 at the second position 1720 and acquires the second image 1721 at the third position 1730. [ (1731).

Referring to FIG. 18, a predetermined number of images may be stored in the storage unit 150. FIG. In addition, when a predetermined number or more of images are acquired, the image at the earlier time point can be deleted and the newly acquired image can be stored.

The mobile robot 100 can start image recognition in accordance with a trigger signal by sensing an ultrasonic signal.

However, since the range of the ultrasonic sensor used as the trigger signal is short, when the obstacle 1700 is detected and triggered, the captured image 1731 may disappear have.

Accordingly, the present invention stores images consecutively stored in the storage unit 150, determines whether the traveling direction is a straight travel, and then uses the first image 1711 without using the third image 1731 at the time of the trigger, The obstacle recognition can be performed.

Since the mobile robot 100 often travels at a constant speed, it is only determined whether the robot 100 is traveling straight ahead. The robot 100 selects an image at a predetermined time point, for example, an image of the past time point two frames ahead of the obstacle sensing point of the sensor unit 170 The obstacle recognition can be performed.

In addition, the controller 140 may determine how much earlier the image of the past time point is to be selected than the time of detecting the obstacle of the sensor unit 170, taking into consideration the sensing range, performance, processing speed of the obstacle recognition process, .

On the other hand, according to the embodiment, the recognition rate can be increased by extracting and recognizing a part of the image at the selected time point without using the entire image at the selected time point as the obstacle recognition input data.

The present embodiment can extract the center, left, and right regions of the image based on the direction in which the obstacle is detected, rather than simply cropping the center of the entire image to a predetermined size.

For example, when the obstacle is detected in the right direction of the front surface of the main body, a right lower end region of the selected specific viewpoint image is extracted, and when the obstacle is detected in the left direction of the front surface of the main body, Extracts a lower left region of the viewpoint image, and extracts a lower center region of the selected point-in-time image when the obstacle is detected in the front face direction of the main body.

Accordingly, the input data for recognition can include as many obstacles as possible. Since the machine recognizes the greatest weight in the image, it can improve the recognition rate of the obstacle.

19A, a sensor unit 170 according to an embodiment of the present invention includes a first sensor S1 disposed on a front surface of a main body of the mobile robot 100, A second sensor S2 and a third sensor S3 arranged to be spaced apart from each other in the left and right direction.

In this case, the first sensor S1 acts as a transmitter, and the second sensor S2 and the third sensor S3 act as a receiver. For example, the first sensor S1 may emit an ultrasonic signal, and the second sensor S2 and the third sensor S3 may receive a signal reflected from the obstacle. When receiving the reflected signal from the obstacle, it is possible to determine the direction in which the obstacle exists and the distance to the obstacle by using recognition methods using the known ultrasonic waves.

19 (a) illustrates a case where the obstacle X1 is detected at the center of the front of the mobile robot 100. The distance l1 between the sensed obstacle X1 and the second sensor S2 and the distance l2 between the sensed obstacle X1 and the third sensor S3 are equal to each other, , It can be determined that the obstacle X1 is detected at the center of the front side of the mobile robot 100.

In this case, a predetermined region 1910 having a size of a2Xb2 at the lower end of the center of the entire original image 1900 acquired by the image acquiring unit 120 having a size of a1Xb1 as shown in FIG. 19 (b) Can be extracted.

20 to 24 illustrate the case where an obstacle is recognized in terms of a side view.

20 illustrates a case in which the obstacle 2000 is detected in the front right direction of the mobile robot 100. [

Referring to FIGS. 20 and 21, while the mobile robot 100 is traveling, the image acquiring unit 120 can continuously photograph and acquire a plurality of images. The mobile robot 100 acquires the first image 2110 in the first position 2010 and acquires the second image 2120 in the second position 2020 and acquires the second image 2120 in the third position 2030, (2130).

Referring to FIG. 22, a predetermined number of images may be stored in the storage unit 150. FIG. In addition, when a predetermined number or more of images are acquired, the image at the earlier time point can be deleted and the newly acquired image can be stored.

The present invention stores images consecutively stored in the storage unit 150. After determining whether the traveling direction is a straight traveling, the first image 2110 at a point in time before the third image 2130 at the trigger point is used, Can be used to perform obstacle recognition.

Meanwhile, it is also possible to recognize the obstacle after cutting out the right area in the first image 2110 without extracting the first image 2110 as it is.

23A illustrates a case where the obstacle X1 is sensed in the right front direction of the mobile robot 100. FIG.24A illustrates a case where the obstacle X1 is detected in the left front direction of the mobile robot 100 As shown in FIG.

The control unit 140 may control the extraction of the area to be extracted from the image acquired by the image obtaining unit 120 based on the distance l1 between the detected obstacle X1 and the second sensor S2, The distance between the obstacle X1 and the third sensor S3 is proportional to the difference between the obstacle X1 and the third sensor S3.

23 (a), when the distance l1 between the detected obstacle X1 and the second sensor S2 is less than the distance between the detected obstacle X1 and the third sensor S3 (l1 > l2), it can be determined that the obstacle X1 is detected in the right front direction of the mobile robot 100. [

In this case, a predetermined region 1920 having a size of a2Xb2 can be extracted from the lower right end of the entire original image 1900 acquired by the image obtaining unit 120 having a size of a1Xb1 as shown in FIG. 23 (b) have.

The center point CP2 and the start point SP2 of the extraction object area 1920 when the obstacle X1 is detected on the right side are the obstacle X1, It can be confirmed that the center point CP1 of the extraction target area 1910 when shifted to the center is shifted to the right by the predetermined value d1 from the starting point SP1.

In this case, the predetermined value d1 to be shifted is determined by the distance l1 between the sensed obstacle X1 and the second sensor S2, the distance between the sensed obstacle X1 and the third sensor S2, Lt; / RTI > the difference in distance l2 between ll-l2.

24 (a), when the distance l1 between the detected obstacle X1 and the second sensor S2 is greater than the distance between the detected obstacle X1 and the third sensor S3 (l1 < l2), it can be determined that the obstacle X1 is detected in the left front direction of the mobile robot 100.

In this case, it is possible to extract a predetermined region 1930 having a size of a2Xb2 at the lower left of the entire original image 1900 acquired by the image obtaining unit 120 having a size of a1Xb1 as shown in FIG. 24 (b) have.

The center point CP3 and the starting point SP3 of the extraction object area 1930 when the obstacle X1 is detected on the left side are the obstacle X1, Is shifted to the left by a predetermined value d2 at the center point CP1 and the starting point SP1 of the extraction target region 1910 when the center is detected at the center.

In this case, the predetermined value d2 to be shifted is determined by the distance l1 between the sensed obstacle X1 and the second sensor S2, the distance between the sensed obstacle X1 and the third sensor S2, Lt; / RTI &gt; and the distance l2 between lanes &lt; RTI ID = 0.0 &gt;

According to an embodiment of the present invention, the present invention can be applied not only to the straight running but also to the rotational running.

For example, when the rotation is less than a predetermined threshold value, a specific point-in-time image prior to the point of time when the sensor unit 170 senses an obstacle among the plurality of consecutive images can be selected.

Referring to FIG. 25, the mobile robot 100 may move in the order of a first position 2510, a second position 2520, and a third position 2530 in this order. In this case, if the mobile robot 100 rotates below a predetermined threshold value, the obstacle recognition can be performed in the same manner as the straight traveling.

For example, when the mobile robot 100 rotates within the field of view 2501 at the first position 2510, the obstacle recognition can be performed in the same manner as the straight traveling.

The field of view 2501 may refer to a range in which the image acquisition unit 120 can acquire an image.

 Meanwhile, the mobile robot according to an embodiment of the present invention performs a learning process by using an image of the selected specific time point or an image of a selected one of the selected point-in-time images as training data, The neural network (ANN) and the deep neural network (DNN) structure can be continuously updated.

Alternatively, the image of the selected specific time point or an image of the selected time point image may be transmitted to a predetermined server, and data related to machine learning may be received from the predetermined server. Thereafter, the mobile robot may update the obstacle recognition module 144 based on data related to machine learning received from the predetermined server.

26 is a diagram for explaining a method of operating a mobile robot and a server according to an embodiment of the present invention.

26, a DNN structure 144a such as a CNN (Convolutional Neural Network) may be installed in the obstacle recognition module 144 of the mobile robot 100. [

The learned DNN structure 144a receives the input data for recognition, recognizes the attribute of the obstacle included in the input data, and outputs the result.

Unknown data not recognized by the DNN structure 144a may be stored in the storage unit 150 or the self storage space 144b in the obstacle recognition module 144. [

On the other hand, data (unknown data) that the obstacle recognition module 144 can not recognize can be transmitted to the server 70 through the communication unit 190. Also, data that the obstacle recognition module 144 has successfully recognized can be transmitted to the server 70 as well.

The server 70 may generate the configuration of the learned weights and the server 70 may learn the depth of neural network (DNN) structure using training data.

The server 70 may learn the DNN based on the received data and then transmit the updated DNN structure data to the mobile robot 100 to update the DNN structure data.

According to at least one of the embodiments of the present invention, it is possible to acquire image data that can improve the accuracy of obstacle attribute recognition.

According to at least one of the embodiments of the present invention, the mobile robot can determine the nature of the obstacle and adjust the traveling pattern according to the obstacle property, thereby performing highly reliable obstacle recognition and avoidance operations.

According to at least one of the embodiments of the present invention, it is possible to improve the stability of the mobile robot itself and the user's convenience by performing operations such as forward, backward, stop, and bypass according to the recognition result of the obstacle, And a control method thereof can be provided.

In addition, according to at least one embodiment of the present invention, it is possible to provide a mobile robot and a control method thereof that can accurately recognize an attribute of an obstacle based on machine learning.

In addition, according to at least one of the embodiments of the present invention, the mobile robot can efficiently perform machine learning and extract data that can be used for obstacle attribute recognition.

The mobile robot according to the present invention is not limited to the configuration and method of the embodiments described above, but the embodiments may be modified such that all or some of the embodiments are selectively combined .

Meanwhile, the control method of the mobile robot according to the embodiment of the present invention can be implemented as a code that can be read by a processor on a recording medium readable by the processor. The processor-readable recording medium includes all kinds of recording apparatuses in which data that can be read by the processor is stored. Examples of the recording medium that can be read by the processor include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may also be implemented in the form of a carrier wave such as transmission over the Internet . In addition, the processor-readable recording medium may be distributed over network-connected computer systems so that code readable by the processor in a distributed fashion can be stored and executed.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It should be understood that various modifications may be made by those skilled in the art without departing from the spirit and scope of the present invention.

Body: 110
Image acquisition unit: 120
Front camera: 120a
Upper camera: 120b
Control panel: 137
Control section: 140
Storage: 150
Driving section: 160
Sensor part: 170
Communication department: 190

Claims (18)

A traveling part for moving the main body;
An image acquisition unit for continuously photographing the periphery of the main body to acquire a plurality of images;
A storage unit for storing a plurality of consecutive images acquired by the image acquisition unit;
A sensor unit including at least one sensor for detecting an obstacle during the movement; And
When the sensor unit detects an obstacle, selects a specific view image before the obstacle sensing point of the sensor unit among the plurality of consecutive images based on the moving direction and the moving speed of the main body, And an obstacle recognition module for recognizing an attribute of the obstacle included in the selected specific viewpoint image based on the learned data. The method according to claim 1,
Wherein,
Further comprising: a travel control module for controlling driving of the travel section based on the recognized attribute of the obstacle. The method according to claim 1,
Wherein,
Wherein the controller selects a specific point-in-time image before the point of time when an obstacle is detected by the sensor unit among the plurality of consecutive images when the moving direction is a straight-line running or a rotation running less than a predetermined reference value. The method according to claim 1,
Wherein,
And selects an image of a past time point based on an obstacle detection time point of the sensor unit as the movement speed is slower. The method according to claim 1,
The obstacle recognition module includes:
And the learning process is performed using the selected point-in-time image as training data. The method according to claim 1,
Further comprising a communication unit for transmitting the selected point-in-time image to a predetermined server and receiving data related to machine learning from the predetermined server. The method according to claim 6,
And updates the obstacle recognition module based on data related to machine learning received from the predetermined server. The method according to claim 1,
Wherein,
Further comprising an image processing module for extracting a portion of the selected specific viewpoint image corresponding to a direction of an obstacle detected by the sensor portion. 9. The method of claim 8,
The image processing module includes:
Extracting a right lower end region of the selected specific viewpoint image when the obstacle is sensed in the right direction of the front face of the main body,
Extracting a left lower region of the selected specific viewpoint image when the obstacle is sensed in the left direction of the front face of the main body,
And extracts a center lower end region of the selected specific viewpoint image when the obstacle is detected in the front face direction of the main body. Acquiring a plurality of images by continuously photographing a periphery of the main body during movement through an image acquisition unit;
Storing a plurality of consecutive images acquired by the image acquiring unit;
Sensing an obstacle through the sensor unit;
Selecting a point-in-time image before the point of time when the sensor unit detects an obstacle among the plurality of consecutive images based on the moving direction and the moving speed of the main body when the sensor unit detects an obstacle;
Recognizing an attribute of an obstacle included in the selected point-in-time image based on data learned through machine learning; And
And controlling the driving of the traveling section based on the recognized attribute of the obstacle. 11. The method of claim 10,
Wherein the specific point-of-view image selection step selects a point-in-time image before the point of time when the sensor unit detects an obstacle among the plurality of consecutive images in the case that the moving direction is a straight-line running or a rotation running less than a predetermined reference value Control method of mobile robot. 11. The method of claim 10,
Wherein the specific viewpoint image selection step further selects an image of a past viewpoint based on the obstacle detection point of time of the sensor unit as the movement speed is slower. 11. The method of claim 10,
And performing a learning process using the selected point-in-time image as training data. 11. The method of claim 10,
Transmitting the selected point-in-time image to a predetermined server;
And receiving data related to machine learning from the predetermined server. 15. The method of claim 14,
And updating the obstacle recognition module based on data related to machine learning received from the predetermined server. 12. The method of claim 11,
And extracting a portion of the selected point-in-time image corresponding to the direction of the obstacle detected by the sensor unit. 17. The method of claim 16,
The step of extracting a partial region of the selected point-
Extracting a lower right region of the image acquired by the image acquiring unit when the obstacle is sensed in the right direction of the front face of the main body,
Extracts a lower left region of the image acquired by the image acquisition unit when the obstacle is sensed in the left direction of the front surface of the main body,
Wherein the central lower end region of the image acquired by the image acquisition unit is extracted when the obstacle is detected in the front direction of the main body. 12. The method of claim 11,
Wherein the drive control step comprises:
And if the detected obstacle is not an obstacle to which the detected obstacle can advance, control is performed to perform the avoiding operation.

Images