JP7554622B2 - Autonomous mobile work device and work management system - Google Patents

Description

The present invention relates to an autonomous mobile work device and a work management system.

Recent autonomous mobile work devices have been proposed that associate images captured by a camera such as a drive recorder with an environmental map created using a measuring device such as an LRF (Laser Range Finder) and allow the operator to check the images during work (see, for example, Patent Documents 1 and 2). The autonomous mobile work device in Patent Document 1 transfers images captured during work to a home server, and allows the operator to check the images displayed on the display screen of an external device such as a mobile phone via the home server. The autonomous mobile work device in Patent Document 2 allows the operator to check images of the area to be updated before updating the environmental map, so that the environmental map is not updated due to the temporary intrusion of an obstacle.

JP 2002-085305 A JP 2009-169845 A

The home server in Patent Document 1 has a function for capturing video or still images of images displayed on the display screen, but the work status of the autonomous mobile work device cannot be fully understood from the video or still images. In addition, the autonomous mobile work device in Patent Document 2 allows the operator to check images of areas of change on the environmental map and can remove temporary obstacles when updating the environmental map, but the work status cannot be fully understood from images of areas of change on the environmental map.

The present invention was made in consideration of these points, and aims to provide an autonomous mobile work device and program that allows an operator to properly understand the work situation.

The autonomous driving work device of the present invention is an autonomous driving work device that performs work on a work area while traveling autonomously, and is equipped with a working unit that performs work on the work area in accordance with a work plan, an evaluation unit that generates evaluation information allowing the work status to be compared against the work plan, an imaging unit that takes images of the work area while traveling autonomously, and an image memory unit that stores the imaged images in association with position information of each point on the travel route, and the images can be compared by extracting imaged images associated with the same point from multiple imaged images taken when the work plan is created and during work according to the work plan, and the evaluation unit detects obstacles that did not exist when the work plan was created by comparing the imaged images taken when the work plan was created and during work according to the work plan, and the evaluation information includes the imaged image associated with the point where the obstacle was imaged during work according to the work plan and the imaged image associated with that point when the work plan was created, and an evaluation screen including the evaluation information is displayed on the display device.

According to this configuration, the evaluation screen displayed on the display device is visually confirmed by the operator, and the evaluation information contained in the evaluation screen is confirmed by the operator. The operator compares the work plan and the current work situation from the evaluation information, and evaluates whether the work has been performed from start to finish in accordance with the work plan. This allows the operator to properly understand the work situation of the autonomous mobile work device.

The autonomous mobile work device of the present invention is equipped with an imaging unit that captures an image of the work area during autonomous travel, and the evaluation information includes the captured image. With this configuration, the captured image of the work area allows the operator or the like to properly understand the work status of the autonomous mobile work device.

In the autonomous mobile work device of the present invention, the evaluation information includes an image of the work area captured during work according to the work plan, so that it can be compared with an image of the work area captured when the work plan was created. With this configuration, the image of the work area captured when the work plan was created and the current image of the work area can be compared by an operator, etc., allowing the operator, etc. to properly understand the work status of the autonomous mobile work device.

The autonomous mobile work device of the present invention includes a creation unit that creates an environmental map of the work area during autonomous travel, and the evaluation information includes an environmental map that is created during work according to the work plan so as to be comparable to the environmental map that was created when the work plan was created. With this configuration, the environmental map at the time the work plan was created and the current environmental map can be compared by an operator, etc., allowing the operator, etc. to properly understand the work status of the autonomous mobile work device.

The autonomous mobile work device of the present invention includes a change unit that changes the work content of the work plan to the work content performed by the work unit, and the evaluation screen includes a change button that accepts changes to the work content by the change unit. With this configuration, the work content of the work plan and the current work content are compared by an operator or the like based on the evaluation information on the evaluation screen, and the operator or the like presses the change button on the evaluation screen, thereby changing the work content of the work plan.

The work management system of the present invention is a work management system comprising an autonomous traveling work device that performs work in a work area while traveling autonomously, a management server that manages the autonomous traveling work device, and a user terminal that can communicate with the management server, wherein the autonomous traveling work device has a working unit that performs work in the work area in accordance with a work plan, an evaluation unit that generates evaluation information with which the work status can be compared against the work plan, an imaging unit that takes images of the work area while traveling autonomously, and an image storage unit that stores the captured images in a state associated with position information of each point on the travel route, and when the work plan is created and during work in accordance with the work plan, The captured images can be compared by extracting images associated with the same location from the multiple captured images, and the evaluation unit detects obstacles that did not exist when the work plan was created by comparing the images taken at the time the work plan was created and the images taken during work according to the work plan, and the evaluation information includes the captured images associated with the location where the obstacle was imaged during work according to the work plan and the captured images associated with that location at the time the work plan was created, the management server obtains the evaluation information from the autonomously traveling work device and generates an evaluation screen including the evaluation information, and the user terminal displays the evaluation screen obtained from the management server.

According to this configuration, the management server acquires evaluation information from the autonomous mobile work device, and the management server generates an evaluation screen including the evaluation information. The user terminal acquires the evaluation screen from the management server, and the evaluation screen is displayed on the user terminal. The operator or the like checks the evaluation information on the evaluation screen from the user terminal, and the operator or the like compares the work plan and the current work status from the evaluation information, and evaluates whether the work has been performed from start to finish in accordance with the work plan. This allows the operator or the like to properly understand the work status of the autonomous mobile work device.

In the work management system of the present invention, the autonomous mobile work device has a change unit that changes the work content of the work plan to the work content performed by the work unit, and the management server generates an evaluation screen that includes, in addition to evaluation information, a change button that accepts changes to the work content by the change unit, and when the change button is pressed on the evaluation screen displayed on the user terminal, an instruction to change the work content of the work plan is sent from the management server to the autonomous mobile work device. According to this configuration, the work plan and the current work situation are compared by an operator or the like from the evaluation information, and the change button on the evaluation screen is pressed by the operator or the like, so that the work content of the work plan of the autonomous mobile work device can be remotely changed.

The present invention allows an operator to properly understand the working status of an autonomous mobile work device.

1 is a configuration diagram of a work management system according to an embodiment of the present invention. FIG. 1 is a block diagram of a work management system according to an embodiment of the present invention. FIG. 2 is a control block diagram of the cleaning device according to the present embodiment. 3A and 3B are diagrams illustrating an imaging range of an imaging unit and a detection range of an obstacle sensor according to the present embodiment. FIG. 2 is a diagram showing a travel path of the cleaning device of the present embodiment. FIG. 2 is a diagram showing a cleaning area of the present embodiment. FIG. 2 is a diagram showing an example of a captured image according to the present embodiment. FIG. 13 is a diagram illustrating an example of a notification email according to the present embodiment. FIG. 4 is a diagram showing an example of an evaluation screen according to the present embodiment. FIG. 13 is a diagram showing an example of a captured image on which image segmentation has been performed. FIG. 2 is a diagram showing an example of a three-dimensional environmental map. 4 is a flowchart showing the operation of the cleaning device of the present embodiment.

The cleaning device of this embodiment will be described below with reference to the drawings. FIG. 1 is a configuration diagram of the work management system of this embodiment. FIG. 2 is a block diagram of the work management system of this embodiment. In the following description, an autonomously traveling cleaning device will be used as an example of the autonomously traveling work device, but the autonomously traveling work device may be a work device that performs various tasks such as cleaning, polishing, polishing, and wax application on a cleaning area (work area) by autonomous traveling.

As shown in FIG. 1, the work management system 1 includes a cleaning device 30 that cleans the area to be cleaned while autonomously traveling, a management server 10 that manages the cleaning device 30, and a user terminal 20 that can remotely operate the cleaning device 30. The user terminal 20 is a portable device carried by an operator, such as a smartphone or a tablet terminal. The cleaning device 30, the management server 10, and the user terminal 20 are connected to each other via a network 5 so that they can communicate with each other. When the user terminal 20 accesses the management server 10, an operation screen that accepts remote operation of the cleaning device 30 and the cleaning status of the cleaning device 30 are displayed on the user terminal 20. The operator includes a staff member who operates and monitors the work device 30 from a remote location away from the work site, a worker who normally moves, prepares, and inspects the work device 30, and a technician who creates and changes the program of the work device 30, and all of these are hereafter referred to as operators.

The cleaning device 30 is configured to be capable of autonomously cleaning floor surfaces in commercial facilities, manufacturing facilities, train station concourses, etc. Front wheels 32 as drive wheels and a pair of left and right rear wheels 33 as auxiliary wheels are provided at the bottom of the device body 31 of the cleaning device 30. Cleaning members 34 such as cleaning pads and cleaning brushes are provided between the front wheels 32 and rear wheels 33, and a squeegee 35 is provided behind the cleaning members 34. A handle 36 is provided at the upper rear side of the device body 31, and a display unit 37 is provided in front of the handle 36. An imaging unit 38 is provided at the upper front side of the device body 31, and an obstacle sensor 39 is provided in a recess below the imaging unit 38.

2, the management server 10 is provided with a communication unit 11, a storage unit 12, and a control unit 13, similar to a general server. The communication unit 11 performs communication processing between the management server 10 and the cleaning device 30, and between the management server 10 and the user terminal 20. For example, the management server 10 and the cleaning device 30 are connected via a network 5 such as the Internet, and at the work site, the management server 10 and the user terminal 20 are connected via a wireless LAN such as WiFi (registered trademark), and the management server 10 and the user terminal 20 are connected via a wide-area mobile communication system such as LTE (registered trademark), 4G, or 5G. The storage unit 12 is composed of various storage media, and the control unit 13 is composed of a CPU (Central Processing Unit) and the like. The storage unit 12 stores a program that causes the management server 10 to function as a monitoring server for the cleaning device 30.

The user terminal 20, like a typical mobile device, is provided with a communication unit 21, a display operation unit 22, a memory unit 23, and a control unit 24. The communication unit 21 performs communication processing between the user terminal 20 and the management server 10. The display operation unit 22 is a so-called touch panel display, which displays various information obtained from the management server 10 and accepts various operations on the cleaning device 30. The memory unit 23 is composed of various storage media, and the control unit 24 is composed of a CPU, etc. The memory unit 23 stores a program that causes the user terminal 20 to function as a remote controller for the cleaning device 30.

The cleaning device 30 is also provided with a travel unit 41, a cleaning unit (working unit) 42, a communication unit 43, an imaging unit 38, a display unit 37, a measuring unit 46, an operation unit 47, a memory unit 48, a power supply unit 49, and a control unit 50. The travel unit 41 is composed of the above-mentioned front wheels 32, rear wheels 33, a travel motor (not shown), a steering motor (not shown), an encoder (not shown), etc. The travel unit 41 drives the front wheels 32 with the travel motor to travel the cleaning device 30. Note that, although a front-wheel drive cleaning device 30 is exemplified in this embodiment, it may also be rear-wheel drive in which a pair of left and right rear wheels 33 are independently driven by the travel unit 41.

The cleaning unit 42 cleans the cleaning area of the floor surface, for example, by wet cleaning, and is composed of the above-mentioned cleaning member 34 and squeegee 35, a cleaning motor (not shown), an actuator (not shown), a cleaning liquid supply unit (not shown), a wastewater recovery unit (not shown), etc. The cleaning unit 42 presses the cleaning member 34 against the floor surface F using the actuator, and rotates the cleaning member 34 relative to the floor surface using the cleaning motor. At this time, cleaning liquid is sprayed onto the floor surface from the cleaning liquid tank by the cleaning liquid supply unit, and wastewater collected by the squeegee 35 is collected in a wastewater tank by the wastewater recovery unit including a blower motor (not shown).

The communication unit 43 performs communication processing between the cleaning device 30 and the management server 10. The communication unit 43 transmits various information related to the cleaning status, etc. to the management server 10. The communication unit 43 also receives control commands, such as a command to start cleaning, from the management server 10. The imaging unit 38 is a camera capable of capturing an image of the foreground including the floor surface F, and can capture an image of obstacles that exist in front of the cleaning device 30 in the traveling direction. The imaging unit 38 may be a spherical camera capable of capturing panoramic images, or a stereo camera capable of capturing an image of the area in front of the cleaning device 30.

The display unit 37 has a touch panel display screen. The display screen of the display unit 37 displays the captured image captured by the imaging unit 38, as well as an operation screen that accepts operations for the cleaning device 30. Note that the display unit 37 may be a portable device such as a tablet terminal or a smartphone that is detachable from the device body 31. When a portable device is used as the display unit 37, the wireless communication function of the portable device may be used instead of the communication unit 43. The display unit 37 may also be taken outside as a portable device that can be remotely controlled. In this case, the display unit 37 may be provided with an operation function for remotely operating the cleaning device 30.

The measuring unit 46 is equipped with the obstacle sensor 39 described above. The obstacle sensor 39 is, for example, configured with an LRF (Laser Range Finder) that measures the distance and angle from an obstacle or a wall surface. The obstacle sensor 39 emits laser light into the surroundings, and by receiving reflected light from an object, the distance and angle from the obstacle sensor 39 to a nearby obstacle are measured, making it possible to recognize the surrounding environment of the cleaning device 30. Furthermore, instead of detecting obstacles using laser light, the obstacle sensor 39 may be configured to be capable of detecting obstacles using a method that uses an ultrasonic sensor or a captured image of the surrounding environment.

The operation unit 47 accepts manual operation and is composed of the above-mentioned handle 36 (see FIG. 1), throttle, etc. The operation unit 47 converts the steering amount of the handle 36 and the twist amount of the throttle into an electrical signal and outputs it to a control board of the driving motor of the device main body 31 to control the driving state. By operating the operation unit 47, the forward/reverse movement and driving speed of the device main body 31 are adjusted, and left and right turns are performed. In addition, a power switch (not shown) for turning the power ON/OFF and an emergency stop switch (not shown) for completely stopping the operation of the device main body 31 are provided near the operation unit 47 of the device main body 31.

The memory unit 48 is composed of one or more storage media such as a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), a flash memory, etc., depending on the application. The ROM stores, for example, a program for controlling the cleaning device 30. The HDD or flash memory stores, for example, a cleaning plan (operation plan) that sets the autonomous driving route, driving speed, amount of cleaning liquid sprayed, pressing force of the cleaning member 34, etc., and other parameters. The power supply unit 49 is composed of a battery (not shown), a charging circuit, etc. Power is supplied to each part of the device from the power supply unit 49.

The control unit 50 controls each part of the device. The various processes of the control unit 50 may be realized by software using a CPU, or may be realized by a logic circuit (hardware) formed in an integrated circuit or the like. When a CPU is used, the CPU reads out and executes a program stored in the storage unit 48 to perform various processes. Note that, for example, a GPU (Graphics Processing Unit) may be used separately from the CPU for image processing, which will be described later. Note that the control blocks of the control unit 50 and the storage unit 48 will be described in detail later.

The cleaning device 30 has a learning mode in which the cleaning device 30 learns a cleaning plan, and a reproduction mode in which the cleaning device 30 reproduces the cleaning plan. In the learning mode, the cleaning device 30 memorizes the cleaning area (area to be worked on) by manually operating the cleaning device 30. In the reproduction mode, the cleaning device 30 cleans while autonomously traveling within the cleaning area according to the cleaning plan. To set the cleaning area in the learning mode, it is necessary to operate the cleaning device 30 so that the cleaning members 34 pass through the entire cleaning area. Of course, by having the cleaning device 30 travel throughout the entire cleaning area while performing cleaning operations, it is also possible to have the cleaning device 30 learn how to change the cleaning route and cleaning operations.

However, in such a work management system 1, there is a demand for the operator to check the cleaning status of the cleaning device 30. The image captured by the imaging unit 38 of the cleaning device 30 is displayed on the display operation unit 22 of the user terminal 20, allowing the operator to check the cleaning status, but it does not allow the operator to check whether the cleaning device 30 cleaned appropriately according to the cleaning plan. Therefore, the cleaning device 30 of this embodiment generates evaluation information that compares the cleaning status when the cleaning plan is learned with the cleaning status when it is reproduced, and displays the evaluation information on the display operation unit 22 of the user terminal 20.

The detailed configuration of the control unit and memory unit of the cleaning device will be described below. Figure 3 is a control block diagram of the cleaning device of this embodiment. In addition, the symbols in Figures 1 and 2 will be used appropriately in the description.

As shown in FIG. 3, the control unit 50 of the cleaning device 30 is provided with a cleaning control unit 51, a travel control unit 52, a SLAM control unit 53, a travel path creation unit 54, an environmental map creation unit 55, and a captured image acquisition unit 56, which function in the learning mode and reproduction mode. The control unit 50 is also provided with a cleaning plan creation unit 57 that functions in the learning mode, and an evaluation unit 58 and a cleaning plan change unit (change unit) 59 that function in the reproduction mode. Furthermore, the memory unit 48 of the cleaning device 30 is provided with a travel path memory unit 61, an environmental map memory unit 62, a cleaning plan memory unit 63, a device setting memory unit 64, and a captured image memory unit 65.

The cleaning control unit 51 controls the manual cleaning of the cleaning device 30 according to the manual operation of the operator, or the automatic cleaning of the cleaning device 30 according to a cleaning plan. The control contents of the manual cleaning and automatic cleaning include, for example, the amount of cleaning liquid sprayed, the pressing force of the cleaning member 34, and the suction force of dirty water. The driving control unit 52 controls the manual driving of the cleaning device 30 according to the manual operation of the operator, or the autonomous driving of the cleaning device 30 according to a cleaning plan. The control contents of the manual driving and autonomous driving include, for example, the driving speed. Furthermore, the control contents of the autonomous driving include the driving route set in the cleaning area.

The SLAM control unit 53 executes SLAM (Simultaneous Localization Mapping) in real time to estimate the self-position of the cleaning device 30 and create a local map based on the distance and angle from obstacles around the cleaning device 30 measured by the measurement unit 46. The travel path creation unit 54 creates a travel path for the cleaning device 30 by connecting multiple self-positions arranged in chronological order. The environmental map creation unit 55 creates an environmental map by connecting multiple local maps arranged in chronological order. The travel path is stored in the travel path memory unit 61, and the environmental map is stored in the environmental map memory unit 62.

The captured image acquisition unit 56 acquires captured images from the imaging unit 38. The imaging unit 38 constantly captures images of the foreground, including the floor surface, and the captured image acquisition unit 56 acquires the captured images by associating them with position information of each point from the start point to the end point of the travel route. The captured images may be moving images or stop-motion still images as long as they are associated with each point on the travel route. For example, the elapsed time from the start point of the travel route is associated as a property of the captured image, and the captured image is associated with position information of each point on the travel route calculated from the elapsed time. The captured images are stored in the captured image storage unit 65 in a state associated with the position information of each point on the travel route.

The cleaning plan creation unit 57 creates (learns) a cleaning plan including the cleaning area and cleaning conditions during the learning mode. The cleaning area is set based on the travel path (movement trajectory) traveled by the cleaning device 30 (cleaning member 34). In this case, the travel path is read from the travel path memory unit 61, the width of the cleaning member 34 is read from the device setting memory unit 64, and the cleaning area is set on the floor surface based on the travel path and the width of the cleaning member 34. The cleaning conditions set during cleaning by the cleaning device 30 include the amount of cleaning liquid sprayed at each point from the start point to the end point of the travel path, the pressing force of the cleaning member 34, the suction force of dirty water, the travel speed, etc.

When learning (generating) a cleaning plan, the cleaning plan creation unit 57 detects the cleaning status of the cleaning device 30. The cleaning status includes the cleaning area and cleaning time of the cleaning area. The cleaning area is calculated from the length of the travel path and the width of the cleaning member 34, and the cleaning time is measured by a timer (not shown) as the elapsed time from the start of cleaning. Furthermore, when learning a cleaning plan, the captured image acquisition unit 56 acquires captured images from the imaging unit 38, and the environmental map creation unit 55 creates an environmental map. Note that multiple cleaning plans are stored in the cleaning plan storage unit 63.

During reproduction mode, the evaluation unit 58 generates evaluation information that allows the current cleaning status to be compared against the cleaning plan. The evaluation unit 58 detects the cleaning area and cleaning time of the cleaning region as the cleaning status when the learning plan is reproduced. The cleaning area and cleaning time are obtained in the same manner as when the cleaning plan is learned. Furthermore, the evaluation unit 58 acquires captured images from the imaging unit 38 via the captured image acquisition unit 56, and acquires an environmental map from the environmental map creation unit 55. The evaluation information includes various information that allows the cleaning status when the cleaning plan is learned to be compared with the cleaning status when the cleaning plan is reproduced.

The evaluation information includes the cleaning area and cleaning time when the cleaning plan is learned and reproduced. The evaluation information also includes captured images taken when the cleaning plan is learned and reproduced. The evaluation information also includes an environmental map created when the cleaning plan is learned and reproduced. As described above, the captured images are associated with each point on the travel route, so captured images can be compared by extracting captured images associated with the same point from multiple captured images taken when the cleaning plan is created and during cleaning according to the cleaning plan.

The evaluation unit 58 may also have an obstacle detection function. By comparing the captured image when learning the cleaning plan with the captured image when reproducing the cleaning plan, an obstacle that did not exist when learning the cleaning plan is detected. Furthermore, the evaluation unit 58 may include in the evaluation information the difference between the cleaning area when learning the cleaning plan and the cleaning area when reproducing the cleaning plan. In this way, when an obstacle that did not exist when learning the cleaning plan is placed in the cleaning area when reproducing the cleaning plan, the amount of change in the cleaning area and cleaning time caused by the avoidance action (slowing down, stopping, detouring, etc.) is included in the evaluation information. Details of the evaluation information will be described later.

The cleaning plan change unit 59 changes the cleaning content (work content) of the cleaning plan to the cleaning content performed by the cleaning unit 42. The cleaning area and cleaning conditions, etc. included in the cleaning plan are changed to the cleaning area and cleaning conditions, etc. when cleaning is performed according to the cleaning plan. For example, if an obstacle is included in the captured image when the cleaning plan is reproduced and the obstacle is an immovable structure, etc., it is necessary to change the cleaning plan partially or entirely. In such a case, the cleaning plan change unit 59 replaces some or all of the cleaning area (travel route), cleaning conditions, etc. when the cleaning plan is reproduced with learning data, and changes the cleaning plan.

When the cleaning device 30 generates the evaluation information, the cleaning device 30 transmits the evaluation information to the management server 10. The control unit 13 of the management server 10 acquires the evaluation information from the cleaning device 30 and generates an evaluation screen 75 (see FIG. 9) including the evaluation information. The evaluation screen 75 is generated in a web page format. The evaluation screen 75 displays the cleaning status when the cleaning plan was learned and when it was reproduced so that they can be compared. The evaluation screen 75 also displays captured images and an environmental map (not shown in FIG. 9) when the cleaning plan was learned and when it was reproduced so that they can be compared. The evaluation screen 75 also displays a plan change button (change button) 77 (see FIG. 9) that accepts changes to the cleaning content by the change unit.

When the management server 10 generates the evaluation screen, a notification email 73 (see FIG. 8) is automatically sent from the management server 10 to the user terminal 20. A hyperlink 74 (see FIG. 8) to the evaluation information is included in the body of the email, and when the operator of the user terminal 20 clicks the hyperlink 74, the user terminal 20 obtains an evaluation screen 75 from the management server 10. When the evaluation screen 75 is displayed on the display operation unit 22 of the user terminal 20, the cleaning status at the time the cleaning plan was created is compared with the current cleaning status from the evaluation information, allowing the operator to properly grasp the cleaning status of the cleaning device 30.

The evaluation information generation process will be described with reference to Figs. 4 to 9. Fig. 4 is a diagram showing the imaging range of the imaging unit and the detection range of the obstacle sensor in this embodiment. Fig. 5 is a diagram showing the travel path of the cleaning device in this embodiment, where (A) shows the travel path during learning, and (B) shows the travel path during reproduction. Fig. 6 is a diagram showing the cleaning area in this embodiment, where (A) shows the cleaning area during learning, and (B) shows the cleaning area during reproduction. Fig. 7 shows a captured image in this embodiment, where (A) shows the image during learning, and (B) shows the image during reproduction. Fig. 8 is a diagram showing an example of a notification email in this embodiment. Fig. 9 is a diagram showing an example of an evaluation screen in this embodiment. Here, the symbols in Figs. 1 to 3 will be used as appropriate for explanation.

The image pickup unit 38 and the obstacle sensor 39 are provided at the center of the front surface of the device body 31 of the cleaning device 30. As shown in FIG. 4, the image pickup range (angle of view) S1 of the image pickup unit 38 is set to a fan shape with a central angle of about 90 degrees so as to be symmetrical with respect to the center line L extending in the traveling direction of the cleaning device 30. The image pickup range can be expanded by using a wider-angle lens, but the wider the angle, the more image processing to correct image distortion increases, so an appropriate angle of view is selected. The detection range S2 of the obstacle sensor 39 is set to a fan shape with a central angle of 180 degrees or more so as to be symmetrical with respect to the center line L. The image pickup distance of the image pickup unit 38 is from near 0 to infinity, but the detection distance of the obstacle sensor 39 is limited by the detection distance of the laser light, so in reality, the detection range S2 of the obstacle sensor 39 is set to be approximately semicircular as shown by the diagonal lines.

As shown in FIG. 5(A), when learning a cleaning plan, the cleaning device 30 is manually moved in a zigzag pattern from start point A to end point B. The movement trajectory of the cleaning device 30 forms a travel route R shown by the dashed line. Images of the floor surface are constantly captured by the imaging unit 38, and each captured image captured within the imaging range shown by the dashed line (only a portion of which is shown in FIG. 5) is associated with each point on the travel route R and stored. In addition, SLAM is performed using the obstacle sensor 39, and an environmental map is created from walls and the like detected within the detection range shown by the dashed line (only a portion of which is shown in FIG. 5).

As shown in FIG. 5B, when the cleaning plan is reproduced, the cleaning device 30 travels autonomously along the travel route R of the cleaning plan. Near the end point B of the travel route R, an obstacle OB that was not present during learning is placed along the wall. The obstacle OB is, for example, a cardboard box, and is formed lower than the height of the window frame (see FIG. 7). When the cleaning device 30 travels autonomously according to the cleaning plan, an avoidance operation is performed at point D. When reproduced, cleaning ends at point B' and the travel route R is partially overlapped. Even during reproduction, images are captured during cleaning and an environmental map is created.

As shown in FIG. 6(A), the cleaning area and cleaning time of cleaning area A1 are obtained as the cleaning status when learning the cleaning plan. In this case, the trajectory of the cleaning member 34 is obtained from the travel path R and the width of the cleaning member 34, and the cleaning area is obtained by integrating the trajectory of the cleaning member 34. The cleaning time is also obtained by measuring the elapsed time from the start point A of the travel path R to the current travel point using a timer. Note that the movement trajectory of the cleaning member 34 zigzags so that part of the movement trajectory overlaps. The cleaning area may be obtained by ignoring the overlapping area of the movement trajectory of the cleaning member 34.

As shown in FIG. 6(B), the cleaning situation when reproducing (working) according to the cleaning plan is calculated by determining the cleaning area and cleaning time of cleaning area A2, similar to the cleaning situation when learning the cleaning plan. In this case, because part of the travel path R overlaps to avoid the obstacle OB, the cleaning area of cleaning area A2 when reproducing the cleaning plan is smaller than the cleaning area of cleaning area A1 when learning the cleaning plan. When reproducing the cleaning plan, an avoidance operation is performed to go around the obstacle OB, so the cleaning time from cleaning start point A to cleaning end point B' when reproducing the cleaning plan is longer than the cleaning time from cleaning start point A to cleaning end point B when learning the cleaning plan.

When the cleaning area and cleaning time when the cleaning plan is reproduced are obtained, evaluation information is generated that includes the cleaning status when the cleaning plan was learned and when it was reproduced. The evaluation information may include the difference in area between the cleaning area when the cleaning plan was learned and the cleaning area when the cleaning plan was reproduced, and the difference in time between the cleaning time when the cleaning plan was learned and the cleaning time when the cleaning plan was reproduced. The evaluation information also includes captured images and an environmental map when the cleaning plan was learned, as well as captured images and an environmental map when the cleaning plan was reproduced. The captured images and environmental map are used as additional information for determining the presence or absence of an obstacle OB.

Figure 7 (A) shows an image 71 captured at point C (see Figure 5 (B)) when learning a cleaning plan. Image 71 captured when learning a cleaning plan does not include an obstacle OB. Figure 7 (B) shows an image 72 captured at point C when reproducing a cleaning plan. Image 72 captured when reproducing a cleaning plan includes a full view of the obstacle OB. Image 72 of point C where obstacle OB was detected by obstacle sensor 39 when reproducing a cleaning plan is extracted, and image 71 captured at point C when learning a cleaning plan is extracted. By comparing images 71, 72 captured when learning a cleaning plan and when reproducing, it is determined whether or not there is an obstacle OB that did not exist when learning.

The determination of the obstacle OB may be performed by a known image recognition process. In this case, the obstacle OB is detected when the environmental map is created using the obstacle sensor 39, and the captured images 71, 72 at the detection point C of the obstacle OB are extracted when the cleaning plan is learned and when it is reproduced. Then, the pixel values of the captured images 71, 72 when the cleaning plan is learned and when it is reproduced are compared, and if there are a certain number of pixels with a pixel value difference of a predetermined value or more, it is determined that the obstacle OB is an obstacle OB that did not exist when the cleaning plan was learned. In this way, the captured image 72 containing the obstacle OB when the cleaning plan is reproduced and the captured image 71 when the cleaning plan is learned that corresponds to this captured image 72 are extracted.

As shown in FIG. 8, the management server 10 generates a notification email 73 of the evaluation information. The notification email 73 includes the cleaning date, the scheduled time for starting cleaning, the time for finishing cleaning, and a hyperlink 74 to the evaluation information. The hyperlink 74 links to an evaluation screen 75 (see FIG. 9) in web page format generated by the management server 10. The notification email 73 is sent to the user terminal 20, and the evaluation screen 75 can be viewed from the hyperlink 74 in the notification email 73. The notification email 73 may also be used as the evaluation screen, in which case the evaluation information, etc. is included in the email body.

As shown in FIG. 9, the upper half of the evaluation screen 75 displays an image 71 captured when learning the cleaning plan. Next to the captured image 71, the date the cleaning plan was created, the date and time the image was captured, the cleaning area at the time of capture, the total cleaning area at the end of cleaning, and the cleaning completion time until cleaning is completed are displayed. The lower half of the evaluation screen 75 displays an image 72 captured when the cleaning plan was reproduced. Next to the captured image 72, the work date, the date and time the image was captured, the cleaning area at the time of capture, the total cleaning area at the end of cleaning, and the cleaning completion time until cleaning is completed are displayed. In this way, each item of evaluation information is arranged on the evaluation screen 75 so that it can be compared.

No obstacle OB is present in the captured image 71 when the cleaning plan is learned, and an obstacle OB is present in the captured image 72 when the cleaning plan is reproduced. Because the cleaning device 30 performs an avoidance operation, the total cleaning area of 58.6 [ ^m2 ] when the cleaning plan is reproduced is smaller than the total cleaning area of 66.2 [ ^m2 ] when the cleaning plan is learned. In addition, the cleaning end time of 48 minutes 15 seconds when the cleaning plan is reproduced is longer than the cleaning end time of 45 minutes 50 seconds when the cleaning plan is learned. Therefore, the operator can recognize the amount of change in the total cleaning area and cleaning end time due to the avoidance operation.

At the bottom of the evaluation screen 75, an end button 76 and a plan change button 77 are displayed. Pressing the end button 76 closes the evaluation screen 75. Pressing the plan change button 77 sends an instruction to change the cleaning content of the cleaning plan from the management server 10 to the cleaning device 30. The cleaning content of the cleaning plan is changed partially or entirely to the current cleaning content by the cleaning plan change unit 59 of the cleaning device 30. If the obstacle OB is an immovable structure, it will be recognized as an obstacle OB every time cleaning is performed, so the cleaning plan can be changed remotely from the evaluation screen 75, making re-learning unnecessary. In the example shown in the figure, the environmental map is not displayed on the evaluation screen 75, but the environmental map at the time of learning and reproduction of the cleaning plan may be displayed by switching the screen, etc.

The instruction to change the cleaning plan is not limited to being sent from the management server 10 to the cleaning device 30 after cleaning is completed. For example, if cleaning is interrupted due to an obstacle OB, the management server 10 may send a notification email 73 to the user terminal 20 when the cleaning device 30 detects the obstacle OB, and the notification email 73 may display the evaluation screen 75 on the user terminal 20 before cleaning is completed. This allows an operator who views the evaluation screen 75 to change the cleaning plan or replace it with another cleaning plan before the cleaning device 30 reaches the obstacle OB.

As shown in FIG. 10, the captured image 75 (see FIG. 7B) may be subjected to image segmentation (segment recognition function: semantic segmentation) to generate a segmentation image 81. Image segmentation is a technique for distinguishing objects for each pixel of the captured image 75 and displaying them in an identifiable manner. This method is an image annotation method for determining which category each pixel of an image belongs to. Pixels of the same category are treated as the same label, and floors, walls, people, other obstacles OB, etc. are identified. In the figure, the floor F and obstacles OB in particular are displayed in different colors and hatching. The segmentation image 81 is also displayed on the evaluation screen 75, making it easier for the operator to recognize obstacles OB.

Also, as shown in FIG. 11(A), the environmental map may be created three-dimensionally. In general, when creating an environmental map, a two-dimensional LRF is used as the obstacle sensor 39 to detect walls 82 and obstacles OB. In this case, when comparing an environmental map created when learning a cleaning plan with an environmental map created when reproducing a cleaning plan, it is difficult to determine whether the difference is due to a detection error or the presence or absence of an obstacle OB. Therefore, by using a three-dimensional LRF as the obstacle sensor 39, when comparing the environmental maps created when learning a cleaning plan and when reproducing a cleaning plan, it is possible to determine whether the difference is due to a detection error or the presence or absence of an obstacle OB.

More specifically, as shown in FIG. 11(B), the three-dimensional LRF takes into account the factor of object height, so that an obstacle OB lower than the window frame 83 is determined not to be an immovable structure such as a wall 82 or a pillar. On the other hand, a change at a position higher than the height, such as when a door 84 is opened and closed, is determined to be a change in an immovable structure, i.e., an LRF detection error. In the example shown, an object detected in the second layer environmental map M2 lower than the window frame 83 is not detected in the sixth layer environmental map M6 near the window frame 83, so this object is determined to be a movable obstacle OB. A three-dimensional environmental map may be created by sliding or swinging a two-dimensional LRF up and down, or may be created by a three-dimensional Lidar module.

Now, the operation of the cleaning device will be described with reference to FIG. 12. FIG. 12 is a flowchart showing the operation of the cleaning device of this embodiment. Here, the explanation will be given using the symbols in FIG. 1 to FIG. 3 as appropriate. Note that, although an example is shown here in which a captured image, cleaning area, and cleaning time are generated as evaluation information, an environmental map may also be generated as evaluation information.

As shown in FIG. 12, when a cleaning plan is selected (step S01), automatic cleaning is started by the cleaning device 30 (step S02). While the cleaning device 30 is cleaning, it is monitored whether an obstacle OB is detected by the obstacle sensor 39 (step S03). If an obstacle OB is not detected by the obstacle sensor 39 (No in step S03), the process proceeds to step S07. On the other hand, if an obstacle OB is detected by the obstacle sensor 39 (Yes in step S03), an image captured at the detection point of the obstacle OB when reproducing and learning the cleaning plan is extracted (step S04).

Next, the captured images when the cleaning plan was reproduced and when it was learned are compared to determine whether or not there is an obstacle OB that was not present when the cleaning plan was learned (step S05). If the difference between the captured images when the cleaning plan was reproduced and when it was learned is small and an obstacle OB was present when the cleaning plan was learned (No in step S05), the process proceeds to step S07. If the difference between the captured images when the cleaning plan was reproduced and when it was learned is large and an obstacle OB was not present when the cleaning plan was learned (Yes in step S05), the images captured when the cleaning plan was reproduced and when it was learned are acquired by the evaluation unit 58 (step S06).

The processes in steps S03-S06 are repeated until the cleaning plan is completed (No in step S07). When the cleaning plan is completed (Yes in step S07), the automatic cleaning of the cleaning device 30 is completed (step S08). Next, the evaluation unit 58 generates evaluation information including the cleaning area, cleaning time, captured images, etc. when the cleaning plan is reproduced and when it is learned (step S09). The cleaning area and cleaning time when the cleaning plan is reproduced are obtained after cleaning is completed, and the cleaning area and cleaning time when the cleaning plan is learned are obtained when the cleaning plan is learned. Then, the evaluation information is transmitted from the cleaning device 30 to the management server 10 (step S10).

As described above, according to this embodiment, the evaluation screen 75 displayed on the user terminal 20 is visually recognized by the operator, and the evaluation information contained in the evaluation screen 75 is confirmed by the operator. The operator compares the cleaning plan and the current cleaning status from the evaluation information, and evaluates whether or not cleaning has been performed according to the cleaning plan from start to finish. This allows the operator to properly grasp the cleaning status of the cleaning device 30.

In this embodiment, the cleaning plan is changed by comparing the captured images and environmental maps taken when learning and reproducing the cleaning plan, but the cleaning plan may also be changed by comparing the captured images and environmental maps taken when learning a different cleaning plan. In this case, captured images taken in the same direction from the same travel point are stored for each cleaning plan. The cleaning plan may also be changed by comparing the results of the reproduction travel of each cleaning plan.

In addition, although the evaluation information in this embodiment includes the cleaning area and cleaning time, the evaluation information may also include other information that can be compared when learning and reproducing the cleaning plan, such as the maximum speed while driving, the rate of change of speed, maximum power, etc.

In addition, in this embodiment, the cleaning plan was learned by manually operating the cleaning device, but the cleaning plan may be set with parameters that are experimentally, empirically, or theoretically determined from past data.

In addition, the misalignment of the captured images when the cleaning plan is learned and when it is reproduced may be corrected. The positional misalignment of the images may be corrected based on the shape of the ceiling and the position of the lighting, which are unlikely to change when the cleaning plan is learned and when it is reproduced. Furthermore, by applying the above-mentioned image segmentation to the captured images when the cleaning plan is learned and when it is reproduced, the positions of the lighting, fire alarms, etc. may be recognized, and the positional misalignment of the images may be corrected based on these positions.

In addition, in this embodiment, a cleaning device that performs cleaning work is exemplified as the autonomous traveling work device, but the autonomous traveling work device is not limited to a cleaning device. The autonomous traveling work device may be a work device that performs other work such as dry cleaning work, polishing work, polishing work, wax application work, etc. In other words, the autonomous traveling work device may be a robotic cleaning machine, polisher, buffing machine, or wax application machine. Therefore, instead of cleaning members such as pads, cleaning brushes, polishing brushes, polishing brushes, or peeling brushes may be used in the autonomous traveling work device, and further, abrasives, polishing agents, etc. may be used as cleaning fluids.

In addition, in this embodiment, a floor surface is exemplified as the work area, but the work area may be a wall surface, ceiling surface, window surface, mirror surface, etc. It may also be semi-outdoor or outdoor, in which case it is preferable to control the movement of the main body using a SLAM (V-SLAM) method that uses a stereo camera or the like that can measure the depth of the image.

In addition, in this embodiment, the evaluation screen is displayed on the user terminal, but the evaluation screen may also be displayed on the display unit of the cleaning device, or on the display of the management server.

In addition, in this embodiment, an evaluation unit is provided in the control unit of the cleaning device, but this function may be assigned to a management server, so that when the cleaning device recognizes an obstacle, the cleaning device transmits the corresponding image to the management server, and the control unit of the management server executes high-speed, high-performance image recognition processing, and outputs the image recognition results and calculation results to a display device, etc. Furthermore, the cleaning device and the management server may cooperate to execute image recognition processing, etc.

Although this embodiment has been described, other embodiments may be combinations of the above embodiments and variations in whole or in part.

The technology of the present invention is not limited to the above-described embodiments, and may be modified, substituted, or altered in various ways without departing from the spirit of the technical idea. Furthermore, if the technical idea can be realized in a different way due to technological advances or other derived technologies, it may be implemented using that method. Therefore, the claims cover all embodiments that may fall within the scope of the technical idea.

As explained above, this is useful for industrial robots suitable for automated work in work areas, such as autonomous cleaning equipment.

1: Work management system 10: Management server 20: User terminal (display device)
30: Cleaning device (autonomous mobile work device)
37: Display section (display device)
38: Imaging unit 42: Cleaning unit (working unit)
55: Environmental Map Creation Department (Creation Department)
56: Captured image acquisition unit 57: Cleaning plan creation unit 58: Evaluation unit 59: Cleaning plan change unit (change unit)
75: Evaluation screen 77: Plan change button (change button)

Claims (5)

An autonomous mobile work device that performs work on a work area while autonomously traveling,
A work unit that performs work on the work area according to a work plan;
an evaluation unit that generates evaluation information that allows a comparison of a work status against a work plan;
An imaging unit that captures an image of the work area during autonomous traveling;
a captured image storage unit that stores the captured image in a state associated with position information of each point on the travel route ,
By extracting captured images associated with the same location from a plurality of captured images taken when creating a work plan and during work according to the work plan, the captured images can be compared with each other;
the evaluation unit detects an obstacle that did not exist when the work plan was created by comparing images captured when the work plan was created with images captured during work in accordance with the work plan;
The evaluation information includes a captured image associated with a point where an obstacle was captured during work according to the work plan and a captured image associated with the point at the time the work plan was created,
An autonomous mobile work device, characterized in that an evaluation screen including evaluation information is displayed on a display device. A creation unit that creates an environmental map of the work area during autonomous driving,
The autonomous mobile work device according to claim 1, characterized in that the evaluation information includes an environmental map created during work according to the work plan so as to be comparable with the environmental map created when the work plan is created. a change unit that changes the work content of the work plan to the work content performed by the work unit;
3. The autonomous mobile work device according to claim 1 , wherein the evaluation screen includes a change button that accepts a change to the work content made by the change section. A work management system including an autonomous traveling work device that performs work in a work area while traveling autonomously, a management server that manages the autonomous traveling work device, and a user terminal that can communicate with the management server,
The autonomous traveling work device has a working unit that performs work in the work area according to a work plan, an evaluation unit that generates evaluation information that allows the work status to be compared against the work plan, an imaging unit that takes images of the work area during autonomous traveling, and an image storage unit that stores the images in a state associated with position information of each point on a traveling route ,
By extracting captured images associated with the same location from a plurality of captured images taken when creating a work plan and during work according to the work plan, the captured images can be compared with each other;
the evaluation unit detects an obstacle that did not exist when the work plan was created by comparing images captured when the work plan was created with images captured during work in accordance with the work plan;
The evaluation information includes a captured image associated with a point where an obstacle was captured during work according to the work plan and a captured image associated with the point at the time the work plan was created,
the management server acquires evaluation information from the autonomous navigation and operation device and generates an evaluation screen including the evaluation information;
The work management system is characterized in that the user terminal displays an evaluation screen acquired from the management server. the autonomous navigation work device has a change unit that changes the work content of the work plan to the work content performed by the work unit,
the management server generates an evaluation screen including, in addition to the evaluation information, a change button for accepting a change of the work content by the change unit;
The work management system described in claim 4 , characterized in that when a change button is pressed on the evaluation screen displayed on the user terminal, an instruction to change the work content of the work plan is sent from the management server to the autonomous mobile work device.

Images