CN117032235A - Mobile robot inspection and remote monitoring method under complex indoor scene - Google Patents

Abstract

The invention discloses a mobile robot inspection and remote monitoring method in a complex indoor scene, which includes: during the movement of the quadruped robot, the host computer receives in real time the two images of the quadruped mobile robot sent by the industrial cameras located on both sides of the quadruped mobile robot. The lane line image on the side is processed, and the straight line that best fits the actual lane line is obtained by fitting, and the straight line is used as the inspection route of the quadruped mobile robot; at the same time, the host computer uses a system located on the head of the quadruped mobile robot. The lidar in the center detects obstacles or walls in front, and switches the quadruped mobile robot to obstacle avoidance mode or turning mode based on the detection results. The invention can effectively overcome the interference of indoor yellow lane line wear and pollution, can accurately adjust the movement posture at no lane lines, and realize remote monitoring and control, and can be suitable for patrol and inspection of large indoor environments.

Description

A mobile robot inspection and remote monitoring method in complex indoor scenes

Technical field

The invention belongs to the field of robot technology, and specifically relates to a mobile robot inspection and remote monitoring method in complex indoor scenes.

Background technique

At present, the development of robotic technology has made significant progress and is widely used in various fields. Especially in the field of inspection and monitoring, robots can effectively complete tasks that are heavy, dangerous or require a long time. However, existing robotic systems still have some challenges in terms of mobility and stability in different terrains and environmental conditions. For the path planning and navigation problems of robots during inspection, researchers have proposed a variety of algorithms and methods. For example, autonomous navigation algorithms based on sensor data and map information use SLAM (Simultaneous Localization and Mapping) technology to achieve real-time map construction and position estimation, and combine with planning algorithms to generate optimal paths. In order to achieve comprehensive inspection tasks, using various sensors (such as cameras, lidar, infrared sensors, etc.), the robot can obtain information about the surrounding environment and then build an accurate environment model to support the execution and decision-making of inspection tasks.

In large indoor environments, the most widely used mobile robot is the AGV car. AGV cars usually use magnetic navigation technology to conduct indoor navigation by sticking magnetic strips on the ground. This method of navigation will be mechanically damaged by hard objects such as metal passing through the loop, which will have a certain impact on navigation and is only suitable for specific and single tasks. used in the environment. The accuracy of SLAM mapping in other indoor scenes such as large warehouses is difficult to guarantee. At the same time, the frequent movement of large goods, large equipment, and people makes it difficult to fix the spatial structure in the scene. The SLAM mapping method is not suitable for such environments. In addition, the use of magnetic navigation in large scenes requires laying a large number of magnetic strips, which affects the original operating activities in the scene.

Compared with common wheeled and crawler mobile robots, quadruped mobile robots have more powerful motion performance and are more suitable for inspections in large indoor environments. However, since quadruped mobile robots cannot use encoders to obtain mileage data, they will The application of quadruped mobile robots can improve efficiency in indoor inspection tasks, but according to the actual environment, it is necessary to design an appropriate strategy and method to guide its movement when the encoder cannot be used.

Wheeled robots such as cleaning robots and painting robots can rely on encoders to calculate mileage to guide their movements. For example, the patent publication number CN111538338B proposes a robot edge motion control system and method, the patent publication number is CN113863195B Among the inventions, edge cleaning methods and sweeping vehicles are proposed, CN115993820A maze robot path planning based on ROS system, CN116175561A a control method, chip and robot for a painting robot, etc. However, although these patents use lidar to detect objects such as walls, they have a single function and cannot switch between line following and walking along walls. They can only be used in fixed situations and are no longer suitable for complex indoor scenes.

Contents of the invention

Technical problem solved: The present invention discloses a mobile robot patrol and remote monitoring method in complex indoor scenes, which solves the problem of patrol and inspection of four-legged mobile robots without an encoder to obtain mileage, and can effectively overcome indoor yellow lane lines. It can accurately adjust the movement posture without interference from wear and pollution, and realize remote monitoring and control. It can be suitable for patrol and inspection of large indoor environments.

Technical solutions:

A mobile robot inspection and remote monitoring method in complex indoor scenes. The mobile robot inspection and remote monitoring method includes the following steps:

Lidar, industrial cameras, and ultrasonic sensors are installed on the quadruped mobile robot body, and are connected to the host computer through a USB expansion dock. The host computer receives control instructions and task instructions issued by the remote control terminal, and switches the quadruped mobile robot to patrol mode. In line mode, the ROS robot operating system is used to process each sensor information to generate the movement instructions of the quadruped mobile robot, which are sent to the lower computer through the UDP protocol, so that the lower computer controls the movement of the quadruped mobile robot according to the movement instructions, so that the quadruped mobile robot moves along the set Movement of inspection routes;

During the movement of the quadruped robot, the host computer receives the lane line images on both sides of the quadruped mobile robot from the industrial cameras located on both sides of the quadruped mobile robot in real time, processes the lane line images, and obtains the best fit with the actual lane line. The straight line that fits the quadruped mobile robot is used as the inspection route; at the same time, the host computer uses the laser radar located in the center of the quadruped mobile robot's head to detect the obstacles or walls in front, and according to the detection results, the four-legged mobile robot The mobile robot can switch to obstacle avoidance mode or turning mode;

When the inspection task is completed and the quadruped mobile robot moves to the origin storage area, the quadruped mobile robot is switched to the storage mode, and the ultrasonic sensor located at the tail of the quadruped mobile robot is used to detect the barrier between the tail of the quadruped mobile robot and the bottom of the origin storage area. board, control the quadruped mobile robot to retreat until the distance between the tail of the quadruped mobile robot and the bottom baffle of the origin storage area is less than the first preset storage distance threshold, and then use lidar to detect the quadruped robot and the side wall of the origin storage area distance, the quadruped mobile robot is controlled to translate toward the side wall until the distance between the quadruped robot and the side wall of the origin storage area is less than the second preset storage distance threshold.

Further, the lidar is a two-dimensional lidar; the information obtained by the lidar scan includes: a one-dimensional distance array [d ₀ , d ₁ , d ₂ , ..., d ₃₅₉ ], where the elements in the array di is the distance between the lidar and the obstacle, and i represents the angle of the radar scanning.

Further, the host computer uses the laser radar located in the center of the quadruped mobile robot's head to detect obstacles or walls in front. The process of switching the quadruped mobile robot to obstacle avoidance mode or turning mode according to the detection results includes:

During the movement of the mobile robot, [d ₁₄₁ , d ₁₄₂ , d ₁₄₃ ,..., d ₂₂₀ ] is intercepted from the 360 frames of distance data detected by the lidar. When 0.2≤min [d ₁₄₁ , d ₁₄₂ , d ₁₄₃ ,..., When d ₂₂₀ ] ≤ 1, generate a stop command and publish the obstacle avoidance ROS topic;

By calculating the time, it is estimated whether the quadruped mobile robot reaches the vicinity of the turning node. If it reaches the vicinity of the turning node, it is intercepted from the 360 frames of distance data detected by the lidar [d ₁₇₈ , d ₁₇₉ , d ₁₈₀ , d ₁₈₁ , d ₁₈₂ ]. The distance d _x between the turning node and the adjacent wall is determined in advance. When min[d ₁₇₈ , d ₁₇₉ , d ₁₈₀ , d ₁₈₁ , d ₁₈₂ ] ≤ _{d x} , a 90° turn instruction is generated and a turning ROS topic is issued.

Further, lidar is used to detect the distance between the quadruped robot and the side wall of the origin storage area, and the quadruped mobile robot is controlled to translate toward the side wall until the distance between the quadruped robot and the side wall of the origin storage area is less than the second preset storage distance threshold. Refers to,

Take [d ₈₉ , d ₉₀ ] in the lidar 360 frame distance data, and pre-set the distance d _z between the quadruped mobile robot and the right wall of the library. When , the quadruped mobile robot stops moving to the right, stands for 2 seconds and then lies down, waiting for the next task command.

Further, the mobile robot inspection and remote monitoring method also includes the following steps:

During the movement of the quadruped robot, if the host computer cannot fit the straight line that best fits the actual lane line, the quadruped mobile robot switches to the wall-moving mode and uses lidar to obtain the distance between itself and the wall in front and side walls. information, determine whether to turn based on the distance information to the wall in front, and adjust itself to be parallel to the side wall based on the distance information to the side wall.

Further, the process of adjusting oneself to be parallel to the side wall based on the distance information to the side wall includes the following steps:

The distances between the left and right sides of the quadruped mobile robot and the wall [d ₆₀ , d ₆₁ , d ₆₂ , d ₆₃ ] and [d ₁₁₉ , d ₁₂₀ , d ₁₂₁ are intercepted from the 360 frames of distance data detected by lidar , d ₁₂₂ ], calculate the average and And the difference ΔD=D ₁ -D ₂ , use formula (1) to calculate the adjustment speed of the quadruped mobile robot:

In the formula, v _r is the actual adjustment speed, v ₀ is the preset adjustment speed, v ₀ is positive for left-turn adjustment, and v ₀ is negative for right-turn adjustment.

Furthermore, if the host computer cannot fit the straight line that best fits the actual lane line, the quadruped mobile robot switches to the wall-moving mode to determine whether the current area belongs to the independently set wireless segment area. If so, drive the four-legged mobile robot to move along the wall. The quadruped mobile robot moves along the wall until it leaves the current area; if not, the quadruped mobile robot is driven to move along the wall and the timing is started at the same time. If the lane line is not detected after the preset waiting time, the quadruped mobile robot is stopped and moves toward The remote terminal sends a task termination command. If the lane line is detected again within the preset waiting time, it switches back to the line patrol mode.

Furthermore, in the line patrol mode, the quadruped mobile robot pre-sets each route to use the left industrial camera field of view or the right industrial camera field of view according to the relative position of the lane line and the quadruped mobile robot.

Further, the process of processing the lane line image and fitting the straight line that best fits the actual lane line includes the following steps:

Divide the RGB image collected by the industrial camera into regions, shield the field of view interference of the quadruped robot body, and intercept the ROI area; the intercepted ROI area is a three-channel RGB image, which is used as the input image, represented by R, G, and B respectively. For the pixel values of the three channels, calculate the corresponding H, S, and V channel pixel values according to equations (2) to (4) to obtain the HSV image:

V=C _max (4)

in, C _max =max (R′, G′, B′); C _min =min (R′, G′, B′); Δ = C _max -C _min ;

Binarize the HSV image according to formula (5) and output the original binary image:

In the formula, dst(I) is the output pixel value, lowerb(I) is the lower threshold of the corresponding channel, upperb(I) is the upper threshold of the corresponding channel, src(I) is the pixel value of the corresponding channel of the input image; when each When the pixel values of the channel are within the set upper and lower threshold ranges, dst(I) is 255, if not, it is 0;

Extract multiple sets of ground lane line HSV color space thresholds according to the on-site lane line wear and pollution conditions, fuse them according to formula (6) to generate masks, use the original binary image as input for color filtering, and obtain the fused binary image, thus obtaining Approximate area of lane lines:

dst＝mask ₁ |mask ₂ |…|mask _j (6)

In the formula, dst is the fused binary image, mask _j is the mask under different color range processing, | means bitwise OR;

Opening and closing operations are performed on the fused binary image to remove the interference of small pollution and wear, and the complete lane line area is obtained; the opening operation first corrodes the image to separate the tiny adhesion points and remove burrs and isolated points. Then dilation is performed to maintain the original shape of the area; specifically, when A is the initial image and B is the m×n structural element, the corrosion and expansion operation processes are as shown in Equations (7) and (8):

In the formula, z represents the translation of B on A, represents the expansion operator formula, and Θ represents the corrosion operator, then the opening operation is as follows (9):

Small noise holes are eliminated through closed operations, the lane line contours are smoothed, and breaks and gaps are filled; the closed operation calculation process is shown in Equation (10):

Taking the optimal binary image as input, Canny edge detection is performed. The main process includes: using Gaussian distribution Perform Gaussian filtering on pixels (x, y) to remove image noise; use Sobel operators S _x and S _y to calculate the amplitude of the gradient of each pixel of image I after Gaussian filtering/> with direction Perform non-maximum suppression based on the gradient direction and amplitude of each pixel to remove pixels that may not constitute edges; set double thresholds to retain strong edges and virtual edges, determine the connection between strong edges and virtual edges, and retain connections with strong edges virtual edge;

Perform Hough straight line transformation on the image after edge detection, and fit multiple sets of straight lines that are suspected to fit the actual lane lines. Set the angle limit θ, take tanθ as the slope limit, and retain the straight line between -tanθ~tanθ;

By judging the midpoint ordinates [y ₁ , y ₂ ,..., y _a ] of multiple sets of straight lines parallel to the actual lane lines, find y _max =max (y ₁ , y ₂ ,..., y _a ), which corresponds to y _max The straight line is the straight line at the bottom of the ROI area, and this straight line is used as the straight line that best fits the actual lane line and the line patrol standard for the quadruped mobile robot;

Calculate the slope k and midpoint pixel coordinates (x, y) of the straight line at the bottom of the field of view that best fits the actual lane line. When k ≤ -0.016, turn left and adjust, when k ≥ 0.016, turn right and adjust. When using the left camera field of view When y ≤ 400px, translate to the right, when y ≥ 480px, translate to the left. When using the right camera field of view, when y ≤ 400px, translate to the left, when y ≥ 480px, translate to the right, ensuring that the mobile robot is parallel to the lane line and The distance is about 0.3m, and the ROS topic of line patrol adjustment is released.

The invention also discloses a mobile robot inspection platform in complex indoor scenes. The mobile robot inspection platform includes a laser radar, a host computer, a slave computer, a left industrial camera, a right industrial camera, an ultrasonic sensor, a quadruped mobile Robot body, joint motors, USB docking station, power supply, head industrial camera and remote monitoring terminal;

A laser radar is placed at the front end of the vertical section of the central axis of the rectangular body surface of the quadruped mobile robot body. The left and right industrial cameras are respectively placed on the left and right sides of the cube frame above the rectangular body and in the head space of the quadruped mobile robot body. The side industrial camera and the head industrial camera, the central axis of the cube frame coincides with the central axis of the quadruped mobile robot body, and an ultrasonic sensor is placed at the tail of the rectangular body of the quadruped robot body;

There are a host computer and a lower computer placed inside the quadruped mobile robot body. The upper computer and the lower computer are connected through network cables. The upper computer contains a WiFi module;

The remote monitoring terminal is wirelessly connected to the host computer, and the quadruped mobile robot is controlled using the mobile robot inspection and remote monitoring methods in complex indoor scenes as mentioned above.

Beneficial effects:

The mobile robot inspection and remote monitoring method in complex indoor scenes of the present invention enables the quadruped mobile robot to complete patrol and inspection tasks in complex indoor environments without an encoder to output mileage data.

Description of the drawings

Figure 1 is a principle framework diagram of a mobile robot inspection and remote monitoring method in a complex indoor scene according to an embodiment of the present invention;

Figure 2 is a schematic structural diagram of a quadruped mobile robot according to an embodiment of the present invention;

Figure 3 is a flow chart of a mobile robot inspection and remote monitoring method in a complex indoor scene according to an embodiment of the present invention.

Detailed ways

The following examples can enable those skilled in the art to understand the present invention more comprehensively, but do not limit the present invention in any way.

Referring to Figure 3, the present invention discloses a mobile robot inspection and remote monitoring method in a complex indoor scene. The mobile robot inspection and remote monitoring method includes the following steps:

Lidar, industrial cameras, and ultrasonic sensors are installed on the quadruped mobile robot body, and are connected to the host computer through a USB expansion dock. The host computer receives control instructions and task instructions issued by the remote control terminal, and switches the quadruped mobile robot to patrol mode. In line mode, the ROS robot operating system is used to process each sensor information to generate the movement instructions of the quadruped mobile robot, which are sent to the lower computer through the UDP protocol, so that the lower computer controls the movement of the quadruped mobile robot according to the movement instructions, so that the quadruped mobile robot moves along the set Movement of inspection routes;

During the movement of the quadruped robot, the host computer receives the lane line images on both sides of the quadruped mobile robot from the industrial cameras located on both sides of the quadruped mobile robot in real time, processes the lane line images, and obtains the best fit with the actual lane line. The straight line that fits the quadruped mobile robot is used as the inspection route; at the same time, the host computer uses the laser radar located in the center of the quadruped mobile robot's head to detect the obstacles or walls in front, and according to the detection results, the four-legged mobile robot The mobile robot can switch to obstacle avoidance mode or turning mode;

When the inspection task is completed and the quadruped mobile robot moves to the origin storage area, the quadruped mobile robot is switched to the storage mode, and the ultrasonic sensor located at the tail of the quadruped mobile robot is used to detect the barrier between the tail of the quadruped mobile robot and the bottom of the origin storage area. board, control the quadruped mobile robot to retreat until the distance between the tail of the quadruped mobile robot and the bottom baffle of the origin storage area is less than the first preset storage distance threshold, and then use lidar to detect the quadruped robot and the side wall of the origin storage area distance, the quadruped mobile robot is controlled to translate toward the side wall until the distance between the quadruped robot and the side wall of the origin storage area is less than the second preset storage distance threshold.

As shown in Figures 1 and 2, the present invention also discloses a mobile robot inspection platform in complex indoor scenes. The mobile robot inspection platform includes a laser radar, a host computer, a slave computer, an industrial camera on the left, and an industrial camera on the right. Industrial camera, ultrasonic sensor, quadruped mobile robot body, joint motor, USB docking station, power supply, head industrial camera and remote monitoring terminal;

A laser radar is placed at the front end of the vertical section of the central axis of the rectangular body surface of the quadruped mobile robot body. The left and right industrial cameras are respectively placed on the left and right sides of the cube frame above the rectangular body and in the head space of the quadruped mobile robot body. The side industrial camera and the head industrial camera, the central axis of the cube frame coincides with the central axis of the quadruped mobile robot body, and an ultrasonic sensor is placed at the tail of the rectangular body of the quadruped robot body;

There are a host computer and a lower computer placed inside the quadruped mobile robot body. The upper computer and the lower computer are connected through network cables. The upper computer contains a WiFi module;

The remote monitoring terminal is wirelessly connected to the host computer, and the quadruped mobile robot is controlled using the mobile robot inspection and remote monitoring methods in complex indoor scenes as mentioned above.

The quadruped robot body 7 includes a rectangular body and limbs. The rectangular body is a transverse bracket structure. The upper computer 2 and the lower computer 3 and the power supply 10 are placed inside the rectangular body. A laser radar 1 is placed in the front section of the central axis of the upper layer of the transverse bracket. Through the serial port and USB expansion dock 9 connection, a cube frame is placed in the middle of the upper layer of the horizontal bracket, and an ultrasonic sensor 6 is placed at the tail, which is connected to the USB docking station 9 through the serial port; the left industrial camera 4 and the right industrial camera are placed at the midpoints of the uppermost beams on the left and right sides of the cube frame respectively. 5. The left industrial camera 4 and the right industrial camera 5 are parallel to the central axis of the cube frame and connected to the USB docking station 9. The USB docking station 9 is placed inside the cube frame and serves as a space for line layout; the USB docking station 9 is connected via USB The wiring is connected to the host computer 2. There are two joint motors 8 fixed on the left side of the front end of the rectangular body included in the quadruped robot body 7, and two identical joint motors 8 on the right side of the front end; the left side of the rear end of the rectangular body included in the quadruped robot body 7 is fixed with two joint motors 8. 2 joint motors 8, 2 identical joint motors 8 are symmetrical to the right side of the front end; each outermost joint motor 8 is connected to the leg structure, there are 4 leg structures in total, and the 4 leg structures are composed of the rectangular body Quadruped robot body 7.

Lidar 1 is a 0.2m-16m lidar. The scanning range of Lidar 1 is 360° around the body. The left industrial camera 4, the right industrial camera 5 and the head industrial camera have no distortion, and the field of view range is 100°, sandwiched with the ground. The angle is adjustable. The detection distance of ultrasonic sensor 6 is 0.03m-5m, and the detection method of ultrasonic sensor 6 is linear measurement; lidar 1 and ultrasonic sensor 6 obtain distance information. The lidar coordinate system takes the central axis of the mobile robot as the polar axis, and the front is the positive axis. direction. Use lidar 1 to detect obstacles within the range of 0.2m-1m in front of the mobile robot during inspection operations; use lidar 1 to detect the distance from the wall in front of the mobile robot during inspection operations, and transmit the detected information through the serial port The host computer 2 uses this distance as a reference to determine whether to make a turn: According to the on-site environment, when the mobile robot needs to move along the wall, lidar is used to detect the distance between the side of the mobile robot and the wall, and the detected information is transmitted to the host computer 2 through the serial port. Calculate the difference between the read distance data within the three angle ranges to determine the parallel status of the mobile robot and the wall. If not, make adjustments; the ultrasonic sensor 6 measures the straight-line distance from the tail of the mobile robot to the wall and baffle for The mobile robot returns to the origin; the left industrial camera 4 and the right industrial camera 5 are fixed on the left and right sides of the mobile robot and are used to detect lane lines on both sides of the road.

The upper computer and the lower computer are microcomputers that can install the Ubuntu system. The lower computer 3 receives and sends information to the upper computer 2, processes the received information at the same time, determines the inspection task type and sends action instructions to the joint motor 8, and sends lidar 1, ultrasonic sensor 6, The start and stop instructions and inspection task completion instructions of sensors such as the left industrial camera 4 and the right industrial camera 5; the host computer 2 is used to obtain the information of the laser radar 1, the ultrasonic sensor 6, and the left industrial camera 4 and the right industrial camera 5. After the detection data is judged and processed, the task instructions and control instructions are sent to the lower computer 3, and the upper computer 2 communicates with the remote monitoring terminal.

The mobile robot inspection and remote monitoring method in complex indoor scenes of the present invention is based on the ROS operating system to realize remote communication of intelligent mobile devices. Specifically, it includes: installing the Ubuntu system on both the upper computer and the lower computer, and configuring the ROS environment; operating the ROS on the upper computer Install the function package under the system, which includes: publish and subscribe service function package, call radar function package, serial communication function package, flask framework, mjpg-streamer; in the host computer ROS operating system environment, run the startup file, the startup file includes: web framework program, UDP communication program between the upper computer and lower computer, lidar ranging program, line patrol program, warehousing program, mjpg-streamer; the upper computer is connected to the LAN to determine the IP address and port of the upper computer, and configures the upper computer and lower computer , the connection between the upper computer and the remote terminal, the lower computer is connected to the LAN, the IP address and port of the lower computer are determined, the connection between the lower computer and the upper computer is configured, the mobile robot motion program is written on the lower computer, and an executable file is generated , write a startup script, and set up automatic startup; after the upper computer and lower computer are powered on, the procedures described in steps three and four will start automatically. The remote terminal is connected to the LAN, open the browser web page on the remote terminal, enter the address configured by the upper computer, and open Remote control terminal web page; enter the remote control terminal web page, select the task mode, the mobile robot will patrol along the lane line or wall according to the preset route, and conduct inspection after arriving at the designated location. After the inspection task is completed, the host computer will report to the remote terminal web page After reporting the task completion result, the mobile robot returns to the origin library and lies down on standby; on the remote control terminal web page, select the control mode, and real-time video, virtual keyboard, and running status will appear. Use virtual buttons to move the mobile robot forward, backward, left pan, and Right pan, left turn and right turn control, view the real-time image in front of the current mobile robot through the window.

Among them, the web application starts the flask framework to configure the task interface, receives the task request sent by the remote terminal, and passes it to the UDP program in the form of a ROS topic. When the lidar ranging program is running, the host computer receives the lidar distance detection information during the movement of the mobile robot, makes judgments based on the obstacle detection and wall detection through the program, and generates stop, turn left, and turn right control instructions. The form of a ROS topic is passed to the UDP sender. The characteristic of the line patrol program is that the host computer receives the detection information of the left industrial camera 4 and the right industrial camera 5 during the movement of the mobile robot, and generates left move, right move, left turn, and right turn control instructions based on the detected lane line conditions through the program. , based on the presence or absence of lane lines, determine whether to switch to the motion state along the wall, and pass it to the UDP sending program in the form of a ROS topic. When the warehousing program is running, the host computer receives the detection information of the ultrasonic sensor 6 during the movement of the mobile robot. When the mobile robot returns to the origin for the warehousing action, the program detects the distance from the tail of the mobile robot to the origin warehouse area baffle to generate a retreat. Or forward control instructions are passed to the UDP sending program in the form of ROS topics; the host computer receives the topic through ROS, generates a UDP data frame from the set of tasks and control instructions described in S31-S34, and sends it to the lower computer through UDP, while receiving the lower computer Data frames sent via UDP; calling the head industrial camera 11 through mjpg-stremer for remote video transmission within the LAN.

Lidar detects the distance between the front end of the mobile robot and the obstacle. During the movement of the mobile robot, when the object appears in the area with a detection angle range of 140°-220° and a length range of 0.2m-1m, the program will make a stop judgment. And released a ROS topic; lidar detects the distance between the front end of the mobile robot and the wall. The detection angle range is 178°-182°. The length range is adjusted according to the inspection site environment. The mobile robot works in task mode and moves to the turn in the route. When , the lidar detection program detects the distance to the wall directly in front. When the distance reaches the set turning point distance, a turning instruction is generated and a ROS topic is released; the lidar detects the distance between the right side of the mobile robot and the wall. When the mobile robot is at When it is necessary to move along the wall in mission mode, the adjustment instructions are issued through the ROS topic.

Specifically, the lidar used in the present invention is a two-dimensional lidar; the information obtained by lidar scanning includes: a one-dimensional distance array [d ₀ , d ₁ , d ₂ , ..., d ₃₅₉ ], where the element d _i in the array is the distance between the lidar and the obstacle, i represents the angle of radar scanning, and the distance required by the radar is determined based on the change in the value of _di .

The lidar detection program implements obstacle avoidance, turning point judgment and wall detection.

The lidar obstacle avoidance method includes: during the movement of the mobile robot, [d ₁₄₁ , d ₁₄₂ , d ₁₄₃ ,..., d ₂₂₀ ] is intercepted from the 360 frames of distance data detected by the lidar. When 0.2≤min[d ₁₄₁ , d ₁₄₂ , d ₁₄₃ , ..., d ₂₂₀ ] ≤ 1, generate a stop command and publish the obstacle avoidance ROS topic;

The lidar turning point judgment method includes: estimating whether the quadruped mobile robot reaches the turning node by calculating the time. If it reaches the turning node, it starts to be intercepted from the 360 frames of distance data detected by the lidar [d ₁₇₈ , d ₁₇₉ , d ₁₈₀ , d ₁₈₁ , d ₁₈₂ ], predetermine the actual distance d _x between the turning node and the adjacent wall. When min [d ₁₇₈ , d ₁₇₉ , d ₁₈₀ , d ₁₈₁ , d ₁₈₂ ] ≤ d _x , a 90° turn is generated Instructions, turn directions are set according to on-site needs, and turn ROS topics are published;

The lidar wall detection method includes: in the area without lane lines, switch to the wall movement mode. At this time, the distance between the right side of the quadruped mobile robot and the wall is intercepted from the 360 frames of distance data detected by the lidar [d ₆₀ , d ₆₁ , d ₆₂ , d ₆₃ ] and [d ₁₁₉ , d ₁₂₀ , d ₁₂₁ , d ₁₂₂ ], calculate the average with/> And the difference ΔD=D ₁ -D ₂ :

In the formula, v _r is the actual adjustment speed, v ₀ is the preset adjustment speed, v ₀ is positive for left-turn adjustment, and negative for right-turn adjustment. Post the ROS topic of wall-moving to keep the quadruped mobile robot parallel to the wall. .

When the mobile robot needs to move along the lane line in mission mode, choose to use the left industrial camera 4 field of view or the right industrial camera 5 field of view according to the relative position of the lane line and the mobile robot; ensure that the mobile robot is parallel to the lane line and about 0.3m apart. .

When the mobile robot is in mission mode, if there is a lane line, each route is preset to use the 4-field view of the left industrial camera or the 5-field field of view of the right industrial camera according to the relative position of the lane line and the mobile robot;

Divide the RGB image collected by the industrial camera into regions, shield the field of view interference of the quadruped robot body 7, and intercept the ROI area;

The intercepted ROI area is a three-channel RGB image, which is used as the input image. R, G, and B represent the pixel values of the three channels respectively. The corresponding H, S, and V channels are calculated according to equations (2) to (4). Pixel value, get HSV image:

V=C _max (4)

in, C _max =max(R', G', B'); C _min =min(R', G', B'); Δ=C _max -C _min .

Binarize the HSV image according to formula (5) and output the original binary image:

In the formula, dst(I) is the output pixel value, lowerb(I) is the lower threshold of the corresponding channel, upperb(I) is the upper threshold of the corresponding channel, and src(I) is the pixel value of the corresponding channel of the input image. When the pixel value of each channel is within the set upper and lower threshold range, dst(I) is 255, if not, it is 0;

Extract multiple sets of ground lane line HSV color space thresholds according to on-site lane line wear, pollution, etc., fuse and generate masks according to formula (6), use the original binary image as input for color filtering, and obtain the fused binary image, thus Get the approximate area of the lane line:

dst＝mask ₁ | mask ₂ |…|mask _i (6)

In the formula, dst is the fused binary image, mask _i is the mask under different color range processing, | means bitwise OR;

Perform opening and closing operations on the fused binary image to remove interference from small contamination and wear, and obtain a complete lane line area;

The opening operation first erodes the image to separate the fine points of adhesion and remove burrs and isolated points, and then expands to maintain the original shape of the area. When A is the initial image and B is the m×n structural element, the corrosion and expansion operation processes are as shown in equations (7) and (8):

In the formula, z represents the translation of B on A, represents the expansion operator formula, and Θ represents the corrosion operator, then the opening operation is as follows (9):

Through closed operations, small noise holes are eliminated, lane line contours are smoothed, and breaks and gaps are filled. The closed operation calculation process is shown in Equation (10):

Taking the optimal binary image as input, Canny edge detection is performed. The main process includes: using Gaussian distribution Perform Gaussian filtering on pixels (x, y) to remove image noise; use Sobel operators S _x and S _y to calculate the amplitude of the gradient of each pixel of image I after Gaussian filtering/> with direction Perform non-maximum suppression based on the gradient direction and amplitude of each pixel to remove pixels that may not constitute edges; set double thresholds to retain strong edges and virtual edges, determine the connection between strong edges and virtual edges, and retain connections with strong edges virtual edge;

Perform Hough straight line transformation on the image after edge detection, and fit multiple sets of straight lines that are suspected to fit the actual lane lines. Set the angle limit θ, take tanθ as the slope limit, and retain the straight line between -tanθ~tanθ;

By judging the midpoint ordinates [y ₁ , y ₂ ,..., y _i ] of multiple sets of straight lines parallel to the actual lane lines, find y _max =max (y ₁ , y ₂ ,..., y _i ), corresponding to y _max The straight line is the straight line at the bottom of the ROI area. This straight line is the straight line that best fits the actual lane line. This straight line is used as the line patrol standard for the quadruped mobile robot;

Calculate the slope k and midpoint pixel coordinates (x, y) of the straight line at the bottom of the field of view that best fits the actual lane line. When k ≤ -0.016, turn left and adjust, when k ≥ 0.016, turn right and adjust. When using the left camera field of view When y ≤ 400px, translate to the right, when y ≥ 480px, translate to the left. When using the right camera field of view, when y ≤ 400px, translate to the left, when y ≥ 480px, translate to the right, ensuring that the mobile robot is parallel to the lane line and The distance is about 0.3m, and the ROS topic of line patrol adjustment is released.

When there are lane lines, the host computer processes the video images of the on-site lane lines collected by the industrial cameras on the left and right sides. After image processing, it effectively overcomes the interference of lane line wear and pollution, and fits a straight line that fits the lane lines. The output is judged by the control strategy. Adjust the instruction to keep the quadruped mobile robot parallel to the lane line; when there is no lane line, the lidar detects the side wall, and outputs the adjustment instruction based on the control strategy judgment to keep the quadruped mobile robot parallel to the wall; the lidar obtains the distance to the obstacle directly ahead Make obstacle avoidance judgments and detect the distance between the turning node and the wall to determine the turning action; when the task is about to be completed, the warehousing attitude is adjusted through lidar and ultrasonic sensing to complete the warehousing action. All turning nodes, task designated points, line patrol areas, and wall movement areas have been predetermined. Lane-free areas include: wireless segment areas set by users and areas where lane lines are not detected by the line patrol program. For example, when moving on a long straight road, if the industrial camera fails to detect lane lines in the field of view for 20 consecutive frames, it is determined that there is no lane line. At this time, the state is switched and the movement is moved along the wall for 20 seconds. If the lane line is re-detected during this process, When it reaches the lane line, it switches to the line patrol mode. If the lane line is not detected after 20 seconds, it stops moving and sends a task termination command to the remote terminal.

Inbound program features include:

It is estimated that the quadruped mobile robot is about to complete the inspection task and move to the vicinity of the origin warehouse area. At this time, it starts to decelerate, and the laser radar performs turning judgment through the turning point judgment method described in claim 9;

After completing the turn, control the quadruped mobile robot to retreat and start the ultrasonic sensor to detect the distance d _y between the tail and the bottom baffle of the warehouse area. When d _y ≤ 168mm, stop retreating and start to translate to the right;

During the right translation process, the robot is kept parallel to the right wall through the wall adjustment method described in claim 9. At this time, [d ₈₉ , d ₉₀ ] in the 360-frame distance data of the lidar is taken, and the quadruped mobile robot is preset The distance d _z from the right wall of the library, when , the quadruped mobile robot stops moving to the right, stands for 2 seconds and then lies down, waiting for the next task command.

The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above-mentioned embodiments. All technical solutions that fall under the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those of ordinary skill in the art, several improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims (10)

1. The mobile robot inspection and remote monitoring method under the complex indoor scene is characterized by comprising the following steps of:

the method comprises the steps that a laser radar, an industrial camera and an ultrasonic sensor are additionally arranged on a quadruped mobile robot body, the quadruped mobile robot body is connected to an upper computer through a USB (universal serial bus) docking station, the upper computer receives a control instruction and a task instruction issued by a remote control terminal, the quadruped mobile robot is switched into a line inspection mode, an ROS (remote operation system) robot operation system is utilized to process information of each sensor to generate a motion instruction of the quadruped mobile robot, the motion instruction is sent to a lower computer through a UDP (user datagram protocol), and the lower computer controls the quadruped mobile robot to move according to the motion instruction, so that the quadruped mobile robot moves along a set line inspection route;

in the motion process of the four-foot robot, the upper computer receives lane line images of two sides of the four-foot mobile robot sent by industrial cameras positioned at two sides of the four-foot mobile robot in real time, processes the lane line images, fits to obtain a straight line which is most attached to an actual lane line, and takes the straight line as a routing inspection route of the four-foot mobile robot; meanwhile, the upper computer detects an obstacle or a wall positioned in front by adopting a laser radar positioned in the center of the head of the four-foot mobile robot, and the four-foot mobile robot is switched to an obstacle avoidance mode or a turning mode according to a detection result;

when the inspection task is completed and the quadruped mobile robot moves to the origin warehouse area, switching the quadruped mobile robot into a warehouse-in mode, detecting the distance between the tail of the quadruped mobile robot and a baffle at the bottom of the origin warehouse area by adopting an ultrasonic sensor positioned at the tail of the quadruped mobile robot, controlling the quadruped mobile robot to retreat until the distance between the tail of the quadruped mobile robot and the baffle at the bottom of the origin warehouse area is smaller than a first preset warehouse-in distance threshold value, detecting the distance between the quadruped robot and the side wall of the origin warehouse area by adopting a laser radar, and controlling the quadruped mobile robot to translate towards the side wall until the distance between the quadruped robot and the side wall of the origin warehouse area is smaller than a second preset warehouse-in distance threshold value.

2. The method for mobile robot inspection and remote monitoring in a complex indoor scene according to claim 1, wherein the lidar is a two-dimensional lidar; the information obtained by laser radar scanning comprises: one-dimensional distance array d ₀ ，d ₁ ，d ₂ ，...，d ₃₅₉ ]Wherein element d in the array _i For the laser radar distance to the obstacle, i represents the angle of radar scan.

3. The method for inspecting and remotely monitoring the mobile robot in the complex indoor scene according to claim 2, wherein the process of using the laser radar positioned in the center of the head of the quadruped mobile robot to detect the obstacle or the wall positioned in front and switching the quadruped mobile robot to the obstacle avoidance mode or the turning mode according to the detection result comprises the following steps:

in the moving process of the mobile robot, the [ d ] is obtained by intercepting 360-frame distance data detected by the laser radar ₁₄₁ ，d ₁₄₂ ，d ₁₄₃ ，…，d ₂₂₀ ]When 0.2 is less than or equal to min d ₁₄₁ ，d ₁₄₂ ，d ₁₄₃ ，…，d ₂₂₀ ]When the value is less than or equal to 1, generating a stop instruction and issuing a obstacle avoidance ROS topic;

calculating time to estimate whether the quadruped mobile robot reaches the vicinity of a turning node, if so, intercepting the 360-frame distance data detected by the laser radar to obtain [ d ] ₁₇₈ ，d ₁₇₉ ，d ₁₈₀ ，d ₁₈₁ ，d ₁₈₂ ]Determining a distance d between the turning node and the adjacent wall in advance _x When min [ d ] ₁₇₈ ，d ₁₇₉ ，d ₁₈₀ ，d ₁₈₁ ，d ₁₈₂ ]≤d _x And generating a turning 90-degree instruction and issuing a turning ROS topic.

4. The method for inspecting and remotely monitoring a mobile robot in a complex indoor scene according to claim 2, wherein detecting the distance between the quadruped robot and the sidewall of the origin warehouse area by using a laser radar, controlling the quadruped mobile robot to translate toward the sidewall until the distance between the quadruped robot and the sidewall of the origin warehouse area is smaller than a second preset warehouse-in distance threshold value,

taking [ d ] in laser radar 360-frame distance data ₈₉ ，d ₉₀ ]Presetting a distance d between the four-foot mobile robot and the right side wall in the warehouse _z When (when)And when the four-foot mobile robot stops moving rightwards, stands for 2 seconds, and then falls down to wait for the next task instruction.

5. The method for mobile robot inspection and remote monitoring in a complex indoor scene according to claim 1, further comprising the steps of:

in the motion process of the quadruped robot, if the upper computer cannot be fitted to obtain a straight line which is most attached to an actual lane line, the quadruped mobile robot is switched to a wall-following motion mode, the laser radar is adopted to acquire distance information of the quadruped mobile robot, a front wall and a side wall, whether turning is carried out or not is judged according to the distance information of the front wall, and the quadruped mobile robot is adjusted to be parallel to the side wall according to the distance information of the side wall.

6. The method for mobile robot inspection and remote monitoring in a complex indoor scene according to claim 5, wherein the process of adjusting itself to be parallel to the side wall according to the distance information from the side wall comprises the steps of:

the distance d between the left side surface and the right side surface of the quadruped mobile robot and the wall is obtained by intercepting 360 frames of distance data detected by the laser radar ₆₀ ，d ₆₁ ，d ₆₂ ，d ₆₃ ]And [ d ] ₁₁₉ ，d ₁₂₀ ，d ₁₂₁ ，d ₁₂₂ ]Calculate the average valueAnd (3) withDifference Δd=d ₁ -D ₂ Calculating by adopting the formula (1) to obtain the adjustment speed of the quadruped mobile robot:

in the formula, v _r To actually adjust the speed v ₀ To preset the speed of adjustment v ₀ For positive indication of left turn adjustment, v ₀ Negative indicates right turn adjustment.

7. The method for inspecting and remotely monitoring the mobile robot in the complex indoor scene according to claim 5, wherein if the upper computer cannot fit to obtain the line most fit to the actual lane line, the quadruped mobile robot is switched to a wall-following motion mode, whether the current area belongs to an autonomously set wireless section area is judged, and if so, the quadruped mobile robot is driven to move along the wall until the quadruped mobile robot leaves the current area; if the four-foot mobile robot does not move along the wall, starting timing, stopping moving the four-foot mobile robot if the lane line is not detected after the waiting time is preset, sending a task termination instruction to the remote terminal, and switching back to the line patrol mode if the lane line is detected again in the waiting time.

8. The method for mobile robot inspection and remote monitoring under a complex indoor scene according to claim 1, wherein the four-legged mobile robot adopts a left industrial camera view or a right industrial camera view for each section of route according to the relative positions of the lane lines and the four-legged mobile robot in an inspection mode.

9. The method for mobile robot inspection and remote monitoring in complex indoor scene according to claim 1, wherein the process of processing the lane line image and fitting the straight line most fitting with the actual lane line comprises the following steps:

dividing regions of RGB images acquired by an industrial camera, shielding visual field interference of a quadruped robot body, and intercepting a region of interest (ROI); the cut-out ROI area is a three-channel RGB image, which is used as an input image, R, G, B respectively represents pixel values of three channels, and corresponding H, S, V-channel pixel values are calculated according to formulas (2) to (4), so as to obtain an HSV image:

V＝C _max (4)

wherein,C _max ＝max(R′，G′，B′)；C _min ＝min(R′，G′，B′)；Δ＝C _max -C _min ；

performing binarization operation on the HSV image according to a formula (5), and outputting an original binary image:

wherein dst (I) is an output pixel value, lowerb (I) is a lower threshold value of a corresponding channel, upperb (I) is an upper threshold value of the corresponding channel, and src (I) is a pixel value of the corresponding channel of the input image; when the pixel value of each channel is in the set upper and lower threshold ranges, dst (I) is 255, and if not, dst (I) is 0;

extracting a plurality of groups of ground lane line HSV color space thresholds according to the abrasion and pollution conditions of the field lane lines, fusing according to a formula (6) to generate a mask, and performing color filtering by taking an original binary image as input to obtain a fused binary image, so as to obtain a lane line approximate region:

dst＝mask ₁ |mask ₂ |…|mask _j (6)

wherein dst is a fused binary image, mask _j Masks under the treatment of different color ranges, | represents bitwise or;

performing open operation and close operation on the fused binary images to remove the interference of fine pollution and abrasion and obtain a complete lane line area; the open operation is carried out on the image firstly to corrode and separate the adhered fine parts, remove burrs and isolated points, and then expand to maintain the original shape of the area; specifically, when a is an initial image and B is a structural element of mxn, the corrosion and expansion operation processes are as shown in formulas (7) and (8):

where z denotes the translation of B over A,representing the expansion operator formula, Θ represents the corrosion operator, then the open operation is as in formula (9):

the tiny noise cavity is eliminated through closed operation, the contour of the lane line is smoothed, and the fracture and the gap are complemented; the closed-loop calculation process is shown in formula (10):

taking the optimal binary image as input, and carrying out Canny edge detection, wherein the main process comprises the following steps: using gaussian distributionPerforming Gaussian filtering on the pixel points (x, y) to remove image noise; using Sobel operator S _x And S is equal to _y Calculating the magnitude of the gradient of each pixel of the Gaussian filtered image I>Direction and direction ofPerforming non-maximum suppression according to the gradient direction and the amplitude of each pixel, and removing pixels which possibly do not form edges; setting a double threshold, reserving a strong edge and a virtual edge, judging the connection condition of the strong edge and the virtual edge, and reserving the virtual edge connected with the strong edge;

performing Hough straight line transformation on the image after edge detection, fitting a plurality of groups of straight lines suspected to be attached to an actual lane line, setting an included angle limit theta, taking tan theta as a slope limit, and reserving the straight lines between-tan theta and tan theta;

by judging the midpoint ordinate [ y ] of a plurality of groups of straight lines parallel to the actual lane line ₁ ，y ₂ ，…，y _a ]Find y _max ＝max(y ₁ ，y ₂ ，…，y _a )，y _max The corresponding straight line is the straight line at the lowest part of the ROI area, and the straight line is used as the straight line which is most attached to the actual lane line and the line inspection standard of the four-foot mobile robot;

calculating the slope k and midpoint pixel coordinates (x, y) of a straight line which is most attached to an actual lane line at the lowest part of the visual field, adjusting left rotation when k is less than or equal to-0.016, adjusting right rotation when k is more than or equal to 0.016, translating right when y is less than or equal to 400px when a left camera visual field is used, translating left when y is more than or equal to 480px, translating left when y is less than or equal to 400px when a right camera visual field is used, translating right when y is more than or equal to 480px, ensuring that the mobile robot is parallel to the lane line and is about 0.3m away from the lane line, and issuing a patrol line to adjust ROS topics.

10. The mobile robot inspection platform under the complex indoor scene is characterized by comprising a laser radar, an upper computer, a lower computer, a left industrial camera, a right industrial camera, an ultrasonic sensor, a four-foot mobile robot body, a joint motor, a USB (universal serial bus) docking station, a power supply, a head industrial camera and a remote monitoring terminal;

a laser radar is placed at the front end of a vertical cut surface of the central axis of the surface of the rectangular body of the quadruped mobile robot body, a left industrial camera, a right industrial camera and a head industrial camera are respectively placed in the head space of the quadruped mobile robot body on the left side and the right side of a square frame above the rectangular body, the central axis of the square frame coincides with the central axis of the quadruped mobile robot body, and an ultrasonic sensor is placed at the tail of the rectangular body of the quadruped mobile robot body;

an upper computer and a lower computer are arranged in the quadruped mobile robot body, the upper computer is connected with the lower computer through a network cable, and the upper computer comprises a WiFi module;

the remote monitoring terminal is in wireless connection with the upper computer, and the four-foot mobile robot is controlled by adopting the mobile robot inspection and remote monitoring method in the complex indoor scene as set forth in any one of claims 1 to 9.