CN113808128A - Intelligent compaction overall process visualization control method based on relative coordinate positioning algorithm - Google Patents
Intelligent compaction overall process visualization control method based on relative coordinate positioning algorithm Download PDFInfo
- Publication number
- CN113808128A CN113808128A CN202111195585.3A CN202111195585A CN113808128A CN 113808128 A CN113808128 A CN 113808128A CN 202111195585 A CN202111195585 A CN 202111195585A CN 113808128 A CN113808128 A CN 113808128A
- Authority
- CN
- China
- Prior art keywords
- image
- road
- foreign matter
- foreign
- coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Quality & Reliability (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to an intelligent compaction whole-process visual control method based on a relative coordinate positioning algorithm, which comprises the following steps: collecting road surface images of all road sections by using a land leveler, and processing and storing the road surface images of all road sections according to the pile numbers to form an image set; outputting the image set through a Segnet network in a deep learning neural network, wherein the image set is an image set with a foreign object label; converting the image set with the foreign matter label into an image set with a foreign matter circumcircle and an actual distance of the circle center on the whole road coordinate through a relative coordinate positioning algorithm, which is called a road whole-section foreign matter image set for short; processing the whole-section foreign matter image set of the road through a CNN network in a deep learning neural network, identifying each strip image under the driving strip of the road roller, and judging whether foreign matters exist or not; and determining early warning information according to the foreign matter condition, guiding compaction operation, solving the problems of large satellite positioning error and difficulty in fitting the path of the land leveler and the road roller, and adjusting roadbed compaction parameters in real time.
Description
Technical Field
The invention relates to the technical field of roadbed and pavement engineering, in particular to an image identification and relative coordinate positioning method applied to an intelligent highway compaction process, which is used for identifying foreign matters (oversized particles) existing in a roadbed soil paving process, solving the problems of influence of the foreign matters on compaction quality and large satellite positioning error and realizing overall road section management and control in the overall roadbed compaction process.
Background
The roadbed compaction is an important ring in roadbed construction, can reduce gaps among soil particles, rearrange the soil particles, increase the density, increase cohesive force and friction among the soil particles, and improve the stability of the roadbed. If the roadbed compactness does not meet the requirement, roadbed settlement and other diseases are easy to occur under the action of repeated load of vehicles, and then pavement cracks, ruts, pits and other diseases are generated, so that pavement damage is accelerated, the service life is shortened, and the maintenance cost is increased.
The grading of the aggregate has obvious influence on the compactness achieved after rolling. Practice proves that gaps are generated by single grading or discontinuous grading, and compaction is difficult to crush. The large soil blocks in the soil without being broken are one of the reasons for the settlement of the roadbed. In the traditional compaction process, the soil on the construction site is brought back to a laboratory for detection by adopting a spot inspection method, and the detection result has randomness. In addition, when an indoor compaction test is carried out, the experimenter can pick out foreign matters for carrying out the test, and the construction site can not find the foreign matters existing in the paving process in time and can not objectively evaluate the compaction quality.
With the rise of artificial intelligence, a great deal of results are applied to particle size recognition scenes, and good results are obtained, but the following problems still exist: 1. the existing particle size identification method is used for realizing the full-section identification of foreign matters in a roadbed by arranging an image acquisition device on a road roller, but the photographed image is distorted due to the large-amplitude construction attribute of the road roller, and the aggregate particle size cannot be accurately identified. 2. Most of the existing patents are directed at improving a particle size recognition algorithm and a shooting device, do not innovate database establishment, storage and quality tracing, lack of a systematic whole-process construction scheme, and have no practical engineering significance. 3. The existing scheme carries out real-time positioning through satellites such as a GPS (global positioning system) and a Beidou, but the satellite positioning has larger errors, and a specific solution is not given.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to solve the technical problems that the control method for the whole-section visualization of the intelligent compaction process based on the relative coordinate positioning algorithm can solve the problem that the image distortion collected by an image collecting device caused by the large amplitude attribute of a road roller cannot be well applied to the training and prediction processes of a deep learning neural network, and the problems of small data volume, low accuracy and the like exist during the neural network training. Meanwhile, the method avoids the influence of foreign matters on compaction quality in the roadbed compaction construction process, obtains the road surface image of the whole road section, adopts a deep learning neural network to identify the foreign matters, obtains accurate foreign matter positioning through a relative coordinate positioning algorithm, sends out early warning, can call the foreign matter image in real time, provides compaction information and forms a reliable quality tracing system. The method solves the problems that the satellite positioning error is large and the path of the land leveler and the road roller is difficult to fit, and adjusts the roadbed compaction parameters in real time.
The technical scheme for solving the technical problems is as follows:
an intelligent compaction whole process visualization control method based on a relative coordinate positioning algorithm comprises the following contents:
collecting road surface images of all road sections by using a land leveler, and processing and storing the road surface images of all road sections according to the pile numbers to form an image set;
outputting the image set through a Segnet network in a deep learning neural network, wherein the image set is an image set with a foreign object label;
converting the image set with the foreign matter label into an image set with a foreign matter circumcircle and an actual distance of the circle center on the whole road coordinate through a relative coordinate positioning algorithm, which is called a road whole-section foreign matter image set for short;
processing the whole-section foreign matter image set of the road through a CNN network in a deep learning neural network, identifying each strip image under the driving strip of the road roller, and judging whether foreign matters exist or not;
and determining early warning information according to the foreign matter condition and guiding compaction operation.
The method comprises the following steps of collecting road surface images of all road sections by using a land leveler, processing and storing the road surface images of all road sections according to pile numbers, and forming an image set in the following specific process:
the method comprises the following steps that a three-axis mechanical anti-shake tripod head is installed at the tail of the grader, a plurality of cameras are installed on the tripod head and are responsible for shooting the road surface of a current road stripe, the camera lens is vertically downward, the cameras record the road surface in the movement process of the grader in a stripe mode along with the grader, and after the recording is finished, the video is guided into a monitoring center to cut the video, so that the starting time and the stopping time of a single stripe video shot by each camera correspond to the starting point and the ending point of the whole road;
the cut video obtains the length of an actual road corresponding to the pictures in the video through the actual length of a scale, the pixel length of the scale and the width of the screen resolution of the camera, and then the number of interval frames between two adjacent pictures when the pictures can be connected end to form the length of the whole road is obtained according to the running speed of the land leveler and the frame rate of the camera;
intercepting pictures in a video according to a video image in a mode of intercepting the pictures according to the determined interval frame number, wherein the intercepted pictures form an image set, outputting the relative coordinates of the intercepted pictures relative to the origin of the road, connecting all the intercepted pictures in a video sequence end to obtain an image with the length of one strip, splicing a plurality of video sequences acquired by a plurality of cameras to form an image with the length of the whole road of one strip, and jointly splicing the images with the length of the whole road of all the strips to obtain an image covering the whole road section; when the pictures are connected end to end, the width and the length of the pictures respectively correspond to the width and the stake number in the actual road, the relative coordinate of the image origin of each picture relative to the road origin is recorded, the recording of the corresponding stake number information is realized, and an image set is obtained.
The specific process of the relative coordinate positioning algorithm is as follows:
the image with foreign matter label can store the coordinate information of the image origin in a centralized way, when the foreign matter is identified, the coordinates of the pixel points of the circumscribed circle of the foreign matter under the image coordinate system are output and substituted into the formula (4), and the actual distance information x of the foreign matter from the image origin is obtained2(ii) a Then, the coordinate information x of the image origin is obtained through the formula (5)3Calculating to obtain the actual distance of the circle center of the foreign object circumscribed circle on the coordinates of the whole road, including the abscissa and the ordinate of the circle center;
x4＝x2+x3 (5)
s1: the actual distance of the scale;
x2: actual distance of the foreign object from the image origin;
p1: scale pixel length;
p'1: foreign matter circumcircle center pixel point coordinates under an image coordinate system;
x3: relative coordinate information of the picture of the foreign matter at the origin of the road;
x4: coordinates of foreign bodies on whole roadThe actual distance of (d);
equations (4) and (5) are calculated by taking the abscissa x as an example, and the calculation process of the ordinate y is the same.
The specific process of determining the early warning information according to the foreign matter condition is as follows:
calculating early warning information: if foreign bodies exist, calculating the vertical coordinate position information of the foreign bodies on the road roller strip according to the formula (6):
s2: the actual length of the foreign matter image scale of the whole road section is measured;
p2: the proportion scale pixel length of the foreign matter image of the whole road section;
y5: the actual distance of the foreign bodies on the longitudinal coordinate of the road roller strip, namely the running distance, is m;
p'2: the number of longitudinal coordinate pixel points of the foreign matter on the road roller strip;
after the longitudinal coordinate position information of the foreign matter in the road roller strip is obtained, calculating the early warning time according to a formula (7):
t2＝(y5-0.5)/v2 (7)
t2: early warning time;
v2: the running speed of the road roller;
the running distance and the early warning time are sent to a monitoring center in real time through calculation of early warning information, the monitoring center and the road roller end communicate in a wireless mode to send out early warning, and meanwhile, the monitoring center and the road roller end are reminded of the existence of foreign matters which can affect the compaction quality, and corresponding operation is carried out to solve the problem; if no foreign matter exists, no early warning is generated, and the calculation of the early warning information can send road normal information to the monitoring center.
An intelligent compaction overall process visualization control system based on a relative coordinate positioning algorithm is characterized by comprising an image acquisition processing module, a three-dimensional drawing module, a deep learning neural network module, a database module and a calculation module;
the image acquisition processing module is mainly used for acquiring images by utilizing a camera at the tail of the grader and processing the images, and the grader is provided with the following equipment: the system comprises a land leveler steel frame, a three-axis mechanical anti-shake holder, a camera and a wireless real-time transmission device;
the image processing process comprises the following steps: intercepting pictures in the video image in a mode of intercepting the pictures according to the determined interval frame number, wherein the intercepted pictures are connected end to form an image of the whole road length, and the relative coordinate of the image origin of each picture relative to the road origin can be recorded, so that the recording of corresponding pile number information is realized, and an image set is formed;
the three-dimensional drawing module establishes an initial road model and draws a virtual camera in a monitoring center by using three-dimensional graphic software, and a wireless real-time transmission device arranged on the land scraper and a wireless real-time transmission device of the monitoring center perform wireless transmission, so that the actual detection scene of the land scraper can be synchronized in real time in the initial road model and put on a display screen of the monitoring center; the initial road model receives an image set and a data stream (X, Y, Z, T) from the image acquisition processing module, wherein X is an image origin horizontal coordinate, Y is an image origin vertical coordinate, Z is a camera height, T is an image acquisition time, and the image set is sequentially filled into the initial road model according to the image origin coordinate to realize the reappearance of the actual road scene in the monitoring center;
the road model receives foreign body coordinate data streams (X 'and Y') obtained through conversion between relative coordinates, wherein X 'is the horizontal coordinate of the center of a foreign body circumcircle under a road coordinate system, and Y' is the vertical coordinate of the center of the foreign body circumcircle under the road coordinate system, and foreign body position information is labeled in real time;
the road model is provided with a plurality of parameter interfaces and can access filling body parameters, road roller control parameters and real-time parameters, and integrated data streams and image sets of various parameter data received by the road model are stored in the cloud storage unit together to form a three-dimensional reappearing model; the images can be amplified at any time and the parameter information can be called by clicking the road images in the three-dimensional reproduction model or inputting the positioning coordinates, so that the supervision, control and compaction quality tracing of the whole roadbed compaction process are realized;
the deep learning neural network module is used for outputting an image with a foreign matter label and judging whether the image under the road roller strip is a foreign matter;
the database module is used for storing and updating a corresponding image set of the training deep learning neural network module;
and the calculation module is used for conversion between relative coordinates and calculation of early warning information.
The database module comprises a first database and a second database, and an image set output by the image acquisition processing module is stored in the first database; the second database stores an image set for determining the actual distance of the circle center of the foreign object circumscribed circle in the image set with the foreign object label on the whole road coordinate, which is called a road whole-section foreign object image set for short. Both databases include big data for training the corresponding network and current data collected in real time.
Compared with the prior art, the invention has the beneficial effects that:
1) the invention uses equipment arranged at the tail part of the land leveler to collect road information, intercepts the whole road image by a method of video spacing with a certain number of frames, improves the camera efficiency and the coverage rate, has the advantage of small vibration amplitude, and can effectively avoid the characteristics of low quality and easy distortion of the shot image caused by the large amplitude attribute of the road roller.
2) According to the invention, a standardized, unified and huge data volume database is constructed in a supervised learning manner, so that the accuracy and generalization capability of model prediction are greatly improved, and the occurrence of over-fitting problems is reduced. The image after each recognition is directly stored in the database for expansion of the database, so that the data volume is improved, and the self-updating of the database is realized.
3) According to the invention, the images output by the image acquisition and processing module are spliced into the three-dimensional reproduction model according to the corresponding positions, so that the visualization of the real scene of the road is realized, meanwhile, the three-dimensional reproduction model continuously receives the images with the foreign object labels output by the Segnet network, and replaces the images at the corresponding positions, so that the foreign object labels are displayed by the three-dimensional reproduction model. And the three-dimensional reappearing model also displays all parameters, establishes a road model (including the size, the size and the shape of foreign matters) (instead of mathematical modeling), updates the compaction construction condition in real time, realizes the visualization of the whole construction process and displays the actual scene. The model is provided with a plurality of parameter interfaces, is input into the road model in a data stream mode, and is stored into the cloud storage unit together with the image set. The monitoring center can call images and parameters at any time through time (or coordinate positioning), and transmits the images and parameters to the road roller end through 4G, so that a complete quality tracing system is established.
4) The invention discloses a process construction method for observing longitudinally and integrally compacting the whole construction process. Road image data are transmitted to a monitoring center in real time through 4G wireless transmission, and meanwhile, a three-dimensional visualization model (a three-dimensional reproduction model) is built. And identifying foreign matters in the roadbed paving process by adopting a deep learning neural network trained by a database and a positioning algorithm globally corresponding to relative coordinates, accurately positioning, and sending out early warning and calling foreign matter images in real time through the relative coordinate algorithm.
5) The method realizes the accurate positioning of the foreign matters in a relative coordinate calculation mode, overcomes the problems of large satellite positioning error and difficult fitting of the paths of the land leveler and the road roller, and has a relative coordinate positioning mode far lower than the installation of satellite positioning equipment in terms of cost.
6) In conclusion, the camera device is arranged on the land leveler, the influence of large vibration amplitude of the road roller is eliminated, the whole road section supervision, control and quality tracing in the whole process of road foundation soil foreign matter identification can be realized, the three-dimensional visual model is established, the real-time detection of the quality of the road foundation filler paved on the whole road section is realized (the old method adopts sampling detection), the problem that the materials used for indoor compaction tests are different from the materials used in construction sites can be solved due to the real-time monitoring, the large-scale database which is continuously updated is established for image identification, the identification accuracy is improved, the position of the foreign matter is accurately positioned by adopting a relative coordinate conversion mode, and the whole process refers to paving, compacting, operating and maintaining; the whole road section refers to that the whole paved road can be detected, and the main body is the identification of foreign matters in the roadbed soil.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present invention;
FIG. 2 is a flow chart of the present invention;
FIG. 3 is a three-view of a steel frame; wherein (a) is a schematic view of a main structure of the steel frame, and (b) is a schematic view of a side structure of the steel frame; (c) is a schematic view of a top view structure of the steel frame.
FIG. 4 is a side view schematic diagram of a grader carrying steel frame.
Fig. 5 is a diagram of a Segnet network structure according to the present invention;
FIG. 6 is a photograph of a grader strip in accordance with the present invention;
fig. 7 is a strip chart of the roller of the present invention.
Detailed Description
Specific examples of the present invention are given below. The specific examples are only for illustrating the present invention in further detail and do not limit the scope of protection of the present application.
And (3) roadbed construction process: after the road surface is cleaned, a square grid is drawn on the ground surface by lime, and the transport vehicle unloads soil according to the loose paving thickness determined by the roadbed filling test section. And adopting a grader to successively carry out primary leveling and fine leveling on the roadbed soil. After the roadbed soil is finely leveled, the road roller enters a field to compact the roadbed soil, no other interference is generated in the process, and the grain distribution of the finely leveled roadbed soil is the compacted grain distribution. Therefore, the present application creatively places the camera on the grader to eliminate the interference of the road roller vibration on image acquisition.
According to the foreign matter positioning method and device, the foreign matter can be accurately positioned and the working efficiency is high in the mode of relative coordinate positioning. The difference between the width of the grader blade and the width of the steel wheel of the road roller leads to the difference between the strip of the grader running on the road surface and the strip of the road roller, if GPS positioning is adopted, a complex calculation formula is needed, and the working efficiency is low. The relative position of the foreign body can be calculated according to a simple formula by adopting a relative coordinate positioning mode, so that the working efficiency is greatly improved. Moreover, the method can realize the switching between the land scraper belt and the road roller belt. What this application will do is detect the foreign matter, feeds back to surveillance center and road roller end, for the compaction process provides corresponding parameter, need not obtain the absolute position of foreign matter, is equivalent to the effect of highway stake number.
The invention relates to a method for identifying foreign matters in a roadbed compaction process by utilizing a deep learning neural network and accurately positioning and feeding back the foreign matters in real time, which can be used for the front end of intelligent compaction and provides stable roadbed filling condition parameters to guide the intelligent compaction process and provide feedback.
The intelligent compaction whole-process visualization control system used by the method comprises an image acquisition processing module, a three-dimensional drawing module, a deep learning neural network module, a database module and a calculation module;
the hardware equipment used mainly comprises three components: land leveller, monitoring center, road roller. The land leveler is provided with a shooting steel frame, a three-axis mechanical anti-shaking holder, a GOPRO HERO9 BLACK and a 4G real-time transmission device; the monitoring center comprises a database, a computing center, a 4G real-time transmission device and a display screen; the road roller comprises a 4G real-time transmission device and a display screen. The road bed compaction process is firstly carried out by using a grader to pave and level soil, then the road roller is used for rolling, the amplitude of the road roller during working is large, and the image acquisition quality is influenced, so that the camera device is arranged at the tail part of the grader, and the road roller is used for compaction after foreign matters are identified.
Image acquisition and processing module
The image acquisition processing module is mainly used for acquiring images by utilizing a camera at the tail of the grader and processing the images, and the grader is provided with the following equipment: a grader steel frame, a three-axis mechanical anti-shake pan head, a GOPRO HERO9 BLACK camera and a 4G real-time transmission device.
At leveler afterbody fixed mounting steelframe, as shown in fig. 4, the steelframe is whole to be triangle-shaped, the steelframe comprises many member bars, one of them member bar is the level and fixes at the leveler afterbody, another slope is fixed at the leveler afterbody, these two member bar one end are fixed at the upper and lower position of leveler, and the two other ends are crossing, the triaxial machinery anti-shake cloud platform is installed to the one end that the leveler was kept away from at the intersection point, the cloud platform is used for the shock attenuation, use the screw fixation at the intersection point, namely mark 1's position in fig. 3, install a plurality of cameras on the cloud platform, be responsible for the shooting on current road stripe road surface, the camera lens is perpendicular downwards, camera quantity is 2 in this embodiment, two cameras can shoot a complete stripe image. The tripod head carries a GOPRO HERO9 BLACK camera, and the camera parameters are as follows: video Resolution (RES)1080p, FPS (60Hz/50Hz)24/24, screen resolution 1920 × 1080, aspect ratio 16: 9.
the camera records the road surface along with the grader in the strip mode in the moving process, the video is guided into a monitoring center after the recording is finished, and video processing software, such as Premiere Pro CC2019 software, is installed in a computing center of the monitoring center.
And the Premiere Pro CC2019 software cuts the video, so that the starting time and the stopping time of the single-strip video shot by each camera correspond to the starting point and the ending point of the whole road.
The cut video obtains the length of an actual road corresponding to the pictures in the video through the actual length of a scale, the pixel length of the scale and the width of the screen resolution of the camera, and then the number of interval frames between two adjacent pictures when the pictures can be connected end to form the length of the whole road is obtained according to the running speed of the land leveler and the frame rate of the camera;
the number of the interval frames is determined according to the height, the resolution, the frame rate and the running speed of the land leveler, and the parameters are different under different construction conditions and need to be calculated according to actual conditions.
In the clipped video in this embodiment, the number of frames between two adjacent photos when the pictures can be connected end to form the length of the whole road is calculated by the formulas (1) to (3).
Number of frames t1×24 (3)
s1: the actual length of a video scale;
p1: video scale pixel length;
x1: the length of the actual road corresponding to the picture;
t1: a time interval;
v1: the running speed of the grader;
1080 is the width of the screen in this embodiment.
Intercepting pictures in the video image according to the determined interval frame number and intercepting the pictures, wherein the pixels of each picture are as follows: 1920 × 1080p, the intercepted pictures are pictures in an image set, relative coordinates of the intercepted pictures relative to the original point of the road are output, all the intercepted pictures in one video sequence are connected end to obtain an image of the whole road length of a half stripe, and the images of the whole road length of all the stripes are jointly spliced to obtain an image covering the whole road section. When the pictures are connected end to end, the width and the length of the pictures respectively correspond to the width and the pile number in an actual road, the width is taken as an X axis, the pile number is taken as a Y axis, a road coordinate system is established, and correspondingly, the width of the pictures in the image coordinates is taken as the X axis and the length of the pictures is taken as the Y axis.
The artificial intelligence algorithm intercepts the pictures according to the formulas (1) to (3) and the interval frame number, and can record the relative coordinate of the image origin of each picture relative to the road origin, so as to realize the recording of the corresponding pile number information.
Two, three-dimensional drawing module
The three-dimensional drawing module establishes an initial road model and draws a virtual camera in the monitoring center by using three-dimensional graphic software, and a 4G real-time transmission device arranged on the land scraper and a 4G real-time transmission device of the monitoring center are wirelessly transmitted by 4G, so that the actual detection scene of the land scraper can be synchronized in the initial road model in real time and put on a display screen of the monitoring center.
The initial road model receives an image set, data stream (X, Y, Z, T) from an image acquisition processing module, where X is an image origin abscissa, Y is an image origin ordinate, Z is a camera height, and T is an image acquisition time. And the image set is sequentially filled into the initial road model according to the image origin coordinates, so that the road actual scene reappears in the monitoring center.
And the road model receives a foreign object image set with a foreign object label output by a Segnet network of the deep learning neural network module, and updates the road model in real time according to the image origin coordinates in the foreign object image set.
The road model receives a foreign matter coordinate data stream (X 'and Y') obtained by conversion between relative coordinates, wherein X 'is the horizontal coordinate of the center of the foreign matter circumcircle under the road coordinate system, and Y' is the vertical coordinate of the center of the foreign matter circumcircle under the road coordinate system. And marking the foreign body position information in real time.
The road model is provided with a plurality of parameter interfaces, and can be connected with filling body parameters (water content and foreign body identification), road roller control parameters (paving thickness, vibration frequency, driving direction and driving speed) and real-time parameters (ICMV and temperature).
The integrated data stream and the image set (with the foreign object label) of the three-dimensional reproduction model which is continuously updated, namely the foreign object-free part in the original image set and the foreign object image set with the foreign object label) are jointly stored in the cloud storage unit by various parameter data (including data obtained by a parameter interface and also including foreign object coordinate data) received by the road model, so that the three-dimensional reproduction model is formed. The pictures can be amplified and the parameter information can be called at any time by clicking the road pictures (or inputting the positioning coordinates) in the three-dimensional reappearing model, so that the supervision, control and compaction quality tracing of the whole roadbed compaction process are realized.
Third, deep learning neural network module
The deep learning neural network module comprises two parts, namely a Segnet network and a CNN network, wherein the Segnet network is a network which operates independently and comprises a Mobilene Encoder part and a Segnet Decoder part. CNN is another network that operates independently.
1. And constructing a Segnet network based on a Mobilenet model.
The Segnet network is based on a Mobilene model and has a main structure of an Encoder-Decoder structure, and the structure of the Segnet network is shown in FIG. 5. The structure can be divided into a Mobilene Encoder part and a Segnet Decoder part.
The core of the Mobilene Encoder is Depthwise Separable Convolution, which decomposes a complete Convolution operation into two steps, i.e., Depthwise Convolution and Pointwise Convolution. Unlike conventional Convolution operations, one Convolution kernel of Depthwise Convolition is responsible for only one channel, and the number of layers of the Convolution kernel per Convolution is equal to the input depth number. Depthwise contribution does not efficiently utilize feature information of different channels at the same spatial position. And the poitwise contribution performs weighted combination on the images in the previous step in the depth direction to generate a new feature image.
The invention obtains 5 convolutional layers through a large amount of model training, and each convolutional layer is compressed for multiple times and extracted with characteristics. The activation function is a ReLU function, Depthwise _ conv _ block in the graph comprises Depthwise Convolution and Pointwise Convolution, the size of a Depthwise Convolution kernel is set to be 3 x 3, the step size is set to be 1 x 1, the size of a Pointwise Convolution kernel is set to be 1 x 1, and the step size is set to be 1 x 1. Finally, the Segnet Decoder structure is introduced by processing with f4 having features extracted a plurality of times.
The output layer is obtained by three UpSampling in the Segnet Decoder structure using the UpSampling2D function.
The deep learning neural network needs to train a large amount of data before being used, and after the image set obtained by interception is subjected to corresponding image preprocessing operation, the network is trained, and the trained network is stored as a model. When image recognition is carried out, an image set output by the real-time image acquisition processing module is directly led into the trained model.
Storing the trained Segnet network and the weight for subsequent image recognition, outputting an image with a foreign object label, and performing image preprocessing operation on the trained Segnet network model, wherein the image preprocessing operation comprises the following steps: image binaryzation, threshold denoising, foreign body expansion and foreign body corrosion.
2. And constructing the CNN network.
The total 13 layers of the CNN network used in the invention comprise: 1 input layer, 4 convolutional layers, 4 pooling layers, 3 full-link layers, and 1 output layer.
An input layer: images with a size of 100 x 100 and 3 channels were input.
And (3) rolling layers: the method is used for performing feature extraction of a feature map through convolution operation, wherein the size of a convolution kernel is 5 x 5, and an activation function is a ReLU function.
A pooling layer: and a maximum pooling method is adopted for compressing and reducing the dimension of the picture, reducing parameters and preventing overfitting.
Full connection layer: and the characteristic diagram is processed, so that the output of an output layer is facilitated. The use of the dropout function prevents or mitigates the overfitting problem. Where the drop ratio of the dropout function is 0.5.
An output layer: output was performed using the Softmax function.
The parameters needing to be manually set in the CNN network training process are hyper-parameters, and the hyper-parameters included in the invention are as follows: epoch, batch _ size, learning rate, normalized using L2, with initial parameters set to 0.0001. The CNN hyper-parameters are optimized using genetic algorithms.
And storing the trained CNN network for subsequent prediction. And extracting foreign matter information from the image with the foreign matter label output by the Segnet network, drawing a new picture by using conversion between relative coordinates, and putting the new picture into the CNN network for identification.
Database module
Large-scale tagged datasets are critical to deep learning, and large datasets can make tasks appear to grow as logarithmic functions.
The database module can expand the data volume, establish a large-scale database which is continuously updated, and improve the identification accuracy, and comprises a first database and a second database, wherein the first database stores an image set (comprising big data for a training network and current data acquired in real time) output by the image acquisition processing module; the second database stores an image set (called road whole-section foreign matter image set for short) for determining the actual distance of the circle center of the foreign matter circumscribed circle in the image set with the foreign matter label on the whole road coordinate (comprising big data for training the network and current data collected in real time).
Establishing a first database for the image set output by the image acquisition processing module (the initial data of the database is the picture intercepted by the grader), and adopting the following image preprocessing operations:
1. disorder of images: the shuffle function is used for disordering the image, and the generalization capability of the model is improved;
2. image labeling: labelme is used for labeling the output image set, and the label is used for labeling a part (foreign matter) to be identified, so that the foreign matter is found out, the outline of the foreign matter is marked, the foreign matter is displayed in different colors, the viewing is convenient, the observation is obvious, and the label is used for training a Segnet network;
3. image cutting: cutting the picture into a height 416, a width 416 and a channel number of 3;
4. image normalization: normalizing the image by using a normalization function;
5. image interpolation: enlarging the image by using BICUBIC (BICUBIC interpolation technology) to obtain a high-resolution image;
6. storing in a first database: and carrying out twenty-eight classification on the processed images, wherein 80% of data is used for network training, 20% of data is used for verification, and folders are named as follows: train and val for Segnet network training.
Establishing a second database for the whole road foreign body image set, and performing image preprocessing operation:
1. disorder of images: the shuffle function is used for disordering the image, and the generalization capability of the model is improved;
2. image cutting: cutting the image into 100 height and 100 width, and setting the number of channels as 3;
3. image normalization: normalizing the image by using a normalization function;
4. image labeling: the folder is named as: 0, indicating that the image is free of foreign matter; the folder is named as: 1, indicating that the image has foreign matter; the label is used for being divided into two folders to be respectively stored according to whether foreign matters exist or not, and only the images are classified and respectively stored into the corresponding folders;
5. storing into a second database: and carrying out twenty-eight classification on the processed images, wherein 80% of data is used for network training, 20% of data is used for verification, and folders are named as follows: train and val for CNN network training.
Both databases are used for training the network, and the corresponding databases are used for training the corresponding network before work.
And fifthly, the calculation module is used for converting relative coordinates and calculating early warning information.
1. Conversion between relative coordinates: the pictures in the image set output by the image acquisition processing module output relative coordinates of the pictures through a Segnet network, and the circumscribed circle center pixel point coordinates p 'of the foreign matters are output when the foreign matters exist'1And marking the foreign matters to obtain an image set with a foreign matter label.
By formula (4):
s1: the actual distance of the scale;
x2: actual distance of the foreign object from the image origin;
p1: scale pixel length;
p'1: foreign matter circumcircle center pixel point coordinates under an image coordinate system;
calculating the actual distance x between the foreign matter circumcircle center pixel point and the image origin according to the formula (4)2Substituting the coordinate information of the image origin to obtain the position x of the foreign object on the coordinate axis of the road origin (with the lower left corner of the road surface with the initial stake mark of the detected road section as the origin)4. As shown in fig. 6, phi represents a belt that the grader runs over, □ represents a camera photographing range,
represents a foreign substance. The picture is processed by image acquisitionAfter the module is cut, the coordinate information of the image origin is stored, when the foreign matter is identified, the coordinates of the pixel point of the circumscribed circle of the foreign matter in the image coordinate system are output and substituted into the formula (4), and the actual distance information x of the foreign matter from the image origin is obtained2. Then, the information is processed by formula (5) and image origin coordinate information x3And (4) calculating to obtain the actual distance (including the horizontal coordinate and the vertical coordinate of the circle center) of the foreign body circumcircle on the coordinate of the whole road.
x4＝x2+x3 (5)
x3: relative coordinate information of the picture of the foreign matter at the origin of the road;
x4: the actual distance of the foreign object in the whole road coordinate;
equations (4) and (5) are calculated by taking the abscissa x as an example, and both have the corresponding ordinate y.
After the conversion is finished, an image set (called a road whole-section foreign matter image set for short) with the actual distance of the circle center of the foreign matter circumcircle on the whole road coordinate is drawn by using matplotlib (packed and placed in a calculation module by an algorithm).
2. And (3) introducing the road whole-section foreign matter image set into a CNN network, and segmenting the image in the road whole-section foreign matter image set by the CNN network according to the road roller driving strip, as shown in FIG. 7. Phi represents a strip of a road roller,
represents a foreign substance. Judging whether foreign matter exists in the image for identifying each strip by using CNN network, and outputting the coordinate p 'of the foreign matter on the strip of the road roller'2。
Calculating early warning information: if foreign bodies exist, calculating the vertical coordinate position information of the foreign bodies on the road roller strip according to the formula (6):
s2: the actual length of the foreign matter image scale of the whole road section is measured;
p2: the proportion scale pixel length of the foreign matter image of the whole road section;
y5: the actual distance of the foreign bodies on the longitudinal coordinate of the road roller strip is m;
p'2: the number of longitudinal coordinate pixel points of the foreign matter on the road roller strip;
after the longitudinal coordinate position information of the foreign matter in the road roller strip is obtained, calculating the early warning time according to a formula (7):
t2＝(y5-0.5)/v2 (7)
t2: early warning time;
v2: road roller running speed
Calculating the running distance y from the strip origin to the foreign matter of the road roller through calculation of the early warning information5And the running time of the road roller from the original point of the strip to 0.5m before the foreign matter is taken as early warning time, the calculation module sends the running distance and the early warning time to the monitoring center in real time, the monitoring center and the road roller end communicate in a wireless mode to send out early warning, and meanwhile, the monitoring center and the road roller end are reminded of the existence of the foreign matter, so that the compaction quality is influenced, and corresponding operation is carried out to solve the problem. If no foreign matter exists, no early warning is generated, and the calculation of the early warning information in the calculation module can send road normal information to the monitoring center.
The working principle and the working process of the invention are as follows:
first, the working principle
The video camera records a video of the roadbed paving information at a fixed height, and the video is intercepted into pictures which are connected end to end through the processing of intercepting the video, dividing the video and the like, so that the full coverage of the roadbed paving is realized. Respectively extracting information from the processed images through a deep learning neural network, wherein the deep learning neural network comprises a Segnet network and a CNN network, and the extraction information of the Segnet network comprises the following steps: inputting an original image of the Segnet network, an image containing a foreign object label and a foreign object circumcircle center coordinate; the CNN network extraction information includes: inputting original image and foreign object pixel point coordinate p 'of CNN network'2Then through the monitoring centerThe calculation module realizes accurate positioning of the position information of the land leveler, the foreign matters and the road roller, avoids the problem that the position information of the land leveler, the foreign matters and the road roller cannot be accurately fitted due to large satellite positioning errors, controls the precision at the centimeter level and improves the working efficiency. The database improves the accuracy of the deep learning neural network model, improves the generalization capability of the deep learning neural network model, and prevents the over-fitting problem from occurring. The three-dimensional reappearing model carries out visual processing on road material information, wirelessly transmits and stores the road material information to the construction management platform, guides the subsequent rolling process, is convenient for adjusting construction parameters such as rolling speed, frequency and pass in real time, provides a basis for quality tracing after completion from the aspect of filler properties, and realizes monitoring, control, calling and quality tracing in the whole compaction process.
Second, the working process
1. Data acquisition and normalization: a steel frame and a three-axis mechanical anti-shake pan-tilt are mounted at the tail of the grader, a gopro hero9 black camera is mounted, and vertical shooting is carried out. 1080p video is shot at 24 frames per second (fps) using a standard video recording mode. And transmitting the video to a monitoring center in real time, and cutting the video by using Premiere Pro CC2019 software to enable the first picture to be aligned to the origin of the road. Only shoot a road length in the camera walking once process, set up two cameras according to this embodiment of the shooting scope of camera, two cameras are followed the leveler together and are recorded, the leveler is walked once, and the video sequence that two cameras were shot constitutes a strip, carries out the intercepting to the video of camera, recycles artificial intelligence algorithm and carries out the intercepting to the video image according to the mode of the fixed interval frame number intercepting picture of settlement, and the image pixel is: 1920 is multiplied by 1080p, the number of frames at the interval corresponds to the pile number, after the pile number is cut, each picture corresponds to one pile number, the relative coordinates of the original point of each cut picture (the width is taken as an X axis, and the pile number is taken as a Y axis) are output, the mode of cutting pictures with fixed sizes at the fixed interval is a standardization process, and the standardized road pictures are connected end to cover the whole road surface.
2. Establishing a three-dimensional reproduction model: the three-dimensional graphic software establishes a road model (three-dimensional model), draws a virtual camera, and receives an image set and a data stream (X, Y, Z, T) from an image acquisition processing module, wherein X is an image origin abscissa, Y is an image origin ordinate, and the image origin abscissa is constantly changed in the whole process because the road position corresponding to each image is different, the change of the coordinates is set in an artificial intelligence algorithm, the image is automatically output when the image is output, Z is the camera height, and T is the image acquisition time. And the image set is sequentially filled into the road model according to the coordinates to reproduce the actual scene of the road. Meanwhile, the image and the data stream are stored in the cloud storage unit.
3. Recognizing foreign matters by images: and importing the image processed by the image acquisition processing module into the trained Segnet neural network model. The Segnet network can be divided into a mobilen Encoder part and a Segnet Decoder part. And carrying out operations such as image binarization, threshold denoising, foreign matter expansion, foreign matter corrosion, external circle adding and the like on the image through a Segnet network. And outputting an original image imported into the Segnet network, namely, an image processed by the image acquisition processing module to a first database for subsequent updating training of the Segnet network model, and outputting foreign matter circumscribed circle center pixel points to the calculation module. And outputting the image and the coordinates containing the foreign matter label to the road model, updating the road model image according to the coordinates, and displaying the foreign matter label to realize the visualization of the foreign matter in the road model. And storing the image marked with the foreign matter label into a cloud storage unit, wherein the cloud storage unit is used for storing the input image and information.
4. Foreign body coordinate information calculation: after receiving the information output by the Segnet network, the computing module performs conversion between relative coordinates, and obtains coordinates (X ', Y') of the foreign object on a road origin coordinate system through a formula (5), wherein X 'is an abscissa of a circle center circumscribed by the foreign object under the road coordinates, and X' ═ X4(ii) a Y' is longitudinal coordinate of foreign body circumcircle center under road coordinate, Y ═ Y4. After coordinate information of all foreign matters in the road section is obtained, a whole road foreign matter image is drawn by using matplotlib (only the foreign matters are displayed in one image of the whole road, and then the road roller is divided) by taking the road width as an X axis and taking the length pile number as a Y axis, and a whole road foreign matter image set is obtained. At the same time, the road model is connectedAnd receiving data streams (X ', Y') of the horizontal and vertical coordinates of the external circle center of the foreign body under the road coordinates output by the calculation module, marking the position information of the foreign body in real time, and storing the data streams into a cloud storage unit.
5. Recognizing foreign matters by images: and packaging the segmented algorithm into a CNN network, and introducing the whole road foreign matter image set into the trained CNN network model. The CNN network has 13 layers in total, including: 1 input layer, 4 convolutional layers, 4 pooling layers, 3 full-link layers, and 1 output layer. The CNN network carries out operations such as image binarization, threshold denoising, foreign body expansion, foreign body corrosion and the like on images in the whole road segment foreign body image set to complete recognition of foreign bodies, outputs the whole road segment foreign body image set to a second database for CNN subsequent training, and outputs foreign body pixel points to a calculation module. And if the picture without the detected foreign object is not detected, sending the information of the foreign object to the computing module.
6. Calculating early warning information: after receiving the information output by the CNN network, the early warning information calculation part of the calculation module obtains the running distance of the road roller from the original point of the strip to the foreign object through a formula (6) if the foreign object exists, obtains the time of the road roller running 0.5m before the foreign object through a formula (7), and sends the early warning information to a monitoring center for storage.
7. Early warning and feedback: the monitoring center receives the information (whether foreign matters exist, the actual positions of the foreign matters and the early warning time) output by the calculation module. If foreign matters exist, the monitoring center sends out early warning and sends out early warning information, calls foreign matter pictures in the three-dimensional drawing module and sends the foreign matters to the road roller end. If no foreign matter exists, no early warning is generated, and the calculation of the early warning information in the calculation module can send road normal information to the monitoring center. And copying foreign matter early warning information of the monitoring center to send the foreign matter early warning information to a road roller end, so that a driver of the road roller can know the front condition and take corresponding operation, and the specific operation is not related to the invention.
8. Global visualization: the pictures can be amplified and the parameter information can be called at any time by clicking the road pictures (or inputting the positioning coordinates) in the three-dimensional reappearing model, so that the supervision, control and compaction quality tracing of the whole roadbed compaction process are realized.
Nothing in this specification is said to apply to the prior art.
Claims (9)
1. An intelligent compaction whole process visualization control method based on a relative coordinate positioning algorithm comprises the following contents:
collecting road surface images of all road sections by using a land leveler, and processing and storing the road surface images of all road sections according to the pile numbers to form an image set;
outputting the image set through a Segnet network in a deep learning neural network, wherein the image set is an image set with a foreign object label;
converting the image set with the foreign matter label into an image set with a foreign matter circumcircle and an actual distance of the circle center on the whole road coordinate through a relative coordinate positioning algorithm, which is called a road whole-section foreign matter image set for short;
processing the whole-section foreign matter image set of the road through a CNN network in a deep learning neural network, identifying each strip image under the driving strip of the road roller, and judging whether foreign matters exist or not;
and determining early warning information according to the foreign matter condition and guiding compaction operation.
2. The control method according to claim 1, wherein a grader is used for acquiring road surface images of all road sections, the road surface images of all road sections are processed and stored according to stake marks, and the specific process of forming the image set is as follows:
the method comprises the following steps that a three-axis mechanical anti-shake tripod head is installed at the tail of the grader, a plurality of cameras are installed on the tripod head and are responsible for shooting the road surface of a current road stripe, the camera lens is vertically downward, the cameras record the road surface in the movement process of the grader in a stripe mode along with the grader, and after the recording is finished, the video is guided into a monitoring center to cut the video, so that the starting time and the stopping time of a single stripe video shot by each camera correspond to the starting point and the ending point of the whole road;
the cut video obtains the length of an actual road corresponding to the pictures in the video through the actual length of a scale, the pixel length of the scale and the width of the screen resolution of the camera, and then the number of interval frames between two adjacent pictures when the pictures can be connected end to form the length of the whole road is obtained according to the running speed of the land leveler and the frame rate of the camera;
intercepting pictures in a video according to a video image in a mode of intercepting the pictures according to the determined interval frame number, wherein the intercepted pictures form an image set, outputting the relative coordinates of the intercepted pictures relative to the origin of the road, connecting all the intercepted pictures in a video sequence end to obtain an image with the length of one strip, splicing a plurality of video sequences acquired by a plurality of cameras to form an image with the length of the whole road of one strip, and jointly splicing the images with the length of the whole road of all the strips to obtain an image covering the whole road section; when the pictures are connected end to end, the width and the length of the pictures respectively correspond to the width and the stake number in the actual road, the relative coordinate of the image origin of each picture relative to the road origin is recorded, the recording of the corresponding stake number information is realized, and an image set is obtained.
3. The control method according to claim 2, wherein the interval frame number is obtained by calculation by equations (1) to (3),
number of frames t1×24 (3)
s1: the actual length of a video scale;
p1: video scale pixel length;
x1: the length of the actual road corresponding to the picture;
t1: a time interval;
v1: the running speed of the grader.
4. The control method according to claim 1, wherein the Segnet network is a Segnet network based on a mobilent model, comprising a mobilent Encoder part and a Segnet Decoder part; the core of the Mobileneet Encoder is depth separable Convolution, wherein the depth separable Convolution comprises Depthwise Convolution depth Convolution and Pointwise Convolution point by point; one Convolution kernel of the Depthwise Convolition is responsible for only one channel, and the number of layers of the Convolution kernel per Convolution is equal to the input depth number,
the size of the Depthwise Convolution kernel is set to 3 × 3, the step size is set to 1 × 1, the size of the Pointwise Convolution kernel is set to 1 × 1, and the step size is set to 1 × 1; finally, f4 with characteristics extracted for multiple times is used for processing, and a SegnetDecoder structure is introduced;
carrying out three times of UpSampling by utilizing an UpSampling2D function in a Segnet Decoder structure to obtain an output layer, outputting relative coordinates of each picture through a Segnet network after training, and simultaneously outputting circumscribed circle center pixel point coordinates p 'of foreign matters when the foreign matters exist'1And marking the foreign matters to obtain an image set with a foreign matter label.
5. The control method according to claim 1, wherein the specific process of the relative coordinate positioning algorithm is:
the image with foreign matter label can store the coordinate information of the image origin in a centralized way, when the foreign matter is identified, the coordinates of the pixel points of the circumscribed circle of the foreign matter under the image coordinate system are output and substituted into the formula (4), and the actual distance information x of the foreign matter from the image origin is obtained2(ii) a Then, the coordinate information x of the image origin is obtained through the formula (5)3Calculating to obtain the actual distance of the circle center of the foreign object circumscribed circle on the coordinates of the whole road, including the abscissa and the ordinate of the circle center;
x4＝x2+x3 (5)
s1: the actual distance of the scale;
x2: actual distance of the foreign object from the image origin;
p1: scale pixel length;
p'1: foreign matter circumcircle center pixel point coordinates under an image coordinate system;
x3: relative coordinate information of the picture of the foreign matter at the origin of the road;
x4: the actual distance of the foreign object in the whole road coordinate;
equations (4) and (5) are calculated by taking the abscissa x as an example, and the calculation process of the ordinate y is the same.
6. The control method according to claim 1, wherein the CNN network segments an image in which all road foreign object images are collected for each road roller driving band, recognizes an image of each road roller driving band, determines whether or not a foreign object is present, and outputs coordinates p 'of the foreign object on the road roller driving band'2。
7. The control method according to claim 6, wherein the specific process of determining the early warning information according to the foreign matter condition is:
calculating early warning information: if foreign bodies exist, calculating the vertical coordinate position information of the foreign bodies on the road roller strip according to the formula (6):
s2: the actual length of the foreign matter image scale of the whole road section is measured;
p2: the proportion scale pixel length of the foreign matter image of the whole road section;
y5: the actual distance of the foreign bodies on the longitudinal coordinate of the road roller strip, namely the running distance, is m;
p'2: under pressure of foreign matterThe number of longitudinal coordinate pixel points on the road machine strip;
after the longitudinal coordinate position information of the foreign matter in the road roller strip is obtained, calculating the early warning time according to a formula (7):
t2＝(y5-0.5)/v2 (7)
t2: early warning time;
v2: the running speed of the road roller;
the running distance and the early warning time are sent to a monitoring center in real time through calculation of early warning information, the monitoring center and the road roller end communicate in a wireless mode to send out early warning, and meanwhile, the monitoring center and the road roller end are reminded of the existence of foreign matters which can affect the compaction quality, and corresponding operation is carried out to solve the problem; if no foreign matter exists, no early warning is generated, and the calculation of the early warning information can send road normal information to the monitoring center.
8. An intelligent compaction overall process visualization control system based on a relative coordinate positioning algorithm is characterized by comprising an image acquisition processing module, a three-dimensional drawing module, a deep learning neural network module, a database module and a calculation module;
the image acquisition processing module is mainly used for acquiring images by utilizing a camera at the tail of the grader and processing the images, and the grader is provided with the following equipment: the system comprises a land leveler steel frame, a three-axis mechanical anti-shake holder, a camera and a wireless real-time transmission device;
the image processing process comprises the following steps: intercepting pictures in the video image in a mode of intercepting the pictures according to the determined interval frame number, wherein the intercepted pictures are connected end to form an image of the whole road length, and the relative coordinate of the image origin of each picture relative to the road origin can be recorded, so that the recording of corresponding pile number information is realized, and an image set is formed;
the three-dimensional drawing module establishes an initial road model and draws a virtual camera in a monitoring center by using three-dimensional graphic software, and a wireless real-time transmission device arranged on the land scraper and a wireless real-time transmission device of the monitoring center perform wireless transmission, so that the actual detection scene of the land scraper can be synchronized in real time in the initial road model and put on a display screen of the monitoring center; the initial road model receives an image set and a data stream (X, Y, Z, T) from the image acquisition processing module, wherein X is an image origin horizontal coordinate, Y is an image origin vertical coordinate, Z is a camera height, T is an image acquisition time, and the image set is sequentially filled into the initial road model according to the image origin coordinate to realize the reappearance of the actual road scene in the monitoring center;
the road model receives foreign body coordinate data streams (X 'and Y') obtained through conversion between relative coordinates, wherein X 'is the horizontal coordinate of the center of a foreign body circumcircle under a road coordinate system, and Y' is the vertical coordinate of the center of the foreign body circumcircle under the road coordinate system, and foreign body position information is labeled in real time;
the road model is provided with a plurality of parameter interfaces and can access filling body parameters, road roller control parameters and real-time parameters, and integrated data streams and image sets of various parameter data received by the road model are stored in the cloud storage unit together to form a three-dimensional reappearing model; the images can be amplified at any time and the parameter information can be called by clicking the road images in the three-dimensional reproduction model or inputting the positioning coordinates, so that the supervision, control and compaction quality tracing of the whole roadbed compaction process are realized;
the deep learning neural network module is used for outputting an image with a foreign matter label and judging whether the image under the road roller strip is a foreign matter;
the database module is used for storing and updating a corresponding image set of the training deep learning neural network module;
and the calculation module is used for conversion between relative coordinates and calculation of early warning information.
9. The control system of claim 8, wherein the database module comprises a first database and a second database, the first database having stored therein the image set output by the image acquisition processing module; the second database stores an image set for determining the actual distance of the circle center of the foreign body circumscribed circle in the image set with the foreign body label on the whole road coordinate, which is called a road whole-section foreign body image set for short,
the two databases comprise big data for training corresponding networks and current data collected in real time;
establishing a first database for the image set output by the image acquisition processing module, wherein the initial data of the database is the picture intercepted by the grader, and the following operations are carried out on the intercepted picture:
1) disorder of images: scrambling the image by using a shuffle function;
2) image labeling: labelme is used for labeling the output image set, the label is used for labeling a part (foreign matter) to be identified, so that the foreign matter is found out, the outline of the foreign matter is marked, the foreign matter is displayed in different colors, the viewing is convenient, and the observation is obvious;
3) image cutting: cutting the picture into a height 416, a width 416 and a channel number of 3;
4) image normalization: normalizing the image by using a normalization function;
5) image interpolation: adopting BICUBIC to amplify the image to obtain a high-resolution image;
6) storing in a first database: and carrying out twenty-eight classification on the processed images, wherein 80% of data is used for network training, 20% of data is used for verification, and folders are named as follows: train and val for Segnet network training;
establishing a second database for the whole road foreign body image set, and performing image preprocessing operation:
1) disorder of images: scrambling the image by using a shuffle function;
2) image cutting: cutting the image into 100 height and 100 width, and setting the number of channels as 3;
3) image normalization: normalizing the image by using a normalization function;
4) image labeling: the folder is named as: 0, indicating that the image is free of foreign matter; the folder is named as: 1, indicating that the image has foreign matter; the label is used for being divided into two folders to be respectively stored according to whether foreign matters exist or not, and only the images are classified and respectively stored into the corresponding folders;
5) storing into a second database: and carrying out twenty-eight classification on the processed images, wherein 80% of data is used for network training, 20% of data is used for verification, and folders are named as follows: train and val for CNN network training;
both databases are used for training the network, and the corresponding databases are used for training the corresponding network before work.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111195585.3A CN113808128B (en) | 2021-10-14 | 2021-10-14 | Intelligent compaction whole process visualization control method based on relative coordinate positioning algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111195585.3A CN113808128B (en) | 2021-10-14 | 2021-10-14 | Intelligent compaction whole process visualization control method based on relative coordinate positioning algorithm |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113808128A true CN113808128A (en) | 2021-12-17 |
| CN113808128B CN113808128B (en) | 2023-07-28 |
Family
ID=78897669
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111195585.3A Active CN113808128B (en) | 2021-10-14 | 2021-10-14 | Intelligent compaction whole process visualization control method based on relative coordinate positioning algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113808128B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2617453A (en) * | 2022-10-31 | 2023-10-11 | Univ Tongji | Three-dimensional intelligent filling construction management method of subgrade assisted by mixed reality technology |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20060037972A (en) * | 2004-10-29 | 2006-05-03 | 인하대학교 산학협력단 | Crack recognition and control system for use in crack sealer |
| CN106910186A (en) * | 2017-01-13 | 2017-06-30 | 陕西师范大学 | A kind of Bridge Crack detection localization method based on CNN deep learnings |
| CN108909624A (en) * | 2018-05-13 | 2018-11-30 | 西北工业大学 | A kind of real-time detection of obstacles and localization method based on monocular vision |
| CN109472831A (en) * | 2018-11-19 | 2019-03-15 | 东南大学 | Obstacle recognition range-measurement system and method towards road roller work progress |
| KR102026686B1 (en) * | 2018-11-12 | 2019-11-04 | 한국건설기술연구원 | Apparatus for providing dangerous road information using connected car and server for generating deep-learning model |
| WO2020139356A1 (en) * | 2018-12-27 | 2020-07-02 | Didi Research America, Llc | Image pre-processing in a lane marking determination system |
| CN111598095A (en) * | 2020-03-09 | 2020-08-28 | 浙江工业大学 | Deep learning-based urban road scene semantic segmentation method |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| CN112084892A (en) * | 2020-08-21 | 2020-12-15 | 廖赟 | Road abnormal event detection management device and method thereof |
| CN112184625A (en) * | 2020-09-01 | 2021-01-05 | 燊赛(上海)智能科技有限公司 | Pavement defect identification method and system based on video deep learning |
| CN112800911A (en) * | 2021-01-20 | 2021-05-14 | 同济大学 | Pavement damage rapid detection and natural data set construction method |
| CN113111703A (en) * | 2021-03-02 | 2021-07-13 | 郑州大学 | Airport pavement disease foreign matter detection method based on fusion of multiple convolutional neural networks |
| CN113112468A (en) * | 2021-04-02 | 2021-07-13 | 山东高速工程检测有限公司 | Road crack image identification and processing method |
| CN113280820A (en) * | 2021-06-09 | 2021-08-20 | 华南农业大学 | Orchard visual navigation path extraction method and system based on neural network |
-
2021
- 2021-10-14 CN CN202111195585.3A patent/CN113808128B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20060037972A (en) * | 2004-10-29 | 2006-05-03 | 인하대학교 산학협력단 | Crack recognition and control system for use in crack sealer |
| CN106910186A (en) * | 2017-01-13 | 2017-06-30 | 陕西师范大学 | A kind of Bridge Crack detection localization method based on CNN deep learnings |
| CN108909624A (en) * | 2018-05-13 | 2018-11-30 | 西北工业大学 | A kind of real-time detection of obstacles and localization method based on monocular vision |
| KR102026686B1 (en) * | 2018-11-12 | 2019-11-04 | 한국건설기술연구원 | Apparatus for providing dangerous road information using connected car and server for generating deep-learning model |
| CN109472831A (en) * | 2018-11-19 | 2019-03-15 | 东南大学 | Obstacle recognition range-measurement system and method towards road roller work progress |
| WO2020139356A1 (en) * | 2018-12-27 | 2020-07-02 | Didi Research America, Llc | Image pre-processing in a lane marking determination system |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| CN111598095A (en) * | 2020-03-09 | 2020-08-28 | 浙江工业大学 | Deep learning-based urban road scene semantic segmentation method |
| CN112084892A (en) * | 2020-08-21 | 2020-12-15 | 廖赟 | Road abnormal event detection management device and method thereof |
| CN112184625A (en) * | 2020-09-01 | 2021-01-05 | 燊赛(上海)智能科技有限公司 | Pavement defect identification method and system based on video deep learning |
| CN112800911A (en) * | 2021-01-20 | 2021-05-14 | 同济大学 | Pavement damage rapid detection and natural data set construction method |
| CN113111703A (en) * | 2021-03-02 | 2021-07-13 | 郑州大学 | Airport pavement disease foreign matter detection method based on fusion of multiple convolutional neural networks |
| CN113112468A (en) * | 2021-04-02 | 2021-07-13 | 山东高速工程检测有限公司 | Road crack image identification and processing method |
| CN113280820A (en) * | 2021-06-09 | 2021-08-20 | 华南农业大学 | Orchard visual navigation path extraction method and system based on neural network |
Non-Patent Citations (1)
| Title |
|---|
| 徐岩;陶慧青;虎丽丽;: "基于Faster R-CNN网络模型的铁路异物侵限检测算法研究", 铁道学报, no. 05, pages 91 - 98 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2617453A (en) * | 2022-10-31 | 2023-10-11 | Univ Tongji | Three-dimensional intelligent filling construction management method of subgrade assisted by mixed reality technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113808128B (en) | 2023-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103096121B (en) | A kind of camera movement detection method and device | |
| CN111895911B (en) | Method for monitoring hidden danger of ground collapse of shallow sand layer | |
| Xue et al. | Novel SfM-DLT method for metro tunnel 3D reconstruction and Visualization | |
| CN110362895B (en) | Land acquisition removal application management system based on BIM + GIS technology | |
| CN202350749U (en) | Measuring system for deformation characteristic of soil body | |
| CN111899332A (en) | Overhead transmission line three-dimensional design method based on oblique photogrammetry technology | |
| Karantanellis et al. | 3D hazard analysis and object-based characterization of landslide motion mechanism using UAV imagery | |
| CN113808128B (en) | Intelligent compaction whole process visualization control method based on relative coordinate positioning algorithm | |
| Berg et al. | Calculation of 3D displacement and time to failure of an earth dam using DIC analysis of hillshade images derived from high temporal resolution point cloud data | |
| CN106803075B (en) | Geological information intelligent recognition system and method based on image recognition technology | |
| CN113129283B (en) | Digital image-based filler grading intelligent detection compaction control method | |
| CN117953016B (en) | Flood discharge building exit area slope dangerous rock monitoring method and system | |
| JP2009052907A (en) | Foreign matter detecting system | |
| CN117516487B (en) | Medium-small river video flow test method | |
| CN114332658A (en) | Railway engineering equipment and surrounding environment hidden danger investigation method based on unmanned aerial vehicle inspection | |
| CN111897280A (en) | Earthwork engineering filler construction quality control system and analysis method | |
| CN117058586A (en) | Global land illegal activity monitoring method | |
| Gaich et al. | 3D images for data collection in tunnelling–applications and latest developments/3D‐Bilder für die Datenerfassung im Tunnelbau–Anwendung und aktuelle Entwicklungen | |
| CN112037250B (en) | Target vehicle vector track tracking and engineering view modeling method and device | |
| CN109029366B (en) | Method for measuring horizontal drift rate of algae spots through aircraft shooting | |
| Sun et al. | UAV photogrammetry-based accident assessment road condition analysis using image classification | |
| CN112964192A (en) | Engineering measurement online calibration method and system based on image video | |
| CN113989237B (en) | Nuclear power plant pavement crack rate calculation method based on gridding | |
| CN117036611B (en) | Three-dimensional scene construction method and system of non-mine safety monitoring platform | |
| CN114800873B (en) | 5G data transmission-based intelligent detection method and system for gravel material level AI of concrete mixing plant |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |